Table of Contents

BenchmarkDotNet v0.12.1

Highlights

  • .NET 5 support
    As you probably know, .NET Core 5 was officially rebranded to .NET 5. The new version of BenchmarkDotNet supports the new runtime after rebranding.
    #1399 465ebf
  • Perfolizer adoption
    The internal statistical engine of BenchmarkDotNet became mature enough to be transformed into an independent project. Meet perfolizer — a toolkit for performance analysis! While BenchmarkDotNet focuses on obtaining reliable measurements, perfolizer focuses on the decent analysis of measured data. You still can use all the statistical algorithms from BenchmarkDotNet, but you can also install perfolizer as a standalone NuGet package. You can find more details in the official announcement.
    #1386 54a061
  • Cross-platform disassembler
    Now the DisassemblyDiagnoser is cross-platform! The disassembling logic was also improved, now it handles runtime helper methods and references to method tables properly. Internally, it uses the Iced library for formatting assembly code.
    Special thanks to @adamsitnik for the implementation and @0xd4d for Iced!
    #1332 #899 #1316 #1364 294320
  • EventPipe-based cross-platform profiler
    Now you can easily profiler your benchmarks on Windows, Linux, and macOS! Just mark your class with the [EventPipeProfiler(...)] attribute and get a .speedscope.json file that you can browse in SpeedScope.
    Special thanks to @WojciechNagorski for the implementation!
    #1321 #1315 c648ff
  • New fluent API
    We continue to improve our API and make it easier for reading and writing.
    Special thanks to @WojciechNagorski for the implementation!
    #1273 #1234 640d88
  • Ref readonly support
    Now you can use ref readonly in benchmark signatures.
    Special thanks to @adamsitnik for the implementation!
    #1389 #1388 9ac777

Cross-platform disassembler

Just mark your benchmark class with the [DisassemblyDiagnoser] attribute and you will get the disassembly listings for all the benchmarks. The formatting looks pretty nice thanks to Iced. It works like a charm on Windows, Linux, and macOS.

[DisassemblyDiagnoser]
public class IntroDisassembly
{
    private int[] field = Enumerable.Range(0, 100).ToArray();

    [Benchmark]
    public int SumLocal()
    {
        var local = field; // we use local variable that points to the field

        int sum = 0;
        for (int i = 0; i < local.Length; i++)
            sum += local[i];

        return sum;
    }

    [Benchmark]
    public int SumField()
    {
        int sum = 0;
        for (int i = 0; i < field.Length; i++)
            sum += field[i];

        return sum;
    }
}

.NET Core 2.1.16 (CoreCLR 4.6.28516.03, CoreFX 4.6.28516.10), X64 RyuJIT

; BenchmarkDotNet.Samples.IntroDisassembly.SumLocal()
       mov       rax,[rcx+8]
       xor       edx,edx
       xor       ecx,ecx
       mov       r8d,[rax+8]
       test      r8d,r8d
       jle       short M00_L01
M00_L00:
       movsxd    r9,ecx
       add       edx,[rax+r9*4+10]
       inc       ecx
       cmp       r8d,ecx
       jg        short M00_L00
M00_L01:
       mov       eax,edx
       ret
; Total bytes of code 35

.NET Core 2.1.16 (CoreCLR 4.6.28516.03, CoreFX 4.6.28516.10), X64 RyuJIT

; BenchmarkDotNet.Samples.IntroDisassembly.SumField()
       sub       rsp,28
       xor       eax,eax
       xor       edx,edx
       mov       rcx,[rcx+8]
       cmp       dword ptr [rcx+8],0
       jle       short M00_L01
M00_L00:
       mov       r8,rcx
       cmp       edx,[r8+8]
       jae       short M00_L02
       movsxd    r9,edx
       add       eax,[r8+r9*4+10]
       inc       edx
       cmp       [rcx+8],edx
       jg        short M00_L00
M00_L01:
       add       rsp,28
       ret
M00_L02:
       call      CORINFO_HELP_RNGCHKFAIL
       int       3
; Total bytes of code 53

Now we handle runtime helper methods and references to method tables properly. Example:

Before:

; MicroBenchmarks.WithCallsAfter.Benchmark(Int32)
       push    rsi
       sub     rsp,20h
       mov     rsi,rcx
       cmp     edx,7FFFFFFFh
       jne     M00_L00
       call    MicroBenchmarks.WithCallsAfter.Static()
       mov     rcx,rsi
       call    MicroBenchmarks.WithCallsAfter.Instance()
       mov     rcx,rsi
       call    MicroBenchmarks.WithCallsAfter.Recursive()
       mov     rcx,rsi
       mov     rax,qword ptr [rsi]
       mov     rax,qword ptr [rax+40h]
       call    qword ptr [rax+20h]
       mov     rcx,rsi
       mov     edx,1
       mov     rax,7FF8F4217050h
       add     rsp,20h
       pop     rsi
       jmp     rax
M00_L00:
       mov     rcx,offset System_Private_CoreLib+0xa31d48
       call    coreclr!MetaDataGetDispenser+0x322a0
       mov     rsi,rax
       mov     ecx,0ACFAh
       mov     rdx,7FF8F42F4680h
       call    coreclr!MetaDataGetDispenser+0x17140
       mov     rdx,rax
       mov     rcx,rsi
       call    System.InvalidOperationException..ctor(System.String)
       mov     rcx,rsi
       call    coreclr!coreclr_shutdown_2+0x39f0
       int     3
       add     byte ptr [rax],al
       sbb     dword ptr [00007ff9`26284e30],eax
       add     dword ptr [rax+40h],esp
       add     byte ptr [rax],al
       add     byte ptr [rax],al
       add     byte ptr [rax],al
       add     byte ptr [rax-70BC4CCh],ah
; Total bytes of code 157

After:

; BenchmarkDotNet.Samples.WithCallsAfter.Benchmark(Int32)
       push      rsi
       sub       rsp,20
       mov       rsi,rcx
       cmp       edx,7FFFFFFF
       jne       M00_L00 
       call      BenchmarkDotNet.Samples.WithCallsAfter.Static() 
       mov       rcx,rsi
       call      BenchmarkDotNet.Samples.WithCallsAfter.Instance() 
       mov       rcx,rsi
       call      BenchmarkDotNet.Samples.WithCallsAfter.Recursive() 
       mov       rcx,rsi
       mov       rax,[rsi]
       mov       rax,[rax+40]
       call      qword ptr [rax+20]
       mov       rcx,rsi
       mov       edx,1
       mov       rax BenchmarkDotNet.Samples.WithCallsAfter.Benchmark(Boolean) 
       add       rsp,20
       pop       rsi
       jmp       rax
M00_L00:
       mov       rcx MT_System.InvalidOperationException 
       call      CORINFO_HELP_NEWSFAST 
       mov       rsi,rax
       mov       ecx,12D
       mov       rdx,7FF954FF83F0
       call      CORINFO_HELP_STRCNS 
       mov       rdx,rax
       mov       rcx,rsi
       call      System.InvalidOperationException..ctor(System.String) 
       mov       rcx,rsi
       call      CORINFO_HELP_THROW 
       int       3
; Total bytes of code 134

See also: Cross-runtime .NET disassembly with BenchmarkDotNet.

Special thanks to @adamsitnik for the implementation and @0xd4d for Iced!

EventPipe-based cross-platform profiler

Now you can easily profiler your benchmarks on Windows, Linux, and macOS!

If you want to use the new profiler, you should just mark your benchmark class with the [EventPipeProfiler(...)] attribute:

[EventPipeProfiler(EventPipeProfile.CpuSampling)] // <-- Enables new profiler
public class IntroEventPipeProfiler
{
    [Benchmark]
    public void Sleep() => Thread.Sleep(2000);
}

Once the benchmark run is finished, you get a .speedscope.json file that can be opened in SpeedScope:

The new profiler supports several modes:

  • CpuSampling - Useful for tracking CPU usage and general .NET runtime information. This is the default option.
  • GcVerbose - Tracks GC collections and samples object allocations.
  • GcCollect - Tracks GC collections only at very low overhead.
  • Jit - Logging when Just in time (JIT) compilation occurs. Logging of the internal workings of the Just In Time compiler. This is fairly verbose. It details decisions about interesting optimization (like inlining and tail call)

Please see Wojciech Nagórski's blog post for all the details.

Special thanks to @WojciechNagorski for the implementation!

New fluent API

We continue to improve our API and make it easier for reading and writing. The old API is still existing, but it is marked as obsolete and will be removed in the further library versions. The most significant changes:

Changes in Job configuration

Changes in IConfig/ManualConfig

Full fluent API

Special thanks to @WojciechNagorski for the implementation!

Ref readonly support

Now you can use ref readonly in benchmark signatures. Here is an example:

public class RefReadonlyBenchmark
{
    static readonly int[] array = { 1 };

    [Benchmark]
    public ref readonly int RefReadonly() => ref RefReadonlyMethod();

    static ref readonly int RefReadonlyMethod() => ref array[0];
}

Special thanks to @adamsitnik for the implementation!

Milestone details

In the v0.12.1 scope, 31 issues were resolved and 42 pull requests were merged. This release includes 85 commits by 19 contributors.

Resolved issues (31)

  • #641 RPlotExporter hanging (assignee: @m-mccormick)
  • #899 Tiered compilation and disassembler (assignee: @adamsitnik)
  • #1023 Out of process benchmarks fail with ASP.NET Core SDK reference
  • #1211 Binding Redirect Issues When Using Xml Serializers
  • #1234 Strong type fluent API proposal (assignee: @WojciechNagorski)
  • #1238 RunAllJoined Causing Exception (assignee: @gsomix)
  • #1262 Params attribute doesn`t work in F# if you specify more than one enum value in constructor (assignee: @gsomix)
  • #1295 Custom format/culture for report output values (for CSV, and maybe HTML, MD)
  • #1305 Copy UserSecrets from benchmark project
  • #1311 Spelling nit (assignee: @AndreyAkinshin)
  • #1312 Add an option to pass environment variables to the default job
  • #1315 Implement cross platform EventPipeProfiler diagnoser (assignee: @WojciechNagorski)
  • #1316 Implement Unix Disassembler for .NET Core (assignee: @adamsitnik)
  • #1318 use of NugetReference[] causes System.MissingMethodException: No parameterless constructor defined for this object. (assignee: @adamsitnik)
  • #1323 DisassemblyDiagnoser index outside array bounds (assignee: @AndreyAkinshin)
  • #1325 Surface native code size benchmarked code (assignee: @adamsitnik)
  • #1326 BDN does not build using dotnet sdk from the command line in Linux
  • #1339 Generated code and StyleCop.Analyzers (assignee: @adamsitnik)
  • #1348 Different display text for arrays depending on a value source (assignee: @YohDeadfall)
  • #1350 Warn the user if command line arguments were not passed to the BenchmarkSwitcher
  • #1353 Show Length when param type is an array
  • #1361 SimpleJobAttribute with RunStrategy and RuntimeMoniker
  • #1363 Wrong assembly binding redirects for Microsoft.Data.SqlClient.resources ; using in Netcore3.0 project (assignee: @adamsitnik)
  • #1364 Bug: Benchmark class with Console.WriteLine(1) fails for DisassemblyDiagnoser with 'Sequence contains no matching element' (assignee: @adamsitnik)
  • #1369 Parameter column doesn't seem to respect culture info (assignee: @Tyrrrz)
  • #1379 Unix CI builds are red (assignee: @AndreyAkinshin)
  • #1385 Make BaselineCustomColumn.GetValue public
  • #1388 'ref readonly' return is not supported (assignee: @adamsitnik)
  • #1396 MacOS Azure Pipeline build is broken (assignee: @AndreyAkinshin)
  • #1413 Plot with only one "default" Job (assignee: @AndreyAkinshin)
  • #1416 EventPipeProfiler Documentation (assignee: @WojciechNagorski)

Merged pull requests (42)

Commits (85)

Contributors (19)

Thank you very much!

Additional details

Date: April 6, 2020

Milestone: v0.12.1 (List of commits)

NuGet Packages: