Table of Contents

BenchmarkDotNet v0.13.0

It's been a year since our last release. BenchmarkDotNet has been downloaded more than seven million times from nuget.org. It's more than we could have ever possibly imagined! Some could say, that it's also more than we can handle ;) That is why we wanted to once again thank all the contributors who helped us with 0.13.0 release!

Highlights

In BenchmarkDotNet v0.13.0, we have supported various technologies:

  • .NET 5 and .NET 6 target framework monikers
  • .NET SDK installed via snap
  • SingleFile deployment
  • Xamarin applications
  • WASM applications
  • Mono AOT

We have also introduced new features and improvements including:

  • Memory randomization
  • Method-specific job attributes
  • Sortable parameter columns
  • Customizable ratio column
  • Improved CoreRun and CoreRT support
  • Improved Hardware Counters support

Of course, this release includes dozens of other improvements and bug fixes!

Supported technologies

.NET 5, .NET 6, SingleFile and snap

At some point in time, netcoreapp5.0 moniker was changed to net5.0, which required a fix on our side (#1479, btw we love this kind of changes). Moreover, .NET 5 introduced platform-specific TMFs (example: net5.0-windows10.0.19041.0) which also required some extra work: #1560, #1691.

In #1523 support for .NET 6 was added.

<TargetFrameworks>net5.0;net5.0-windows10.0.19041.0;net6.0<TargetFrameworks>

In #1686 @am11 has implemented support for single file deployment (supported in .NET 5 onwards).

Last, but not least in #1652 snap support has been implemented.

adam@adsitnik-ubuntu:~/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples$ dotnet50 run -c Release -f net5.0 --filter BenchmarkDotNet.Samples.IntroColdStart.Foo
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 1 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start /snap/dotnet-sdk/112/dotnet restore  /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b
// command took 1.49s and exited with 0
// start /snap/dotnet-sdk/112/dotnet build -c Release  --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b
// command took 2.78s and exited with 0
// ***** Done, took 00:00:04 (4.37 sec)   *****
// Found 1 benchmarks:
//   IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)

// **************************
// Benchmark: IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)
// *** Execute ***
// Launch: 1 / 1
// Execute: /snap/dotnet-sdk/112/dotnet "9a018ee4-0f33-46dd-9093-01d3bf31233b.dll" --benchmarkName "BenchmarkDotNet.Samples.IntroColdStart.Foo" --job "IterationCount=5, RunStrategy=ColdStart" --benchmarkId 0 in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b/bin/Release/net5.0

Xamarin support

Thanks to the contributions of the amazing @jonathanpeppers BenchmarkDotNet supports Xamarin! The examples can be found in our repo: iOS, Android.

#1360, #1429, #1434, #1509

WASM support

Thanks to the work of @naricc you can now benchmark WASM using Mono Runtime! For more details, please refer to our docs.

#1483, #1498, #1500, #1501, #1507, #1592, #1689.

Mono AOT support

In another awesome contribution (#1662) @naricc has implemented Mono AOT support. The new toolchain supports doing Mono AOT runs with both the Mono-Mini compiler and the Mono-LLVM compiler (which uses LLVM on the back end).

For more details, please go to our docs.

New features and improvements

Memory randomization

In #1587 @adamsitnik has introduced a new, experimental feature called "Memory Randomization".

This feature allows you to ask BenchmarkDotNet to randomize the memory alignment by allocating random-sized byte arrays between iterations and call [GlobalSetup] before every benchmark iteration and [GlobalCleanup] after every benchmark iteration.

Sample benchmark:

public class IntroMemoryRandomization
{
    [Params(512 * 4)]
    public int Size;

    private int[] _array;
    private int[] _destination;

    [GlobalSetup]
    public void Setup()
    {
        _array = new int[Size];
        _destination = new int[Size];
    }

    [Benchmark]
    public void Array() => System.Array.Copy(_array, _destination, Size);
}

Without asking for the randomization, the objects are allocated in [GlobalSetup] and their unmodified addresses (and alignment) are used for all iterations (as long as they are not promoted to an older generation by the GC). This is typically the desired behavior, as it gives you very nice and flat distributions:

dotnet run -c Release --filter IntroMemoryRandomization

-------------------- Histogram --------------------
[502.859 ns ; 508.045 ns) | @@@@@@@@@@@@@@@
---------------------------------------------------

But if for some reason you are interested in getting a distribution that is better reflecting the "real-life" performance you can enable the randomization:

dotnet run -c Release --filter IntroMemoryRandomization --memoryRandomization true

-------------------- Histogram --------------------
[108.803 ns ; 213.537 ns) | @@@@@@@@@@@@@@@
[213.537 ns ; 315.458 ns) |
[315.458 ns ; 446.853 ns) | @@@@@@@@@@@@@@@@@@@@
[446.853 ns ; 559.259 ns) | @@@@@@@@@@@@@@@
---------------------------------------------------

Method-specific job attributes

From now, all attributes that derive from JobMutatorConfigBaseAttribute (full list) can be applied to methods. You no longer have to move a method to a separate type to customize config for it.

[Benchmark]
[WarmupCount(1)]
public void SingleWarmupIteration()

[Benchmark]
[WarmupCount(9)]
public void NineWarmupIterations()

Sortable parameter columns

In order to sort columns of parameters in the results table, you can use the Property Priority inside the params attribute. The priority range is [Int32.MinValue;Int32.MaxValue], lower priorities will appear earlier in the column order. The default priority is set to 0.

public class IntroParamsPriority
{
    [Params(100)]
    public int A { get; set; }

    [Params(10, Priority = -100)]
    public int B { get; set; }

    [Benchmark]
    public void Benchmark() => Thread.Sleep(A + B + 5);
}
Method B A Mean Error StdDev
Benchmark 10 100 115.4 ms 0.12 ms 0.11 ms

This feature got implemented by @JohannesDeml in #1612.

Customizable ratio column

Now it's possible to customize the format of the ratio column.

[Config(typeof(Config))]
public class IntroRatioStyle
{
    [Benchmark(Baseline = true)]
    public void Baseline() => Thread.Sleep(1000);

    [Benchmark]
    public void Bar() => Thread.Sleep(150);

    [Benchmark]
    public void Foo() => Thread.Sleep(1150);

    private class Config : ManualConfig
    {
        public Config()
        {
            SummaryStyle = SummaryStyle.Default.WithRatioStyle(RatioStyle.Trend);
        }
    }
}

|   Method |       Mean |   Error |  StdDev |        Ratio | RatioSD |
|--------- |-----------:|--------:|--------:|-------------:|--------:|
| Baseline | 1,000.6 ms | 2.48 ms | 0.14 ms |     baseline |         |
|      Bar |   150.9 ms | 1.30 ms | 0.07 ms | 6.63x faster |   0.00x |
|      Foo | 1,150.4 ms | 5.17 ms | 0.28 ms | 1.15x slower |   0.00x |

This feature was implemented in #731.

Improved CoreRun support

BenchmarkDotNet was reporting invalid .NET Core version number when comparing performance using CoreRuns built from dotnet/corefx and dotnet/runtime. Fixed by @adamsitnik in #1580

In #1552 @stanciuadrian has implemented support for all GcMode characteristics for CoreRunToolchain. Previously the settings were just ignored, now they are being translated to corresponding COMPlus_* env vars.

Improved CoreRT support

CoreRT has moved from https://github.com/dotnet/corert/ to https://github.com/dotnet/runtimelab/tree/feature/NativeAOT and we had to update the default compiler version and nuget feed address. Moreover, there was a bug in CoreRtToolchain which was causing any additional native dependencies to not work.

Big thanks to @MichalStrehovsky, @jkotas and @kant2002 for their help and support!

#1606, #1643, #1679

Command-line argument support in BenchmarkRunner

So far only BenchmarkSwitcher was capable of handling console line arguments. Thanks to @chan18 BenchmarkRunner supports them as well (#1292):

public class Program
{
    public static void Main(string[] args) => BenchmarkRunner.Run(typeof(Program).Assembly, args: args); 
}

New API: ManualConfig.CreateMinimumViable

ManualConfig.CreateEmpty creates a completely empty config. Without adding a column provider and a logger to the config the users won't see any results being printed. In #1582 @adamsitnik has introduced a new method that creates minimum viable config:

IConfig before = ManualConfig.CreateEmpty()
    .AddColumnProvider(DefaultColumnProviders.Instance)
    .AddLogger(ConsoleLogger.Default);

IConfig after = ManualConfig.CreateMinimumViable();

Benchmarking NuGet packages from custom feeds

In #1659 @workgroupengineering added the possibility to indicate the source of the tested nuget package and whether it is a pre-release version.

Deterministic benchmark builds

BenchmarkDotNet is now always enforcing Deterministic builds (#1489) and Optimizations enabled (#1494) which is a must-have if you are using custom build configurations. MSBuild enforces optimizations only for configurations that are named Release (the comparison is case-insensitive).

<ItemGroup Condition=" '$(Configuration)' == 'X' ">
   <PackageReference Include="SomeLibThatYouWantToBenchmark" Version="1.0.0" />
</ItemGroup>
<ItemGroup Condition=" '$(Configuration)' == 'Y' ">
   <PackageReference Include="SomeLibThatYouWantToBenchmark" Version="2.0.0" />
</ItemGroup>

var config = DefaultConfig.Instance
  .AddJob(Job.Default.WithCustomBuildConfiguration("X").WithId("X").AsBaseline())
  .AddJob(Job.Default.WithCustomBuildConfiguration("Y").WithId("Y"));

#1489, #1494

Improved Hardware Counters support

BenchmarkDotNet is being used by the .NET Team to ensure that .NET is not regressing. More than three thousand benchmarks (they can be found here) are being executed multiple times a day on multiple hardware configs. Recently, .NET Team started to use InstructionsRetired to help to filter unstable benchmarks that report regressions despite not changing the number of instructions retired. This has exposed few bugs in Hardware Counters support in BenchmarkDotNet, which all got fixed by @adamsitnik in #1547 and #1550. Moreover, we have removed the old PmcDiagnoser and extended EtwProfiler with the hardware counters support. It's just much more accurate and futureproof. For details, please go to #1548.

How stable was PmcDiagnoser (same benchmarks run twice in a row on the same machine):

Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 845,746
Burgers_0 .NET Core 2.1 30,154,151
Burgers_0 .NET Framework 4.6.1 4,230,848
Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 34,154,524
Burgers_0 .NET Core 2.1 246,534,203
Burgers_0 .NET Framework 4.6.1 2,607,686

How stable is EtwProfiler:

Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 3,069,978,261
Burgers_0 .NET Core 2.1 3,676,000,000
Burgers_0 .NET Framework 4.6.1 3,468,866,667
Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 3,066,810,000
Burgers_0 .NET Core 2.1 3,674,666,667
Burgers_0 .NET Framework 4.6.1 3,468,600,000

Moreover, in #1540 @WojciechNagorski has added the removal of temporary files created by EtwProfiler.

Improved Troubleshooting

We have the possibility to ask BDN to stop on the first error: --stopOnFirstError true.

The problem was when the users had not asked for that, tried to run n benchmarks, all of them failed to build, and BDN was printing the same build error n times.

In #1672 @adamsitnik has changed that, so when all the build fails, BDN stops after printing the first error.

Moreover, we have also changed the default behavior for the failed builds of the boilerplate code. If the build fails, we don't remove the files. Previously we have required the users to pass --keepFiles to keep them. See #1567 for more details and don't forget about the Troubleshooting docs!

Docs and Samples improvements

Big thanks to @lukasz-pyrzyk, @fleckert, @MarecekF, @joostas, @michalgalecki, @WojciechNagorski, @MendelMonteiro, @kevinsalimi, @cedric-cf, @YohDeadfall, @jeffhandley and @JohannesDeml who have improved our docs and samples!

#1463, #1465, #1508, #1518, #1554, #1568, #1601, #1633, #1645, #1647, #1657, #1675, #1676, #1690.

Template improvements

  • Projects created out of our official templates might have been unexpectedly packed when running dotnet pack on the entire solution. In #1584 @kendaleiv has explicitly disabled packing for the template.
  • The template had netcoreapp3.0 TFM hardcoded. This got fixed by @https://github.com/ExceptionCaught in #1630 and #1632.
  • In #1667 @YohDeadfall has changed the default debug type from portable to pdbonly (required by DisassemblyDiagnoser).

Bug fixes

  • Very long string [Arguments] and [Params] were causing BenchmarkDotNet to crash. Fixed by @adamsitnik in #1248 and #1545. So far trace file names were containing full benchmark names and arguments. Now if the name is too long, the trace file name is a hash (consistent for multiple runs of the same benchmark). The same goes for passing benchmark name by the host process to the benchmark process via command-line arguments.
  • LangVersion set to a non-numeric value like latest was crashing the build. Fixed by @martincostello in #1420.
  • Windows 10 November 2019 was being recognized as 2018. Fixed by @kapsiR in #1437.
  • Assemblies loaded via streams were not supported. Fixed by @jeremyosterhoudt in #1443.
  • NativeMemoryProfiler was detecting small leaks that were false positives. Fixed by @WojciechNagorski in #1451 and #1600.
  • DisassemblyDiagnoser was crashing on Linux. Fixed by @damageboy in #1459.
  • Target framework moniker was being printed as toolchain name for Full .NET Framework benchmarks. Fixed by @svick in #1471.
  • [ParamsSource] returning IEnumerable<object[]> was not working properly when combined with [Arguments]. Fixed by @adamsitnik in #1478.
  • NullReferenceException in MultimodalDistributionAnalyzer. Fixed by @suslovk in #1488.
  • NotSupportedException was being thrown if there was an encoding mismatch between host and benchmark process. Diagnosed by @ChristophLindemann in #1487, fixed by @lovettchris in #1491.
  • MissingMethodException was being thrown in projects that referenced a newer version of Iced. Fixed by @Symbai in #1497 and in #1502.
  • AppendTargetFrameworkToOutputPath set to false was not supported. Fixed by @adamsitnik in #1563
  • A locking finalizer could have hanged benchmark process which would just print // AfterAll and never quit. Fixed by @adamsitnik in #1571. To prevent other hangs from happening, a timeout of 250ms was added. If the process does not quit after running the benchmarks and waiting 250ms, it's being force killed.
  • In some cases, JsonExporter was reporting NaN for some of the Statistics. This was breaking non-.NET JSON deserializers. Fixed by @marcnet80 in #1581.
  • UnitType.Size metrics were not using the provided number format. Fixed by @jodydonetti in #1569 and #1618.
  • MaxColumnWidth setting was not used for type names. Fixed by @JohannesDeml in #1609.
  • Current culture was not respected when formatting Ratio column values. Fixed by @JohannesDeml in #1610.
  • BenchmarkDotNet was redirecting Standard Error of the benchmark process, which was causing deadlocks for benchmarks that were writing to it. Fixed by @adamstinik in #1631
  • DisassemblyDiagnoser was failing to disassemble multiline source code. @YohDeadfall fixed that in #1674.
  • In #1644 @adamstinik has fixed the inconsistency between benchmark filter and hint.

Removal of the dotnet global tool

In #1006 (0.11.4) we have introduced a new dotnet global tool.

By looking at the number of reported bugs we got to the conclusion that the tool has not passed the test of time.

Why? Because it was based entirely on dynamic assembly loading which is very hard to get right in .NET and the fact that we support all existing .NET Runtimes (.NET, .NET Core, Mono, CoreRT) made it even harder (if not impossible).

We have removed it and the old versions are no longer supported. For more details, please refer to #1572.

Milestone details

In the v0.13.0 scope, 53 issues were resolved and 94 pull requests were merged. This release includes 111 commits by 37 contributors.

Resolved issues (53)

  • #721 Add possibility to customize BaselinedScaledColumn with provided func (assignee: @AndreyAkinshin)
  • #1136 benchmark / beforeEverything fails when running in docker-win
  • #1242 bug JSON Exporter exports NaN for some properties. This fails most JSON parsers (assignee: @marcnet80)
  • #1247 Support benchmarks with very long string arguments (assignee: @adamsitnik)
  • #1288 Fix Hardware Counters diagnoser (assignee: @adamsitnik)
  • #1290 BenchmarkRunner should support parsing console line arguments (assignee: @chan18)
  • #1333 Getting "Unknown Processor" Again
  • #1427 NativeMemoryProfiler reports false positive leak (assignee: @WojciechNagorski)
  • #1431 System.IO.FileNotFoundException with EtwProfiler
  • #1433 Benchmarks work in .NET 4.7.2 but not .NET Core 3.1
  • #1436 Wrong OsBrandString for Windows 10 1909
  • #1448 NRE in MultimodalDistributionAnalyzer
  • #1452 Setting LangVersion to Latest Causes an Error
  • #1464 DisassemblyDiagnoserConfig.ExportDiff needs better description
  • #1472 Add support of Wasm to Benchmark Dotnet (assignee: @naricc)
  • #1474 docs: Getting started guide
  • #1480 BenchmarkRunner not working from VSTS agent (assignee: @lovettchris)
  • #1487 Benchmarks do not execute (as service process / VSTS agent) on non US systems
  • #1493 Using WithCustomBuildConfiguration leads to always running with RyuJIT Debug (assignee: @adamsitnik)
  • #1495 [GcServer(true)] is ignored with --corerun
  • #1496 System.MissingMethodException: Method not found: 'Iced.Intel.MasmFormatter.get_MasmOptions()'.
  • #1512 Template has hardcoded netcoreapp3.0 TFM (assignee: @ExceptionCaught)
  • #1532 Auto-generated project has invalid XML
  • #1535 Can't benchmark library that targets 'net5.0-windows' framework (assignee: @adamsitnik)
  • #1537 Failed to run benchmarks on .net 5 RC1 and Preview LangVersion
  • #1539 Inconsistency between benchmark filter and hint (assignee: @adamsitnik)
  • #1544 [EtwProfiler] Merge operation failed return code 0x3 (assignee: @adamsitnik)
  • #1546 Sometimes Hardware Counters per Op are reported as NaNs (assignee: @adamsitnik)
  • #1549 InstructionPointerExporter has been broken for a while (assignee: @adamsitnik)
  • #1559 [Docs] Update Console Args doc (assignee: @kevinsalimi)
  • #1561 NativeMemoryProfiler doesn't report allocations in v0.12.1.1432 Nightly (assignee: @WojciechNagorski)
  • #1564 error MSB4086: A numeric comparison was attempted on "$(LangVersion)" (assignee: @adamsitnik)
  • #1565 More consistent formatting of results
  • #1566 BDN should not delete temporary build directories for failed builds (assignee: @adamsitnik)
  • #1570 Benchmark runs failing when using .NET 5 RC2 SDK installed via snap (assignee: @adamsitnik)
  • #1576 Missing/misleading version number with corerun
  • #1585 Non-optimized dependency
  • #1591 Wasm Benchmark Runs Failing with Target Framework Error (assignee: @naricc)
  • #1598 VB Net Framework project throws exception from command line tool
  • #1605 CoreRT / NativeAOT version (assignee: @adamsitnik)
  • #1607 Exporter approval tests has recently become unstable (assignee: @marcnet80)
  • #1613 EventPipeProfiler generating invalid SpeedScope files.
  • #1616 BenchmarkDotNet fail in WPF project with .NET 5.0-windows target
  • #1620 NullReferenceException in v0.12.1
  • #1623 Can I run Full Framework benchmarks without having a Console App? (assignee: @adamsitnik)
  • #1628 Installation uses legacy/archaic dotnetcore 2.1.503 (assignee: @adamsitnik)
  • #1629 Writing to Console.Error in benchmarked code causes a deadlock (assignee: @adamsitnik)
  • #1654 Update for 0.12.2
  • #1670 dotnet benchmark cli tool errors with .net5.0 assemblies (assignee: @adamsitnik)
  • #1673 Source code provider incorrectly handles multiline source code
  • #1685 Support for SingleFile && SelfContained apps
  • #1692 Bug running wasm benchmarks - Broken Pipe in writeline
  • #1693 Estimate Date for Supporting .NET 6 (assignee: @adamsitnik)

Merged pull requests (94)

Commits (111)

Contributors (37)

Thank you very much!

Additional details

Date: May 19, 2021

Milestone: v0.13.0 (List of commits)

NuGet Packages: