Table of Contents

BenchmarkDotNet v0.11.0

This is one of the biggest releases of BenchmarkDotNet ever. There are so many improvements. We have new documentation, many performance improvements, Job Mutators, better user experience, correct Ctrl+C handling, better generic benchmarks support, more scenarios for passing arguments to benchmarks, awesome support of console arguments, unicode support, LLVM support in MonoDisassembler, and many-many other improvements and bug fixes!

A big part of the features and bug fixes were implemented to meet the enterprise requirements of Microsoft to make it possible to port CoreCLR, CoreFX, and CoreFXLab to BenchmarkDotNet.

The release would not be possible without many contributions from amazing community members. This release is a combined effort. We build BenchmarkDotNet together to make benchmarking .NET code easy and available to everyone for free!

New documentation

We have many improvements in our documentation! The new docs include:

  • DocFX under the hood
  • Detailed changelogs which includes all commits, merged pull requests and resolved issues
  • API references
  • Code samples for main features: we generate it automatically based on the BenchmarkDotNet.Samples project; it means that all samples can always be compiled (no more samples with outdated API)
  • Better UI
  • Documentation versioning: now it's possible to look at the documentation for recent BenchmarkDotNet versions

Performance improvements

BenchmarkDotNet needs to be capable of running few thousands of CoreFX and CoreCLR benchmarks in an acceptable amount of time. The code itself was already optimized so we needed architectural and design changes to meet this requirement.

Generate one executable per runtime settings

To ensure that the side effects of one benchmark run does not affect another benchmark run BenchmarkDotNet generates, builds and runs every benchmark in a dedicated process. So far we were generating and building one executable per benchmark, now we generate and build one executable per runtime settings. So if you want to run ten thousands of benchmarks for .NET Core 2.1 we are going to generate and build single executable, not ten thousand. If you target multiple runtimes the build is going to be executed in parallel. Moreover, if one of the parallel builds fail it's going to be repeated in a sequential way.

Previously the time to generate and build 650 benchmarks from our Samples project was one hour. Now it's something around 13 seconds which means 276 X improvement for this particular scenario. You can see the changes here.

Don't execute long operations more than once per iteration

BenchmarkDotNet was designed to allow for very accurate and stable micro-benchmarking. One of the techniques that we use is manual loop unrolling. In practice, it meant that for every iteration we were executing the benchmark at least 16 times (the default UnrollFactor value). It was of course not desired for the very time-consuming benchmarks.

So far this feature was always enabled by default and users would need to configure UnrollFactor=1 to disable it. Now BenchmarkDotNet is going to discover such scenario and don't perform manual loop unrolling for the very time-consuming benchmarks. BenchmarkDotNet uses Job.IterationTime setting (the default is 0.5s) in the Pilot Experiment stage to determine how many times given benchmark should be executed per iteration.

Example:

public class Program
{
    static void Main() => BenchmarkRunner.Run<Program>();

    [Benchmark]
    public void Sleep1s() => Thread.Sleep(TimeSpan.FromSeconds(1));
}

Time to run with the previous version: 374 seconds. With 0.11.0 it's 27 seconds which gives us almost 14 X improvement. A good example of benchmarks that are going to benefit from this change are computer game benchmarks and ML.NET benchmarks. You can see the changes here and here.

Exposing more configuration settings

The default settings were configured to work well with every scenario. Before running the benchmark, BenchmarkDotNet does not know anything about it. This is why it performs many warmup iterations before running the benchmarks.

When you author benchmarks and run them many times you can come up with custom settings that produce similar results but in a shorter manner of time. To allow you to do that we have exposed:

  • Job.MinIterationCount (default value is 15)
  • Job.MaxIterationCount (default value is 100)
  • Job.MinWarmupIterationCount (default value is 6)
  • Job.MaxWarmupIterationCount (default value is 50)

User Experience

One of the biggest success factors of BenchmarkDotNet is a great user experience. The tool just works as expected and makes your life easy. We want to make it even better!

.NET Standard 2.0

We have ported BenchmarkDotNet to .NET Standard 2.0 and thanks to that we were able to not only simplify our code and build process but also merge BenchmarkDotNet.Core.dll and BenchmarkDotNet.Toolchains.Roslyn.dll into BenchmarkDotNet.dll. We still support .NET 4.6 but we have dropped .NET Core 1.1 support. More information and full discussion can be found here.

Note: Our BenchmarkDotNet.Diagnostics.Windows package which uses EventTrace to implement ETW-based diagnosers was also ported to .NET Standard 2.0 and you can now use all the ETW diagnosers with .NET Core on Windows. We plan to add EventPipe support and make this page fully cross-platform and Unix compatible soon.

Using complex types as benchmark arguments

So far we have required the users to implement IParam interface to make the custom complex types work as benchmark arguments/parameters. This has changed, now the users can use any complex types as arguments and it will just work (more).

public class Program
{
    static void Main(string[] args) => BenchmarkRunner.Run<Program>();

    public IEnumerable<object> Arguments()
    {
        yield return new Point2D(10, 200);
    }

    [Benchmark]
    [ArgumentsSource(nameof(Arguments))]
    public int WithArgument(Point2D point) => point.X + point.Y;
}

public class Point2D
{
    public int X, Y;

    public Point2D(int x, int y)
    {
        X = x;
        Y = y;
    }

    public override string ToString() => $"[{X},{Y}]";
}

Note: If you want to control what will be displayed in the summary you should override ToString.

If IterationSetup is provided run benchmark once per iteration

When Stephen Toub says that something is buggy, it most probably is. BenchmarkDotNet performs multiple invocations of benchmark per every iteration. When we have exposed the [IterationSetup] attribute many users were expecting that the IterationSetup is going to be invoked before every benchmark execution.

It was invoked before every iteration, and iteration was more than one benchmark call if the user did not configure that explicitly. We have changed that and now if you provide an [IterationSetup] method it is going to be executed before every iteration and iteration will invoke the benchmark just once.

public class Test
{
    public static void Main() => BenchmarkRunner.Run<Test>();

    [IterationSetup]
    public void MySetup() => Console.WriteLine("MySetup");

    [Benchmark]
    public void MyBenchmark() => Console.WriteLine("MyBenchmark");
}

Before:

MySetup
MyBenchmark
MyBenchmark
MyBenchmark
MyBenchmark
(...)

After:

MySetup
MyBenchmark
MySetup
MyBenchmark
MySetup
MyBenchmark
(...)

Note: If you want to configure how many times benchmark should be invoked per iteration you can use the new [InvocationCountAttribute].

Job Mutators

Job represents a set of settings to run the benchmarks. We run every benchmark for every job defined by the user. The problem was that so far many jobs were just added to the config instead of being merged with other jobs.

An example:

[ClrJob, CoreJob]
[GcServer(true)]
public class MyBenchmarkClass

Resulted in 3 jobs and 3 benchmark executions: ClrJob, CoreJob and GcServer(true) for current runtime.

Now all Jobs and their corresponding attributes marked as mutators are going to be applied to other jobs, not just added to the config. So in this particular scenario, the benchmarks from MyBenchmarkClass are going to be executed for .NET with Server GC enabled and .NET Core with Server GC enabled.

Mutators are great when you want to have a single, global config for all benchmarks and apply given settings only to selected types. You can find out more about mutators here.

Ctrl+C

When the user:

  • presses Ctrl+C
  • presses Ctrl+Break
  • logs off
  • closes console window

We are now going to close any existing ETW session created by BenchmarkDotNet and restore console colors (read more).

Handle OutOfMemoryException more gracefully

When our benchmark hits OutOfMemoryException we print some nice explanation:

public class Program
{
    static void Main(string[] args) => BenchmarkRunner.Run<Program>();

    private List<object> list = new List<object>();

    [Benchmark]
    public void AntiPattern() => list.Add(new int[int.MaxValue / 2]);
}

OutOfMemoryException!
BenchmarkDotNet continues to run additional iterations until desired accuracy level is achieved. It's possible only if the benchmark method doesn't have any side-effects.
If your benchmark allocates memory and keeps it alive, you are creating a memory leak.
You should redesign your benchmark and remove the side-effects. You can use `OperationsPerInvoke`, `IterationSetup` and `IterationCleanup` to do that.

Trimming long strings

We used to display the values "as is" which was bad for long strings. Now the values are trimmed (more).

public class Long
{
    [Params("text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7")]
    public string Text;

    [Benchmark]
    public int HashCode() => Text.GetHashCode();
}
Method Text
HashCode text/(...)q=0.7 [86]

More features

Generic benchmarks

BenchmarkDotNet supports generic benchmarks, all you need to do is to tell it which types should be used as generic arguments (read more).

[GenericTypeArguments(typeof(int))]
[GenericTypeArguments(typeof(char))]
public class IntroGenericTypeArguments<T>
{
    [Benchmark] public T Create() => Activator.CreateInstance<T>();
}

Arguments

We now support more scenarios for passing arguments to benchmarks:

  • passing arguments to asynchronous benchmarks (more)
  • passing generic types
  • passing arguments by reference
  • passing jagged arrays (more)
  • types with implicit cast operator to stack only types can be passed as given stack-only types to Benchmarks (more)

Example:

public class WithStringToReadOnlySpan
{
    [Benchmark]
    [Arguments("some string")]
    public void AcceptsReadOnlySpan(ReadOnlySpan<char> notString)
}

Console Arguments

BenchmarkSwitcher supports various console arguments (PR), to make it work you need to pass the args to switcher:

class Program
{
    static void Main(string[] args) 
        => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}

Note: to get the most up-to-date info about supported console arguments run the benchmarks with --help.

Filter

The --filter or just -f allows you to filter the benchmarks by their full name (namespace.typeName.methodName) using glob patterns.

Examples:

  1. Run all benchmarks from System.Memory namespace: -f System.Memory*
  2. Run all benchmarks: -f *
  3. Run all benchmarks from ClassA and ClassB -f *ClassA* *ClassB*

Note: If you would like to join all the results into a single summary, you need to use --join.

Categories

You can also filter the benchmarks by categories:

  • --anyCategories - runs all benchmarks that belong to any of the provided categories
  • --allCategories- runs all benchmarks that belong to all provided categories

Diagnosers

  • -m, --memory - enables MemoryDiagnoser and prints memory statistics
  • -d, --disassm- enables DisassemblyDiagnoser and exports diassembly of benchmarked code

Runtimes

The --runtimes or just -r allows you to run the benchmarks for selected Runtimes. Available options are: Clr, Mono, Core and CoreRT.

Example: run the benchmarks for .NET and .NET Core:

dotnet run -c Release -- --runtimes clr core

More arguments

  • -j, --job (Default: Default) Dry/Short/Medium/Long or Default
  • -e, --exporters GitHub/StackOverflow/RPlot/CSV/JSON/HTML/XML
  • -i, --inProcess (Default: false) Run benchmarks in Process
  • -a, --artifacts Valid path to accessible directory
  • --outliers (Default: OnlyUpper) None/OnlyUpper/OnlyLower/All
  • --affinity Affinity mask to set for the benchmark process
  • --allStats (Default: false) Displays all statistics (min, max & more)
  • --attribute Run all methods with given attribute (applied to class or method)

Other small improvements

  • Unicode support: now you can enable support of Unicode symbols like μ or ± with [EncodingAttribute.Unicode], an example: BenchmarkDotNet.Samples.IntroEncoding (see #735)
  • Better benchmark validation (see #693, #737)
  • Improve .NET Framework version detection: now we support .NET Framework 4.7.2 (see #743)
  • OutlierModes: now it's possible to control how to process outliers, an example BenchmarkDotNet.Samples.IntroOutliers (see #766)
  • LLVM support in MonoDisassembler (see a7426e)
  • Grand API renaming we try not to change public API, but sometimes it's necessary because we want to get a consistent and understandable API in v1.0.0. (see #787)
  • Many-many small improvements and bug fixes

Milestone details

In the v0.11.0 scope, 65 issues were resolved and 34 pull requests were merged. This release includes 214 commits by 11 contributors.

Resolved issues (65)

  • #136 Fastcheck for correctness of benchmark implementations
  • #175 Add .NET Core support for Diagnostics package (assignee: @adamsitnik)
  • #368 Memory leak and crash with [Setup] (assignee: @adamsitnik)
  • #420 Make BenchmarkDotNet.Core runtime independent (assignee: @adamsitnik)
  • #464 Iteration setup / cleanup should not be called for Idle() (assignee: @adamsitnik)
  • #484 Broken HTTPS on site (assignee: @jongalloway)
  • #487 Please consider using 'µs' instead of 'us'
  • #551 List of structs and OutOfMemoryException
  • #583 BenchmarkDotNet.Samples refactoring (assignee: @AndreyAkinshin)
  • #586 IParam interface improvement (assignee: @adamsitnik)
  • #638 Config with ryujit but it doesnt actually use ryujit? (assignee: @morgan-kn)
  • #649 Searching docs leads to 404 page (assignee: @AndreyAkinshin)
  • #665 Handle OutOfMemoryException more gracefully (assignee: @adamsitnik)
  • #671 Why does BenchmarkRunner generate an isolated project per each benchmark method/job/params? (assignee: @adamsitnik)
  • #698 Port to .NET Standard 2.0, drop .NET Core 1.1 support (assignee: @adamsitnik)
  • #699 Generate one executable per runtime settings (assignee: @adamsitnik)
  • #700 Improve local CoreCLR support (assignee: @adamsitnik)
  • #701 Extend exported json file with FullName using xunit naming convention for integration purpose (assignee: @adamsitnik)
  • #710 Use DocFX as a documentation generator (assignee: @AndreyAkinshin)
  • #712 [Params] with arrays as params throws System.Reflection.TargetInvocationException (assignee: @adamsitnik)
  • #713 How to specify the invocation/launch count per type when using Config for multiple runtimes? (assignee: @adamsitnik)
  • #718 CoreRT support (assignee: @adamsitnik)
  • #719 If fail to build in Parallel due to file access issues, try to build sequentially (assignee: @adamsitnik)
  • #720 Add SummaryOrderPolicy.Declared
  • #724 Allocated Memory results are not scaled with OperationPerInvoke (assignee: @adamsitnik)
  • #726 Improve building guideline
  • #729 Handle Ctrl+C/Break (assignee: @adamsitnik)
  • #730 IterationSetup is not running before each benchmark invocation (assignee: @adamsitnik)
  • #733 IOException when running in OneDrive Folder (assignee: @adamsitnik)
  • #734 Handle missing Mono runtime more gracefully (assignee: @adamsitnik)
  • #736 Reduce number of initial pilot ops to 1 or make it configurable (assignee: @adamsitnik)
  • #738 Params string containing characters like quotes is not being escaped properly (assignee: @adamsitnik)
  • #741 Give users nice warning when T in generic benchmark is not public
  • #745 It should be possible to specify the generic arguments by using attributes
  • #747 Better docs that explain what is target/launch/iteration/invocation count (assignee: @adamsitnik)
  • #748 Very long string params/arguments should be trimmed (assignee: @adamsitnik)
  • #749 WithId(…) is ignored unless it’s at the end of the fluent calls chain. (assignee: @adamsitnik)
  • #763 Make MaxIterationCount configurable, keep current value as default (assignee: @adamsitnik)
  • #765 Add .NET Core 2.2 support (assignee: @adamsitnik)
  • #769 ArgumentsSource does not support Jagged Arrays (assignee: @adamsitnik)
  • #774 Make it possible to use Span and other ByRefLike types with implicit cast operators as benchmark argument (assignee: @adamsitnik)
  • #778 CS0104: 'Job' is an ambiguous reference between 'BenchmarkDotNet.Jobs.Job' and 'Nest.Job' (assignee: @adamsitnik)
  • #779 StackOnlyTypesWithImplicitCastOperatorAreSupportedAsArguments doesn't work on .NET Core 2.0 (assignee: @adamsitnik)
  • #787 Grand renaming
  • #793 job=core for BenchmarkSwitcher (assignee: @adamsitnik)
  • #794 Don't exclude allocation quantum side effects for .NET Core 2.0+ (assignee: @adamsitnik)
  • #795 Broken BenchmarkSwitcher (assignee: @adamsitnik)
  • #797 Allocated is not divided by OperationsPerInvoke (assignee: @adamsitnik)
  • #802 AdaptiveHistogramBuilder.BuildWithFixedBinSize error when running benchmarks (assignee: @AndreyAkinshin)
  • #804 What is the point of BuildScriptFilePath ? (assignee: @adamsitnik)
  • #809 Make it possible to configure Min and Max Warmup Iteration Count (assignee: @adamsitnik)
  • #810 handle new *Ansi events to make Inlining and TailCall Diagnosers work with .NET Core 2.2 (assignee: @adamsitnik)
  • #811 Question/Suggestion is GcStats forcing a GC.Collect when it doesn't need to (assignee: @adamsitnik)
  • #812 When will the next release be available on NuGet? (assignee: @adamsitnik)
  • #813 Problems with MemoryDiagnoserTests on Mono and .NET Core 2.0 (assignee: @adamsitnik)
  • #814 For type arguments we should display simple, not-trimmed name (assignee: @adamsitnik)
  • #816 BenchmarkDotNet.Autogenerated.csproj is not working on .NET Core 2.1 (assignee: @adamsitnik)
  • #817 Autogenerated project is missing dependencies (assignee: @adamsitnik)
  • #818 Arguments should be passed to asynchronous benchmarks (assignee: @adamsitnik)
  • #820 set DOTNET_MULTILEVEL_LOOKUP=0 when custom dotnet cli path is provided (assignee: @adamsitnik)
  • #821 ArgumentsAttribute causes an error when used with a string containing quotes (assignee: @adamsitnik)
  • #823 Allow to set multiple Setup/Cleanup targets without string concatenation (assignee: @adamsitnik)
  • #827 An easy way to run a specific benchmark class via command line (assignee: @adamsitnik)
  • #829 Error message for wrong command line filter (assignee: @adamsitnik)
  • #832 Compilation Error CS0119 with ParamsSource (assignee: @adamsitnik)

Merged pull requests (34)

Commits (214)

Contributors (11)

Thank you very much!

Additional details

Date: July 23, 2018

Milestone: v0.11.0 (List of commits)

NuGet Packages: