Table of Contents

BenchmarkDotNet v0.12.0

It's been several months since our last release, but we have been working hard and have some new features for you!

Highlights

  • Features and major improvements
    • Advanced multiple target frameworks support
      Now BenchmarkDotNet supports .NET Framework 4.8, .NET Core 3.1, and .NET Core 5.0. Also, we reworked our API that allows targeting several runtimes from the same config: the new API is more consistent, flexible, and powerful. For example, if you want to execute your benchmarking using .NET Framework 4.8 and .NET Core 3.1, you can use the SimpleJob(RuntimeMoniker.Net48), [SimpleJob(RuntimeMoniker.NetCoreApp31)] attributes or Job.Default.With(ClrRuntime.Net48), Job.Default.With(CoreRuntime.Core31) jobs in a manual config. You can find more details below. #1188, #1186, #1236
    • Official templates for BenchmarkDotNet-based projects
      With the help of the BenchmarkDotNet.Templates NuGet package, you can easily create new projects from the command line via dotnet new benchmark. This command has a lot of useful options, so you can customize your new project as you want. #1044
    • New NativeMemoryProfiler
      NativeMemoryProfiler measure the native memory traffic and adds the extra columns Allocated native memory and Native memory leak to the summary table. Internally, it uses EtwProfiler to profile the code using ETW. #457, #1131, #1208, #1214, #1218, #1219
    • New ThreadingDiagnoser
      ThreadingDiagnoser also adds two extra columns to the summary table: Completed Work Items (the number of work items that have been processed in ThreadPool per single operation) and Lock Contentions (the number of times there was contention upon trying to take a Monitor's lock per single operation). Internally, it uses new APIs exposed in .NET Core 3.0. #1154, #1227
    • Improved MemoryDiagnoser
      Now MemoryDiagnoser includes memory allocated by all threads that were live during benchmark execution: a new GC API was exposed in .NET Core 3.0 preview6+. It allows to get the number of allocated bytes for all threads. #1155, #1153, #723
    • LINQPad 6 support
      Now both LINQPad 5 and LINQPad 6 are supported! #1241, #1245
    • Fast documentation search
      We continue to improve the usability of our documentation. In this release, we improved the search experience in the documentation: now it works almost instantly with the help of Algolia engine! #1148, #1158
  • Minor summary and exporter improvements
    • Improved presentation of the current architecture in the environment information
      In the previous version of BenchmarkDotNet, the reports always contained "64bit" or "32bit" which did not tell if it was ARM or not. Now it prints the full architecture name (x64, x86, ARM, or ARM64). For example, instead of .NET Framework 4.8 (4.8.3815.0), 64bit RyuJIT you will get .NET Framework 4.8 (4.8.3815.0), X64 RyuJIT or .NET Framework 4.8 (4.8.3815.0), ARM64 RyuJIT. #1213
    • Simplified reports for Full .NET Framework version
      Previous version: .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3324.0. Current version: .NET Framework 4.7.2 (4.7.3362.0), 64bit RyuJIT. #1114, #1111
    • More reliable CPU info on Windows
      We added a workaround to for a bug in wmic that uses \r\r\n as a line separator. #1144, #1145
    • Better naming for generated plots
      When [RPlotExporter] is used, BenchmarkDotNet generates a lot of useful plots in the BenchmarkDotNet.Artifacts folder. The naming of the plot files was improved: benchmarks without Params doesn't include a double dash (--) in their names anymore. 1183, 1212
    • Better density plot precision The previous version of BenchmarkDotNet used the rule-of-thumb bandwidth selector in RPlotExporter density plots. It was fine for unimodal distributions, but sometimes it produced misleading plots for multimodal distributions. Now, RPlotExporter uses the Sheather&Jones bandwidth selector that significantly improves the presentation of the density plots for complex distributions. 58fde64
    • Better alignment in HtmlExporter
      Now BenchmarkDotNet aligns the content exported by HtmlExporter to the right. #1189 dfa074
    • Better precision calculation in SummaryTable
      4e9eb43
    • Better summary analysis
      BenchmarkDotNet warns the user when benchmark baseline value is too close to zero and the columns derived from BaselineCustomColumn cannot be computed. #1161, #600
    • Make log file datetime format 24-hour
      #1149
    • Improve AskUser prompt message
      The error messages will surround * by quotes on Linux and macOS. #1147
  • Minor API improvements
    • ED-PELT algorithm for changepoint detection is now available
      You can find details in this blog post. f89091
    • Improved OutlierMode API
      BenchmarkDotNet performs measurement postprocessing that may remove some of the outlier values (it can be useful to remove upper outliers that we get because of the natural CPU noise). In the previous version, naming for the OutlierMode values was pretty confusing: None/OnlyUpper/OnlyLower/All. Now, these values were renamed to DontRemove/RemoveUpper/RemoveLower/RemoveAll. For example, if you want to remove all the outliers, you can annotate your benchmark with the [Outliers(OutlierMode.RemoveAll)] attribute. The old names still exist (to make sure that the changes are backward compatible), but they are marked as obsolete, and they will be removed in the future versions of the library. #1199, 0e4b8e
    • Add the possibility to pass Config to BenchmarkSwitcher.RunAll and RunAllJoined
      #1194, ae23bd
    • Improved command line experience
      When user uses --packages $path, the $path will be sent to the dotnet build command as well. 1187
    • Extend the list of supported power plans. Now it supports "ultimate", "balanced", and "power saver" plans. #1132, #1139
    • Make it possible to not enforce power plan on Windows. 1578c5c
    • Guid support in benchmark arguments
      Now you can use Guid instances as benchmark arguments. 04ec20b
    • Make ArgumentsSource support IEnumerable<object[]> for benchmarks accepting a single argument to mimic MemberData behaviour.
      ec296dc
    • Make FullNameProvider public
      So it can be reused by the dotnet/performance repository. 6d71308
    • Extend Summary with LogFilePath #1135, 6e6559
    • Allow namespace filtering for InliningDiagnoser
      #1106, #1130
    • Option to configure MaxParameterColumnWidth
      #1269, 4ec888
  • Other improvements
    • Misc improvements in the documentation
      #1175, #1173, #1180, #1203, #1204, #1206, #1209, #1219, #1225, #1279
    • Copy PreserveCompilationContext MSBuild setting from the project that defines benchmarks
      #1152, 063d1a
    • Add System.Buffers.ArrayPoolEventSource to the list of default .NET Providers of EtwProfiler
      #1179, a106b1
    • Consume CoreRT from the new NuGet feed
      Because CoreRT no longer publishes to MyGet. #1129
  • Breaking changes:
    • The [ClrJob], [CoreJob] and [CoreRtJob] attributes got obsoleted and replaced by a [SimpleJob] which requires the user to provide target framework moniker in an explicit way. (See the "Advanced multiple target frameworks support" section for details.) #1188, #1182, #1115, #1056, #993,
    • The old InProcessToolchain is now obsolete. It's recommended to use InProcessEmitToolchain. If you want to use the old one on purpose, you have to use InProcessNoEmitToolchain. #1123
  • Bug fixes:
    • Invalid arg passing in StreamLogger constructor. The append arg was not passed to the StreamWriter .ctor. #1185
    • Improve the output path of .etl files produced by EtwProfiler. EtwProfiler was throwing NRE for users who were using [ClrJob] and [CoreJob] attributes. #1156, #1072
    • Flush custom loggers at the end of benchmark session. #1134
    • Make ids for tag columns unique - when using multiple TagColumns only one TagColumn was printed in the results. #1146

Advanced multiple target frameworks support

Now BenchmarkDotNet supports .NET Framework 4.8, .NET Core 3.1, and .NET Core 5.0. Also, we reworked our API that allows targeting several runtimes from the same config: the new API is more consistent, flexible, and powerful. For example, if you want to execute your benchmarking using .NET Framework 4.8 and .NET Core 3.1, you can use the SimpleJob(RuntimeMoniker.Net48), [SimpleJob(RuntimeMoniker.NetCoreApp31)] attributes or Job.Default.With(ClrRuntime.Net48), Job.Default.With(CoreRuntime.Core31) jobs in a manual config.

Now let's discuss how to use it in detail. If you want to test multiple frameworks, your project file MUST target all of them and you MUST install the corresponding SDKs:

<TargetFrameworks>netcoreapp3.0;netcoreapp2.1;net48</TargetFrameworks>

If you run your benchmarks without specifying any custom settings, BenchmarkDotNet is going to run the benchmarks using the same framework as the host process (it corresponds to RuntimeMoniker.HostProcess):

dotnet run -c Release -f netcoreapp2.1 # is going to run the benchmarks using .NET Core 2.1
dotnet run -c Release -f netcoreapp3.0 # is going to run the benchmarks using .NET Core 3.0
dotnet run -c Release -f net48         # is going to run the benchmarks using .NET 4.8
mono $pathToExe                        # is going to run the benchmarks using Mono from your PATH

To run the benchmarks for multiple runtimes with a single command from the command line, you need to specify the runtime moniker names via --runtimes|-r console argument:

dotnet run -c Release -f netcoreapp2.1 --runtimes netcoreapp2.1 netcoreapp3.0 # is going to run the benchmarks using .NET Core 2.1 and .NET Core 3.0
dotnet run -c Release -f netcoreapp2.1 --runtimes netcoreapp2.1 net48         # is going to run the benchmarks using .NET Core 2.1 and .NET 4.8

What is going to happen if you provide multiple Full .NET Framework monikers? Let's say:

dotnet run -c Release -f net461 net472 net48

Full .NET Framework always runs every .NET executable using the latest .NET Framework available on a given machine. If you try to run the benchmarks for a few .NET TFMs, they are all going to be executed using the latest .NET Framework from your machine. The only difference is that they are all going to have different features enabled depending on the target version they were compiled for. You can read more about this here and here. This is .NET Framework behavior which can not be controlled by BenchmarkDotNet or any other tool.

Note: Console arguments support works only if you pass the args to BenchmarkSwitcher:

class Program
{
    static void Main(string[] args) 
        => BenchmarkSwitcher
            .FromAssembly(typeof(Program).Assembly)
            .Run(args); // crucial to make it work
}

You can achieve the same thing using [SimpleJobAttribute]:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;

namespace BenchmarkDotNet.Samples
{
    [SimpleJob(RuntimeMoniker.Net48)]
    [SimpleJob(RuntimeMoniker.Mono)]
    [SimpleJob(RuntimeMoniker.NetCoreApp21)]
    [SimpleJob(RuntimeMoniker.NetCoreApp30)]
    public class TheClassWithBenchmarks

Or using a custom config:

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Environments;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace BenchmarkDotNet.Samples
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = DefaultConfig.Instance
                .With(Job.Default.With(CoreRuntime.Core21))
                .With(Job.Default.With(CoreRuntime.Core30))
                .With(Job.Default.With(ClrRuntime.Net48))
                .With(Job.Default.With(MonoRuntime.Default));

            BenchmarkSwitcher
                .FromAssembly(typeof(Program).Assembly)
                .Run(args, config);
        }
    }
}

The recommended way of running the benchmarks for multiple runtimes is to use the --runtimes console line argument. By using the console line argument, you don't need to edit the source code anytime you want to change the list of runtimes. Moreover, if you share the source code of the benchmark, other people can run it even if they don't have the exact same framework version installed.

Official templates for BenchmarkDotNet-based projects

Since v0.12.0, BenchmarkDotNet provides project templates to setup your benchmarks easily. The template exists for each major .NET language (C#, F# and VB) with equivalent features and structure. The templates require the .NET Core SDK. Once installed, run the following command to install the templates:

dotnet new -i BenchmarkDotNet.Templates

If you want to uninstall all BenchmarkDotNet templates:

dotnet new -u BenchmarkDotNet.Templates

The template is a NuGet package distributed over nuget.org: BenchmarkDotNet.Templates. To create a new C# benchmark library project from the template, run:

dotnet new benchmark

If you'd like to create F# or VB project, you can specify project language with -lang option:

dotnet new benchmark -lang F#
dotnet new benchmark -lang VB

The template projects have five additional options - all of them are optional. By default, a class library project targeting netstandard2.0 is created. You can specify -f or --frameworks to change target to one or more frameworks:

dotnet new benchmark -f netstandard2.0;net472

The option --console-app creates a console app project targeting netcoreapp3.0 with an entry point:

dotnet new benchmark --console-app

This lets you run the benchmarks from a console (dotnet run) or from your favorite IDE. The option -f or --frameworks will be ignored when --console-app is set. The option -b or --benchmarkName sets the name of the benchmark class:

dotnet new benchmark -b Md5VsSha256

BenchmarkDotNet lets you create a dedicated configuration class (see Configs) to customize the execution of your benchmarks. To create a benchmark project with a configuration class, use the option -c or --config:

dotnet new benchmark -c

The option --no-restore if specified, skips the automatic NuGet restore after the project is created:

dotnet new benchmark --no-restore

Use the -h or --help option to display all possible arguments with a description and the default values:

dotnet new benchmark --help

The version of the template NuGet package is synced with the BenchmarkDotNet package. For instance, the template version 0.12.0 is referencing BenchmarkDotnet 0.12.0 - there is no floating version behavior. For more info about the dotnet new CLI, please read the documentation.

New NativeMemoryProfiler

NativeMemoryProfiler measure the native memory traffic and adds the extra columns Allocated native memory and Native memory leak to the summary table. Internally, it uses EtwProfiler to profile the code using ETW.

Consider the following benchmark:

[ShortRunJob]
[NativeMemoryProfiler]
[MemoryDiagnoser]
public class IntroNativeMemory
{
    [Benchmark]
    public void BitmapWithLeaks()
    {
        var flag = new Bitmap(200, 100);
        var graphics = Graphics.FromImage(flag);
        var blackPen = new Pen(Color.Black, 3);
        graphics.DrawLine(blackPen, 100, 100, 500, 100);
    }

    [Benchmark]
    public void Bitmap()
    {
        using (var flag = new Bitmap(200, 100))
        {
            using (var graphics = Graphics.FromImage(flag))
            {
                using (var blackPen = new Pen(Color.Black, 3))
                {
                    graphics.DrawLine(blackPen, 100, 100, 500, 100);
                }
            }
        }
    }

    private const int Size = 20; // Greater value could cause System.OutOfMemoryException for test with memory leaks.
    private int ArraySize = Size * Marshal.SizeOf(typeof(int));

    [Benchmark]
    public unsafe void AllocHGlobal()
    {
        IntPtr unmanagedHandle = Marshal.AllocHGlobal(ArraySize);
        Span<byte> unmanaged = new Span<byte>(unmanagedHandle.ToPointer(), ArraySize);
        Marshal.FreeHGlobal(unmanagedHandle);
    }

    [Benchmark]
    public unsafe void AllocHGlobalWithLeaks()
    {
        IntPtr unmanagedHandle = Marshal.AllocHGlobal(ArraySize);
        Span<byte> unmanaged = new Span<byte>(unmanagedHandle.ToPointer(), ArraySize);
    }
}

It will produce the summary table like this one:

Method Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated Allocated native memory Native memory leak
BitmapWithLeaks 73,456.43 ns 22,498.10 ns 1,233.197 ns - - - 177 B 13183 B 11615 B
Bitmap 91,590.08 ns 101,468.12 ns 5,561.810 ns - - - 180 B 12624 B -
AllocHGlobal 79.91 ns 43.93 ns 2.408 ns - - - - 80 B -
AllocHGlobalWithLeaks 103.50 ns 153.21 ns 8.398 ns - - - - 80 B 80 B

As you can see, we have two additional columns Allocated native memory and Native memory leak that contain some very useful numbers!

New ThreadingDiagnoser

ThreadingDiagnoser also adds two extra columns to the summary table:

  • Completed Work Items: The number of work items that have been processed in ThreadPool (per single operation)
  • Lock Contentions: The number of times there was contention upon trying to take a Monitor's lock (per single operation)

Internally, it uses new APIs exposed in .NET Core 3.0.

It can be activated with the help of the [ThreadingDiagnoser] attribute:

[ThreadingDiagnoser]
public class IntroThreadingDiagnoser
{
    [Benchmark]
    public void CompleteOneWorkItem()
    {
        ManualResetEvent done = new ManualResetEvent(initialState: false);
        ThreadPool.QueueUserWorkItem(m => (m as ManualResetEvent).Set(), done);
        done.WaitOne();
    }
}

The above example will print a summary table like this one:

Method Mean StdDev Median Completed Work Items Lock Contentions
CompleteOneWorkItem 8,073.5519 ns 69.7261 ns 8,111.6074 ns 1.0000 -

LINQPad 6 support

Now both LINQPad 5 and LINQPad 6 are supported:

We continue to improve the usability of our documentation. In this release, we improved the search experience in the documentation: now it works almost instantly with the help of Algolia engine! That's how it looks:

Milestone details

In the v0.12.0 scope, 44 issues were resolved and 56 pull requests were merged. This release includes 110 commits by 25 contributors.

Resolved issues (44)

  • #198 [Feature request] No logger for benchmark run? (assignee: @CodeTherapist)
  • #311 How to debug benchmarks that fail with exception on file system access operations (assignee: @adamsitnik)
  • #457 Track Native Memory Allocations and more informations with our ETW Memory Diagnoser
  • #600 Scaling issue
  • #723 MemoryDiagnoser should include memory allocated by all Threads that were live during benchmark execution (assignee: @adamsitnik)
  • #995 Running benchmark fails when targeting netcoreapp2.2 (assignee: @adamsitnik)
  • #1028 Add new template for "dotnet new benchmark" (assignee: @CodeTherapist)
  • #1072 EtwProfiler exports trace file only for a single runtime when Runtimes are controlled via attributes (assignee: @adamsitnik)
  • #1106 Allow user defined namespace filter for InliningDiagnoser
  • #1111 Change the format of printed Full .NET Framework Version (assignee: @adamsitnik)
  • #1115 Running using dotnet benchmark uses wrong core runtime
  • #1132 The power management feature extension
  • #1134 StreamLogger is not properly flushed on shutdown (assignee: @AndreyAkinshin)
  • #1135 The default file logger and summary title are out of sync (assignee: @adamsitnik)
  • #1137 [Discussion] Improve search experience in the documentation
  • #1144 Incorrect CPU info for .NET Core applications
  • #1146 Only the first of multiple custom columns is included in the summary table (assignee: @AndreyAkinshin)
  • #1147 Update benchmark switcher instructions to work on Linux (assignee: @AndreyAkinshin)
  • #1149 Ambiguous hour component in log file name timestamp (assignee: @AndreyAkinshin)
  • #1152 Failed to test Roslyn. (assignee: @adamsitnik)
  • #1153 Use GC.GetTotalAllocatedBytes when available in MemoryDiagnoser (assignee: @adamsitnik)
  • #1154 Add a ConcurrencyDiagnoser? (assignee: @adamsitnik)
  • #1156 Crash when BenchmarkDotNet.Diagnostics.Windows.Session.GetFilePath throws NRE (assignee: @adamsitnik)
  • #1158 🔍 Improving search on docs with Algolia's DocSearch
  • #1162 Incorrect value of BenchmarkDotNet.Toolchains.DotNetCli.NetCoreAppSettings.Default
  • #1168 Consider using default value instead of hardcoded '-' in MetricColumn.GetValue()
  • #1179 Add System.Buffers.ArrayPoolEventSource to the list of default .NET Providers of EtwProfiler (assignee: @adamsitnik)
  • #1181 Log shows a wrong name for plot images
  • #1182 Benchingmarking .NET 4.8 Causes Errors
  • #1183 Plots of benchmarks without params have a double dash (--) in the name
  • #1186 Add support for --runtimes net48 (assignee: @adamsitnik)
  • #1187 When user uses --packages $path, the $path should be sent to dotnet build command as well (assignee: @adamsitnik)
  • #1194 RunAll with ToolChains (assignee: @adamsitnik)
  • #1195 LatestCoreRtVersionIsSupported fails on Mac Os
  • #1202 BenchmarkDotNet Not Recognizing CPU
  • #1220 [Docs] RScript / R_HOME setup
  • #1235 NativeMemoryProfiler exception (assignee: @WojciechNagorski)
  • #1236 Rework new API for target runtimes (assignee: @adamsitnik)
  • #1241 Can BenchmarkDotNet be enabled for LINQPad 6? (assignee: @adamsitnik)
  • #1269 Unable to show full param string in the report (assignee: @adamsitnik)
  • #1280 Improvement in memory statistics (assignee: @WojciechNagorski)
  • #1285 Issue with .Net Core version 3.0
  • #1289 How to config to not save .log files?
  • #1291 MemoryDiagnoser reports weird results for .NET Core 3.0

Merged pull requests (56)

Commits (110)

Contributors (25)

Thank you very much!

Additional details

Date: October 24, 2019

Milestone: v0.12.0 (List of commits)

NuGet Packages: