Table of Contents

Full ChangeLog

BenchmarkDotNet v0.14.0

Highlights

  • Introduce BenchmarkDotNet.Diagnostics.dotMemory #2549: memory allocation profile of your benchmarks using dotMemory, see BenchmarkDotNet.Samples.IntroDotMemoryDiagnoser
  • Introduce BenchmarkDotNet.Exporters.Plotting #2560: plotting via ScottPlot (initial version)
  • Multiple bugfixes
  • The default build toolchains have been updated to pass IntermediateOutputPath, OutputPath, and OutDir properties to the dotnet build command. This change forces all build outputs to be placed in a new directory generated by BenchmarkDotNet, and fixes many issues that have been reported with builds. You can also access these paths in your own .csproj and .props from those properties if you need to copy custom files to the output.

Bug fixes

  • Fixed multiple build-related bugs including passing MsBuildArguments and .Net 8's UseArtifactsOutput.

Breaking Changes

  • DotNetCliBuilder removed retryFailedBuildWithNoDeps constructor option.
  • DotNetCliCommand removed RetryFailedBuildWithNoDeps property and BuildNoRestoreNoDependencies() and PublishNoBuildAndNoRestore() methods (replaced with PublishNoRestore()).

Milestone details

In the v0.14.0 scope, 17 issues were resolved and 28 pull requests were merged. This release includes 55 commits by 16 contributors.

Resolved issues (17)

  • #946 Copy to $(OutDir) does not copy to benchmarkdotnet output folder (assignee: @timcassell)
  • #1270 Docs for SummaryStyle
  • #1377 Forward msbuild properties to project reference (assignee: @timcassell)
  • #1457 Directory.Build.props hijacking generated csproj output path (assignee: @timcassell)
  • #1595 ValueTask-returning benchmarks are not being called correctly (assignee: @timcassell)
  • #1795 'BenchmarkDotNet.Autogenerated\project.assets.json' not found. (assignee: @timcassell)
  • #2253 TaskbarProgressCom is not supported when benchmarking Native AOT
  • #2310 Bug: Getting an error loading BenchmarkDotNet.Diagnostics.Windows.dll for a benchmark that is not windows specific
  • #2366 default(object) takes more time than expected in net7.0
  • #2466 Add support for true (assignee: @timcassell)
  • #2477 MSBuild Params Not Applied to Benchamrk (assignee: @timcassell)
  • #2524 Reference assemblies are searched in the wrong location
  • #2540 Unable to use wasm or monoaotllvm as runtime (assignee: @LoopedBard3)
  • #2542 InvalidMeasurementUnitException when using --statisticalTest argument when comparing private builds (assignee: @AndreyAkinshin)
  • #2545 HideColumns + custom corerun = incorrect table markdown
  • #2554 Question about visibility of DotTraceDiagnoser's second constructor (assignee: @AndreyAkinshin)
  • #2570 BenchmarkDotNet test adapter not running tests with dotnet test

Merged pull requests (28)

Commits (55)

Contributors (16)

Thank you very much!

Additional details

Date: August 06, 2024

Milestone: v0.14.0 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.12

Highlights

The biggest highlight of this release if our new VSTest Adapter, which allows to run benchmarks as unit tests in your favorite IDE! The detailed guide can be found here.

This release also includes to a minor bug fix that caused incorrect job id generation: fixed job id generation (#2491).

Also, the target framework in the BenchmarkDotNet templates was bumped to .NET 8.0.

Milestone details

In the v0.13.12 scope, 3 issues were resolved and 4 pull requests were merged. This release includes 12 commits by 4 contributors.

Resolved issues (3)

  • #2473 Custom SimpleJob Id ignored when the MinIterationTime attribute is used.
  • #2492 Packing of BenchmarkDotNet.TestAdapter is broken (assignee: @AndreyAkinshin)
  • #2494 Rider and R# do not distinguish parametrized TestAdapter cases (assignee: @AndreyAkinshin)

Merged pull requests (4)

Commits (12)

Contributors (4)

Thank you very much!

Additional details

Date: January 05, 2024

Milestone: v0.13.12 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.11

Highlights

Small improvements.

Milestone details

In the v0.13.11 scope, 4 issues were resolved and 8 pull requests were merged. This release includes 28 commits by 7 contributors.

Resolved issues (4)

  • #2060 NativeAOT benchmark started from .Net Framework host doesn't have all intrinsics enabled (assignee: @adamsitnik)
  • #2233 Q: Include hardware counters in XML output (assignee: @nazulg)
  • #2388 Include AVX512 in listed HardwareIntrinsics
  • #2463 Bug. Native AOT .NET 7.0 doesn't work. System.NotSupportedException: X86Serialize (assignee: @adamsitnik)

Merged pull requests (8)

Commits (28)

Contributors (7)

Thank you very much!

Additional details

Date: December 06, 2023

Milestone: v0.13.11 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.10

Highlights

Initial support of .NET 9 and minor bug fixes.

Milestone details

In the v0.13.10 scope, 2 issues were resolved and 3 pull requests were merged. This release includes 10 commits by 4 contributors.

Resolved issues (2)

Merged pull requests (3)

Commits (10)

Contributors (4)

Thank you very much!

Additional details

Date: November 01, 2023

Milestone: v0.13.10 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.9

Highlights

This release contains bug fixes.

Milestone details

In the v0.13.9 scope, 3 issues were resolved and 7 pull requests were merged. This release includes 26 commits by 5 contributors.

Resolved issues (3)

Merged pull requests (7)

Commits (26)

Contributors (5)

Thank you very much!

Additional details

Date: October 05, 2023

Milestone: v0.13.9 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.8

Highlights

This release contains important bug fixes.

Milestone details

In the v0.13.8 scope, 5 issues were resolved and 9 pull requests were merged. This release includes 15 commits by 8 contributors.

Resolved issues (5)

  • #1995 Left Justification option for summary table columns
  • #2394 Multiple MarkdownExporters not possible (assignee: @bstordrup)
  • #2399 Custom MarkdownExporter (assignee: @nietras)
  • #2400 Should StreamLogger be TextWriterLogger? (assignee: @nietras)
  • #2405 0.13.7 BenchmarkDotNet.Autogenerated project fails build (assignee: @timcassell)

Merged pull requests (9)

Commits (15)

Contributors (8)

Thank you very much!

Additional details

Date: September 08, 2023

Milestone: v0.13.8 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.7

Highlights

This release contains important bug fixes listed below.

Milestone details

In the v0.13.7 scope, 12 issues were resolved and 13 pull requests were merged. This release includes 38 commits by 9 contributors.

Resolved issues (12)

  • #1773 Ability to recompile for each target runtime
  • #2311 Detect when MonoAOT compilation fails and abort the benchmark run
  • #2346 Can't use latest version of BenchmarkDotNet
  • #2356 Add tests for expected results (assignee: @timcassell)
  • #2364 Benchmark project using paket dependency manager fails to build auto-generated projects (assignee: @timcassell)
  • #2369 Unable to use variables for PackageReferences after #2347
  • #2373 Regression in parsing arguments with spaces (assignee: @kant2002)
  • #2377 Multiple nuget warnings/errors after moving to 0.13.6 breaking the build
  • #2378 BenchmarkDotNet requires dotnet cli to be installed
  • #2382 -p perf doesn't seem to work right with locally built runtime (assignee: @adamsitnik)
  • #2389 PlatformNotSupportedException thrown on Android in ConsoleTitler
  • #2391 Benchmark fails build with imported project

Merged pull requests (13)

Commits (38)

Contributors (9)

Thank you very much!

Additional details

Date: August 04, 2023

Milestone: v0.13.7 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.6

Highlights

  • New BenchmarkDotNet.Diagnostics.dotTrace NuGet package. Once this package is installed, you can annotate your benchmarks with the [DotTraceDiagnoser] and get a dotTrace performance snapshot at the end of the benchmark run. #2328
  • Updated documentation website. We migrated to docfx 2.67 and got the refreshed modern template based on bootstrap 5 with dark/light theme switcher.
  • Updated BenchmarkDotNet.Templates. Multiple issues were resolved, now you can create new benchmark projects from terminal or your favorite IDE. #1658 #1881 #2149 #2338
  • Response file support. Now it's possible to pass additional arguments to BenchmarkDotNet using @filename syntax. #2320 #2348
  • Custom runtime support. #2285
  • Introduce CategoryDiscoverer, see IntroCategoryDiscoverer. #2306 #2307
  • Multiple bug fixes.

Milestone details

In the v0.13.6 scope, 19 issues were resolved and 33 pull requests were merged. This release includes 127 commits by 14 contributors.

Resolved issues (19)

  • #1658 Pass arguments to runner in dotnet new templates
  • #1694 Length cannot be less than zero
  • #1783 MemoryDiagnoser table output invalid symbol
  • #1856 Not Set Hardware Counter throws Argument Null Exception
  • #1877 Any combination of Params/ParamsSource/ParamsAllValues is broken
  • #1881 broken template for Rider
  • #2036 Publish changelog for each BenchmarkDotNet versions using GitHub releases (assignee: @AndreyAkinshin)
  • #2149 BenchmarkDotNet.Templates not showing in VS2022(17.3.6) new project dialog
  • #2249 BenchmarkSwitcher.Run nullability check warnings in .NET 7 (assignee: @AndreyAkinshin)
  • #2259 CSproj code generator produces invalid syntax after incorrectly parsing .targets file
  • #2264 Benchmarkdotnet crashes when flushing the logger
  • #2266 Incompatible Microsoft.CodeAnalysis.CSharp versioning with Razor
  • #2271 'Unknown processor' yet again (assignee: @br3aker)
  • #2279 Endless compile errors (assignee: @AndreyAkinshin)
  • #2282 TimeUnit Setting Not Being Used in Excel Report
  • #2306 Allow for benchmark categories composition using inheritance (assignee: @AndreyAkinshin)
  • #2312 CodeAnalysisTreatWarningsAsErrors=true in Directory.Build.props breaks auto-generated benchmark assembly build (assignee: @viktorz)
  • #2333 Cannot benchmark netcoreapp3.0
  • #2358 System.MissingMethodException: Method not found: 'System.String Perfolizer.Horology.TimeInterval.ToString(System.Globalization.CultureInfo, System.String, Perfolizer.Common.UnitPresentation)'

Merged pull requests (33)

Commits (127)

Contributors (14)

Thank you very much!

Additional details

Date: July 11, 2023

Milestone: v0.13.6 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.5

Highlights

  • Improved JitStatsDiagnoser. This diagnoser was added in v0.13.4, it shows various stats from the JIT compiler that were collected during entire benchmark run (amount of JITted methods, amount of tiered methods, how much memory JIT allocated during the benchmark). In this release, we improved metric collection (#2246, e715d5) and added the [JitStatsDiagnoser] attribute (#2250 512413).
  • Enable strong-named assemblies in the released NuGet packages #2258 #2263 5cd288
  • Avoid keeping referenced values returned from a benchmark in memory #1942 #2191 ff5dbe
  • Keep generated files when MSBuild bin log is requested #2252 #2254 d3fbc0
  • Add Id for UnresolvedDiagnoser (an exception fix) #2251 a992b5
  • Add brand names for Windows 22H2 and macOS 13 86f212 0c2699
  • Remove deprecated InProcessToolchain #2248 615384

Milestone details

In the v0.13.5 scope, 3 issues were resolved and 11 pull requests were merged. This release includes 16 commits by 4 contributors.

Resolved issues (3)

  • #1942 Consider changing Consume to not hold onto references for very long (assignee: @timcassell)
  • #2252 msbuild binlog for the benchmark projects gets cleaned up too, making diagnosis of build impossible (assignee: @adamsitnik)
  • #2258 Strong name validation failed

Merged pull requests (11)

Commits (16)

Contributors (4)

Thank you very much!

Additional details

Date: February 17, 2023

Milestone: v0.13.5 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.4

Highlights

JitStatsDiagnoser

This new diagnoser introduced in (#2243) allows getting advanced JIT statistics.

Sample usage:

dotnet run -c Release -f net7.0 --filter *IntroBasic.Sleep --profiler jit

Result:

Method Mean Error StdDev Methods JITted Methods Tiered JIT allocated memory
Sleep 15.53 ms 0.034 ms 0.032 ms 1,102 15 221,736 B

Milestone details

In the v0.13.4 scope, 1 issues were resolved and 4 pull requests were merged. This release includes 9 commits by 5 contributors.

Resolved issues (1)

  • #2237 Version 0.13.3 breaks LINQPad (and any non-Console application)

Merged pull requests (4)

Commits (9)

Contributors (5)

Thank you very much!

Additional details

Date: January 13, 2023

Milestone: v0.13.4 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.3

Highlights

Special Thanks

We would like to highlight some important contributors who helped us with this release:

  1. OpenHack'22 (devexperts.com) hackathon sponsored by the DevExperts company. As part of this hackathon we have received following PRs:
  1. Jan Vorlicek helped to implement arm64 disassembler during an internal Microsoft Hackathon:
  • #2107 Implement TryGetReferencedAddress for relative branches (by @janvorli)
  • #2123 Added other arm64 constant form extraction plus other changes (by @janvorli)
  1. Ahmed Garhy (maintainer of Capstone.NET) helped to improve Capstone.NET, which was need to implement arm64 disassembler:

Milestone details

In the v0.13.3 scope, 29 issues were resolved and 71 pull requests were merged. This release includes 87 commits by 22 contributors.

Resolved issues (29)

  • #989 [Suggestion] add API for detecting benchmark run failures. (assignee: @emanuel-v-r)
  • #1422 --disasm switch on ARM64 throws Exception (assignee: @adamsitnik)
  • #1469 Host exe marked /largeaddressaware
  • #1521 Iteration setup and cleanup causes job baseline error with multiple runtimes passed to BenchmarkSwitcher
  • #1684 Getting System.FormatException when passing certain string as params. (assignee: @YegorStepanov)
  • #1709 BenchmarkSwitcher executes all benchmarks that share a base class (assignee: @YegorStepanov)
  • #1736 Consider adding an ExceptionDiagnoser (assignee: @Serg046)
  • #1737 runtime knobs broken link (assignee: @YegorStepanov)
  • #1799 Enable interactive, incremental runs through the terminal (assignee: @melias)
  • #1839 Markdown output should escape the output
  • #1867 Trailing newline characters in input value break summary table
  • #1933 how to debug reflection error
  • #2064 Running with .Net 6/7 Mono JIT (assignee: @Serg046)
  • #2070 BenchmarkDotNet crashing on Linux with DisassemblyDiagnoser (assignee: @adamsitnik)
  • #2088 Running on Linux leaves terminal colors changed (assignee: @farQtech)
  • #2099 WASM is recognized as NativeAOT
  • #2102 Add benchmarking progress to console title (assignee: @franciscomoloureiro)
  • #2125 Tests Just Stop Running During Run (assignee: @adamsitnik)
  • #2126 DotNet 6 - VB - Error: The type or namespace name 'DeserializingBenchmarks' could not be found in the global namespace (are you missing an assembly reference?)
  • #2131 --filter should include argument/params names (assignee: @blouflashdb)
  • #2146 Build warning MSB3245: Could not locate the assembly Mono.Posix
  • #2167 Site: No vertical bar should be displayed for the Main page
  • #2185 WarmupCount=0 doesn't work (assignee: @AndreyAkinshin)
  • #2187 Reports for InProcess jobs don't include non-Result measurements (assignee: @AndreyAkinshin)
  • #2189 broker.ProcessData() hangs if something wrong is the spawned process (assignee: @adamsitnik)
  • #2210 Broken integration tests on Ubuntu 22.04 (assignee: @adamsitnik)
  • #2211 Setting affinity does not work for Environment.ProcessorCount >= 32 (assignee: @Donis-)
  • #2216 ppc64le architecture support required for running benchmarks on Power Systems (assignee: @adamsitnik)
  • #2223 Disassembler fails to disassemble some methods on Linux when using recursive mode (assignee: @adamsitnik)

Merged pull requests (71)

Commits (87)

Contributors (22)

Thank you very much!

Additional details

Date: December 26, 2022

Milestone: v0.13.3 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.2

Highlights

In BenchmarkDotNet v0.13.2, we have implemented support for:

  • .NET 7
  • NativeAOT
  • .NET Framework 4.8.1

We have also introduced new features and improvements including:

  • Possibility to hide selected columns,
  • Allocation Ratio column,
  • Logging progress and estimated finish time,
  • ARM64 support for BenchmarkDotNet.Diagnostics.Windows package,
  • Printing Hardware Intrinsics information,
  • Glob filters support for DisassemblyDiagnoser.

Of course, this release includes dozens of other improvements and bug fixes!

Our special thanks go to @mawosoft, @YegorStepanov and @radical who fixed a LOT of really nasty bugs.

Supported technologies

.NET 7 and .NET Framework 4.8.1

.NET 4.8.1 has been released earlier this month, while .NET 7 should land in autumn this year. Now you can use BenchmarkDotNet to compare both!

BenchmarkDotNet=v0.13.1.1845-nightly, OS=Windows 11 (10.0.22622.575)
Microsoft SQ1 3.0 GHz, 1 CPU, 8 logical and 8 physical cores
.NET SDK=7.0.100-preview.6.22352.1
  [Host]     : .NET 7.0.0 (7.0.22.32404), Arm64 RyuJIT AdvSIMD
  Job-QJVIDT : .NET 7.0.0 (7.0.22.32404), Arm64 RyuJIT AdvSIMD
  Job-FNNXOY : .NET Framework 4.8.1 (4.8.9032.0), Arm64 RyuJIT
Method Runtime Mean Allocated
BinaryTrees_2 .NET 7.0 193.6 ms 227.33 MB
BinaryTrees_2 .NET Framework 4.8.1 192.8 ms 228.01 MB

Credit for adding .NET 7 support in #1816 goes to @am11. @adamsitnik implemented .NET 4.8.1 support in #2044 and #2067. Big thanks to @MichalPetryka who was using preview versions of BenchmarkDotNet and reported a bug related to .NET 4.8.1 support: #2059 that got fixed before we released a new version.

NativeAOT

We are really excited to see the experimental CoreRT project grow and become officially supported by Microsoft (under new name: NativeAOT)! You can read more about it here. Implementing and improving the support was a combined effort of multiple contributors that spawned across multiple repositories:

As every AOT solution, NativeAOT has some limitations like limited reflection support or lack of dynamic assembly loading. Because of that, the host process (what you run from command line) is never an AOT process, but just a regular .NET process. This process (called Host process) uses reflection to read benchmarks metadata (find all [Benchmark] methods etc.), generates a new project that references the benchmarks and compiles it using ILCompiler. The boilerplate code is not using reflection, so the project is built with TrimmerDefaultAction=link (we have greatly reduced build time thanks to that). Such compilation produces a native executable, which is later started by the Host process. This process (called Benchmark or Child process) performs the actual benchmarking and reports the results back to the Host process. By default BenchmarkDotNet uses the latest version of Microsoft.DotNet.ILCompiler to build the NativeAOT benchmark according to this instructions. Moreover, BenchmarkDotNet by default uses current machines CPU features (change: #1994, discussion: #2061) and if you don't like this behavior, you can disable it.

This is why you need to:

  • install pre-requisites required by NativeAOT compiler
  • target .NET to be able to run NativeAOT benchmarks (example: <TargetFramework>net7.0</TargetFramework> in the .csproj file)
  • run the app as a .NET process (example: dotnet run -c Release -f net7.0).
  • specify the NativeAOT runtime in an explicit way, either by using console line arguments --runtimes nativeaot7.0 (the recommended approach), or by using[SimpleJob] attribute or by using the fluent Job config API Job.ShortRun.With(NativeAotRuntime.Net70):
dotnet run -c Release -f net7.0 --runtimes nativeaot7.0

For more examples please go to docs.

BenchmarkDotNet=v0.13.1.1845-nightly, OS=Windows 11 (10.0.22000.856/21H2)
AMD Ryzen Threadripper PRO 3945WX 12-Cores, 1 CPU, 24 logical and 12 physical cores
.NET SDK=7.0.100-rc.1.22423.16
  [Host]     : .NET 7.0.0 (7.0.22.42223), X64 RyuJIT AVX2
  Job-KDVXET : .NET 7.0.0 (7.0.22.42223), X64 RyuJIT AVX2
  Job-HFRAGK : .NET 7.0.0-rc.1.22424.9, X64 NativeAOT AVX2
Method Runtime Mean Ratio Allocated
BinaryTrees_2 .NET 7.0 95.06 ms 1.00 227.33 MB
BinaryTrees_2 NativeAOT 7.0 90.32 ms 0.96 227.33 MB

Some of .NET features are not supported by Native AOT, that is why you may want to filter them out using new [AotFilter] attribute:

[AotFilter("Currently not supported due to missing metadata.")]
public class Xml_FromStream<T>

New features and improvements

Hiding Columns

In #1621 @marcnet80 has reduced the number of columns displayed when multiple runtimes are being compared.

In #1890 @YegorStepanov has implemented a set of new APIs that allow for hiding columns. It's also exposed via -h and --hide command line arguments.

[MemoryDiagnoser] // adds Gen0, Gen1, Gen2 and Allocated Bytes columns
[HideColumns(Column.Gen0, Column.Gen1, Column.Gen2)] // dont display GenX columns
public class IntroHidingColumns
{
    [Benchmark]
    public byte[] AllocateArray() => new byte[100_000];
}

Sample results without [HideColumns]:

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
AllocateArray 3.303 us 0.0465 us 0.0435 us 31.2462 31.2462 31.2462 97.69 KB

With:

Method Mean Error StdDev Allocated
AllocateArray 3.489 us 0.0662 us 0.0763 us 97.69 KB

Imagine how much time @YegorStepanov has saved to all the people who so far were removing the columns manually from the results before publishing them on GitHub!

Allocation Ratio Column

In #1859 @YegorStepanov has added Allocation Ratio Column. It's enabled by default when MemoryDiagnoser is used and one of the benchmarks is marked as [Benchmark(Baseline = true)] or when there are multuple jobs defined and one of them is marked as baseline.

[MemoryDiagnoser]
public class AllocationColumnSample
{
    [Benchmark(Baseline = true)]
    [Arguments("test")]
    public string Builder(string value)
    {
        StringBuilder sb = new (value);

        for (int i = 0; i < 10; i++)
            sb.Append(value);

        return sb.ToString();
    }

    [Benchmark]
    [Arguments("test")]
    public string Concatenation(string value)
    {
        string result = value;

        for (int i = 0; i < 10; i++)
            result += value;

        return result;
    }
}
Method value Mean Error StdDev Ratio Gen 0 Allocated Alloc Ratio
Builder test 127.9 ns 0.49 ns 0.43 ns 1.00 0.0544 456 B 1.00
Concatenation test 120.2 ns 0.94 ns 0.88 ns 0.94 0.0908 760 B 1.67

Progress and estimated finish time

In #1909 @adamsitnik has added logging of progress and estimated finish time.

// ** Remained 5211 (99.9%) benchmark(s) to run. Estimated finish 2022-08-25 22:26 (9h 7m from now) **

arm64 support for BenchmarkDotNet.Diagnostics.Windows package

Due to the update to TraceEvent 3.0 BenchmarkDotNet.Diagnostics.Windows package has now arm64 support. Which means that you can use EtwProfiler and other ETW-based diagnosers on Windows arm64.

It would not be possible without @brianrob who implemented arm64 support for TraceEvent in #1533!

Hardware Intrinsics information

In #2051 @adamsitnik has extended the hardware information printed in the Summary table with Hardware Intrinsics information.

Sine the space in Summary table is quite limited, we full information is printed only in the log:

Special thanks to @tannergooding who provided a lot of very valuable feedback and @MichalPetryka who contributed an improvement #2066 for older runtimes.

Other improvements

  • WASM toolchain has received a lot of improvements from various .NET Team members: #1769, #1936, #1938, #1982.
  • Dependencies and TFMs updates: #1805, #1978, #2012, #2019, #2035.
  • Ensure proper SummaryStyle handling implemented by @mawosoft in #1828.
  • Preserving EnablePreviewFeatures project setting which gives the possibility to benchmark preview .NET features. Implemented by @kkokosa in #1842.
  • CI: Using non-deprecated macOS pool on Azure Pipelines, implemented by @akoeplinger in #1847
  • CI: Updating Cake to 2.0.0, adopting frosting project style. Implemented by @AndreyAkinshin in #1865.
  • Detecting ReSharper's Dynamic Program Analysis. Implemented by @adamsitnik in #1874.
  • Preventing benchmark failure when some of the exporters fail. Implemented by @epeshk in #1902.
  • Don't use the diagnosers when benchmarking has failed. Implemented by @adamsitnik in #1903.
  • Ensuring the default order of benchmarks is the same as declared in source code. Implemented by @adamsitnik in #1907.
  • Making BuildTimeout configurable. Implemented by @adamsitnik in #1906.
  • Notify users about private methods with Setup/Cleanup attributes. Implemented by @epeshk in #1912.
  • Don't run Roslyn Analyzers for the generated code. Implemented by @adamsitnik in #1917.
  • Ensure WorkloadActionUnroll and similar are optimized if possible. Implemented by @AndyAyersMS in #1935.
  • Don't use blocking acknowledgments when there is no need to. Implemented by @adamsitnik in #1940.
  • Executor: Don't use Process.ExitCode, unless the process has exited. Implemented by @radical in #1947.
  • Revise heuristic for initial jitting. Implemented by @AndyAyersMS in #1949.
  • Allow logging build commands output. Implemented by @radical in #1950.
  • Change Mono AOT mode to Normal AOT with LLVM JIT fall back. Implemented by @fanyang-mono in #1990.

Glob filters support for DisassemblyDiagnoser

So far, the disassembler was always loading the type that was generated by BDN, searching for the benchmark method, disassembling it and when encountered direct method calls, disassembling the called methods as well (if their depth was lesser or equal to max configured depth).

This was working fine, but only for direct method calls. For indirect, the disassembly was incomplete.

In #2072 @adamsitnik has added the possibility to filter methods disassembled by the DisassemblyDiagnoser.

The users can now pass --disasmFilter $globPattern and it's going to be applied to full signatures of all methods available for disassembling. Examples:

  • --disasmFilter *System.Text* - disassemble all System.Text methods.
  • --disasmFilter * - disassemble all possible methods.

Moreover, ClrMD was updated to v2 (#2040) and few disassembler bugs have been fixed (#2075, #2078). We are expecting that the disassembler will be more reliable now.

Docs and Samples improvements

Big thanks to @SnakyBeaky, @Distinctlyminty, @asaf92, @adamsitnik and @eiriktsarpalis who have improved our docs, samples and error messages!

#1776, #1797, #1850, #1861, #1939, #1974, #1997, #2042, #2050, #2068.

Bug fixes

  • WASM: #1811, #1846, #1916, #1926, #1932.
  • Diagnoser-provided Analysers weren't automatically added to Config. Fixed by @mawosoft in #1790.
  • Exportes could been duplicated. Fixed by @workgroupengineering in #1796.
  • Small bug in SummaryStyle. Fixed by @mawosoft in #1801.
  • InvalidOperationException/NullReferenceException in SmartParaemter. Fixed by @mawosoft in #1810.
  • Failures caused by colons in benchmark name. Fixed by @ronbrogan in #1823.
  • Some benchmark arugments were not properly escaped and were causing process launcher to crush. Fixed by @adamsitnik in #1841
  • Invalid size specifiers for Memory and Disassembly diagnosers. Fixed by @YegorStepanov in #1854 and #1855.
  • Respect LogicalGroup order in DefaultOrderer. Fixed by @AndreyAkinshin in #1866.
  • Endless loop in user interaction with redirected input. Fixed by @tmds in #.
  • Broken power plan support. Fixed by @YegorStepanov in #1885.
  • BytesAllocatedPerOperation was not being output by the JSON and XML exporters. Fixed by #martincostello in #1919.
  • Incorrect default InvocationCount in the summary table. Fixed by @AndreyAkinshin in #1929.
  • Failed build output was printed in reverse order. Fixed by @radical in #1945.
  • Build failures due to NETSDK1150. Fixed by @OlegOLK in #1981.
  • MetricCoumn was not respecting provided units when formatting values. Fixed by @mawosoft in #2033.
  • Generating invalid code that was causing benchmark failures. Fixed by @mawosoft in #2041.
  • CI: non-master build branches were publishing artifacts to the CI feed. Fixed by @mawosoft in #2047.
  • Comments in the project files were causing build failures. Fixed by @mawosoft in #2056.

Milestone details

In the v0.13.2 scope, 50 issues were resolved and 124 pull requests were merged. This release includes 147 commits by 34 contributors.

Resolved issues (50)

  • #299 Add API to remove columns for baseline comparison (assignee: @AndreyAkinshin)
  • #384 Print Vector.Count as part of machine info (assignee: @adamsitnik)
  • #722 Add scaled column for Allocated Memory
  • #837 Problems with default UnrollFactor in V0.11.0 (assignee: @adamsitnik)
  • #1177 Public types missing from reference assemblies don't work with ParamsSource
  • #1506 BenchmarkDotNet does not force to High Performance Mode during running (assignee: @YegorStepanov)
  • #1603 Don't display Job and Toolchain column when running benchmarks for multiple runtimes
  • #1669 --buildTimeout does not seem to work (assignee: @adamsitnik)
  • #1680 Cannot override RD.xml for NativeAOT
  • #1711 Add support for IBM Z architecture (assignee: @adamsitnik)
  • #1727 Unhelpful rounding in MemoryDiagnoser
  • #1753 "call: command not found" in .sh build script (assignee: @AndreyAkinshin)
  • #1755 EventPipeProfiler: File names are very verbose
  • #1756 EventPipeProfile: speedscope.app cannot parse result file (assignee: @adamsitnik)
  • #1774 Ability to compare --corerun with --runtimes (assignee: @adamsitnik)
  • #1775 Please add an easy way to remove columns
  • #1789 Small bug in ImmutableConfigbuilder (assignee: @mawosoft)
  • #1794 typo in error message
  • #1800 Small bug in SummaryStyle (assignee: @mawosoft)
  • #1803 Benchmark exception stops entire suite run (assignee: @adamsitnik)
  • #1809 Exception when using ParamsSource with (null) objects
  • #1812 Invalid codegen for Enumerable.Empty returned from ParamsSource
  • #1819 How to change exporter output path?
  • #1836 [Bug] System.InvalidOperationException: There is an error in XML document (0, 0).
  • #1857 Github actions ubuntu-latest "Unable to load shared library 'advapi32.dll' or one of its dependencies" when profiling dotnet 5
  • #1864 Is there a way to join summaries as if the benchmarks were run separately? (assignee: @AndreyAkinshin)
  • #1871 Detect ReSharper's Dynamic Program Analysis (assignee: @adamsitnik)
  • #1872 BenchmarkDontNet should make allowance for projects where Preview Features are enabled
  • #1887 MonoAotLLVM runtime is not actually AOTing things (assignee: @naricc)
  • #1900 Failed to benchmark .NET 7 in release mode
  • #1908 BenchmarkRunner.RunSource and BenchmarkRunner.RunUrl doesn't work
  • #1929 Incorrect default InvocationCount in the summary table (assignee: @AndreyAkinshin)
  • #1934 Ensure WorkloadActionUnroll and similar are optimized if possible
  • #1937 PR builds should not be published to BDN nightly feed (assignee: @mawosoft)
  • #1943 GitHub Actions Windows CI leg failing due to lack of native tools (assignee: @adamsitnik)
  • #1948 questions to help with future PRs
  • #1957 Broken pipe (assignee: @adamsitnik)
  • #1977 More Data for JSON and XML Export
  • #1989 Way to summarize all the params results (assignee: @YegorStepanov)
  • #1992 API Docs on website empty (assignee: @AndreyAkinshin)
  • #2000 DisassemblyDiagnoser broken on Linux (assignee: @adamsitnik)
  • #2009 Cleanup the dependencies (assignee: @martincostello)
  • #2011 Release-only build error when using ParamsSource with a LINQ method: Cannot implicitly convert type 'object' (assignee: @mawosoft)
  • #2016 Support for .NET 7?
  • #2028 Trying to build BDN on a windows arm64 host (assignee: @adamsitnik)
  • #2039 Support .NET Framework 4.8.1 (assignee: @adamsitnik)
  • #2055 Comment after breaks generated project
  • #2059 BenchmarkDotNet misreports .Net Framework as 4.8.1 (assignee: @adamsitnik)
  • #2063 BenchmarkDotNet nightly fails to disassemble .Net 7.0 properly
  • #2074 DisassemblyDiagnoser producing invalid disassembly (assignee: @adamsitnik)

Merged pull requests (124)

Commits (147)

Contributors (34)

Thank you very much!

Additional details

Date: August 26, 2022

Milestone: v0.13.2 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.1

BenchmarkDotNet v0.13.1 is a service update with various bug fixes and improvements.

Highlights

  • S390x architecture support (#1712)
  • Various WASM toolchain improvements (#1719, #1722, #1729, #1742, #1743, #1744, #1746, #1757, #1763)
  • Support of CoreRT on 5.0.3XX SDK (#1725)
  • Using Utf8 for reading from standard input (fixes a nasty encoding-related bug) (#1735)
  • Adjusting WaitForExit timeouts (#1745)
  • Support for returning unmanaged types from benchmarks with InProcessToolchain (#1750)
  • Disabled Tiered JIT (#1751)
  • A MemoryDiagnoser bug fix (#1762)
  • Allow users to hide Gen X columns (#1764)
  • Copy GC settings from host process to the benchmark process (#1765)
  • Do not split surrogates in shortified parameter values (#1705)

Milestone details

In the v0.13.1 scope, 3 issues were resolved and 23 pull requests were merged. This release includes 36 commits by 10 contributors.

Resolved issues (3)

  • #1703 Unable to run benchmark when ParamsSource refers to string with surrogate pairs
  • #1713 System.NotSupportedException: Unknown Acknowledgment: Acknowledgment when running in a github action (assignee: @adamsitnik)
  • #1714 AwaitingTasksShouldNotInterfereAllocationResults is flaky (assignee: @adamsitnik)

Merged pull requests (23)

Commits (36)

Contributors (10)

Thank you very much!

Additional details

Date: August 11, 2021

Milestone: v0.13.1 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.13.0

It's been a year since our last release. BenchmarkDotNet has been downloaded more than seven million times from nuget.org. It's more than we could have ever possibly imagined! Some could say, that it's also more than we can handle ;) That is why we wanted to once again thank all the contributors who helped us with 0.13.0 release!

Highlights

In BenchmarkDotNet v0.13.0, we have supported various technologies:

  • .NET 5 and .NET 6 target framework monikers
  • .NET SDK installed via snap
  • SingleFile deployment
  • Xamarin applications
  • WASM applications
  • Mono AOT

We have also introduced new features and improvements including:

  • Memory randomization
  • Method-specific job attributes
  • Sortable parameter columns
  • Customizable ratio column
  • Improved CoreRun and CoreRT support
  • Improved Hardware Counters support

Of course, this release includes dozens of other improvements and bug fixes!

Supported technologies

.NET 5, .NET 6, SingleFile and snap

At some point in time, netcoreapp5.0 moniker was changed to net5.0, which required a fix on our side (#1479, btw we love this kind of changes). Moreover, .NET 5 introduced platform-specific TMFs (example: net5.0-windows10.0.19041.0) which also required some extra work: #1560, #1691.

In #1523 support for .NET 6 was added.

<TargetFrameworks>net5.0;net5.0-windows10.0.19041.0;net6.0<TargetFrameworks>

In #1686 @am11 has implemented support for single file deployment (supported in .NET 5 onwards).

Last, but not least in #1652 snap support has been implemented.

adam@adsitnik-ubuntu:~/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples$ dotnet50 run -c Release -f net5.0 --filter BenchmarkDotNet.Samples.IntroColdStart.Foo
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 1 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start /snap/dotnet-sdk/112/dotnet restore  /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b
// command took 1.49s and exited with 0
// start /snap/dotnet-sdk/112/dotnet build -c Release  --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b
// command took 2.78s and exited with 0
// ***** Done, took 00:00:04 (4.37 sec)   *****
// Found 1 benchmarks:
//   IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)

// **************************
// Benchmark: IntroColdStart.Foo: Job-NECTOD(IterationCount=5, RunStrategy=ColdStart)
// *** Execute ***
// Launch: 1 / 1
// Execute: /snap/dotnet-sdk/112/dotnet "9a018ee4-0f33-46dd-9093-01d3bf31233b.dll" --benchmarkName "BenchmarkDotNet.Samples.IntroColdStart.Foo" --job "IterationCount=5, RunStrategy=ColdStart" --benchmarkId 0 in /home/adam/projects/BenchmarkDotNet/samples/BenchmarkDotNet.Samples/bin/Release/net5.0/9a018ee4-0f33-46dd-9093-01d3bf31233b/bin/Release/net5.0

Xamarin support

Thanks to the contributions of the amazing @jonathanpeppers BenchmarkDotNet supports Xamarin! The examples can be found in our repo: iOS, Android.

#1360, #1429, #1434, #1509

WASM support

Thanks to the work of @naricc you can now benchmark WASM using Mono Runtime! For more details, please refer to our docs.

#1483, #1498, #1500, #1501, #1507, #1592, #1689.

Mono AOT support

In another awesome contribution (#1662) @naricc has implemented Mono AOT support. The new toolchain supports doing Mono AOT runs with both the Mono-Mini compiler and the Mono-LLVM compiler (which uses LLVM on the back end).

For more details, please go to our docs.

New features and improvements

Memory randomization

In #1587 @adamsitnik has introduced a new, experimental feature called "Memory Randomization".

This feature allows you to ask BenchmarkDotNet to randomize the memory alignment by allocating random-sized byte arrays between iterations and call [GlobalSetup] before every benchmark iteration and [GlobalCleanup] after every benchmark iteration.

Sample benchmark:

public class IntroMemoryRandomization
{
    [Params(512 * 4)]
    public int Size;

    private int[] _array;
    private int[] _destination;

    [GlobalSetup]
    public void Setup()
    {
        _array = new int[Size];
        _destination = new int[Size];
    }

    [Benchmark]
    public void Array() => System.Array.Copy(_array, _destination, Size);
}

Without asking for the randomization, the objects are allocated in [GlobalSetup] and their unmodified addresses (and alignment) are used for all iterations (as long as they are not promoted to an older generation by the GC). This is typically the desired behavior, as it gives you very nice and flat distributions:

dotnet run -c Release --filter IntroMemoryRandomization

-------------------- Histogram --------------------
[502.859 ns ; 508.045 ns) | @@@@@@@@@@@@@@@
---------------------------------------------------

But if for some reason you are interested in getting a distribution that is better reflecting the "real-life" performance you can enable the randomization:

dotnet run -c Release --filter IntroMemoryRandomization --memoryRandomization true

-------------------- Histogram --------------------
[108.803 ns ; 213.537 ns) | @@@@@@@@@@@@@@@
[213.537 ns ; 315.458 ns) |
[315.458 ns ; 446.853 ns) | @@@@@@@@@@@@@@@@@@@@
[446.853 ns ; 559.259 ns) | @@@@@@@@@@@@@@@
---------------------------------------------------

Method-specific job attributes

From now, all attributes that derive from JobMutatorConfigBaseAttribute (full list) can be applied to methods. You no longer have to move a method to a separate type to customize config for it.

[Benchmark]
[WarmupCount(1)]
public void SingleWarmupIteration()

[Benchmark]
[WarmupCount(9)]
public void NineWarmupIterations()

Sortable parameter columns

In order to sort columns of parameters in the results table, you can use the Property Priority inside the params attribute. The priority range is [Int32.MinValue;Int32.MaxValue], lower priorities will appear earlier in the column order. The default priority is set to 0.

public class IntroParamsPriority
{
    [Params(100)]
    public int A { get; set; }

    [Params(10, Priority = -100)]
    public int B { get; set; }

    [Benchmark]
    public void Benchmark() => Thread.Sleep(A + B + 5);
}
Method B A Mean Error StdDev
Benchmark 10 100 115.4 ms 0.12 ms 0.11 ms

This feature got implemented by @JohannesDeml in #1612.

Customizable ratio column

Now it's possible to customize the format of the ratio column.

[Config(typeof(Config))]
public class IntroRatioStyle
{
    [Benchmark(Baseline = true)]
    public void Baseline() => Thread.Sleep(1000);

    [Benchmark]
    public void Bar() => Thread.Sleep(150);

    [Benchmark]
    public void Foo() => Thread.Sleep(1150);

    private class Config : ManualConfig
    {
        public Config()
        {
            SummaryStyle = SummaryStyle.Default.WithRatioStyle(RatioStyle.Trend);
        }
    }
}

|   Method |       Mean |   Error |  StdDev |        Ratio | RatioSD |
|--------- |-----------:|--------:|--------:|-------------:|--------:|
| Baseline | 1,000.6 ms | 2.48 ms | 0.14 ms |     baseline |         |
|      Bar |   150.9 ms | 1.30 ms | 0.07 ms | 6.63x faster |   0.00x |
|      Foo | 1,150.4 ms | 5.17 ms | 0.28 ms | 1.15x slower |   0.00x |

This feature was implemented in #731.

Improved CoreRun support

BenchmarkDotNet was reporting invalid .NET Core version number when comparing performance using CoreRuns built from dotnet/corefx and dotnet/runtime. Fixed by @adamsitnik in #1580

In #1552 @stanciuadrian has implemented support for all GcMode characteristics for CoreRunToolchain. Previously the settings were just ignored, now they are being translated to corresponding COMPlus_* env vars.

Improved CoreRT support

CoreRT has moved from https://github.com/dotnet/corert/ to https://github.com/dotnet/runtimelab/tree/feature/NativeAOT and we had to update the default compiler version and nuget feed address. Moreover, there was a bug in CoreRtToolchain which was causing any additional native dependencies to not work.

Big thanks to @MichalStrehovsky, @jkotas and @kant2002 for their help and support!

#1606, #1643, #1679

Command-line argument support in BenchmarkRunner

So far only BenchmarkSwitcher was capable of handling console line arguments. Thanks to @chan18 BenchmarkRunner supports them as well (#1292):

public class Program
{
    public static void Main(string[] args) => BenchmarkRunner.Run(typeof(Program).Assembly, args: args); 
}

New API: ManualConfig.CreateMinimumViable

ManualConfig.CreateEmpty creates a completely empty config. Without adding a column provider and a logger to the config the users won't see any results being printed. In #1582 @adamsitnik has introduced a new method that creates minimum viable config:

IConfig before = ManualConfig.CreateEmpty()
    .AddColumnProvider(DefaultColumnProviders.Instance)
    .AddLogger(ConsoleLogger.Default);

IConfig after = ManualConfig.CreateMinimumViable();

Benchmarking NuGet packages from custom feeds

In #1659 @workgroupengineering added the possibility to indicate the source of the tested nuget package and whether it is a pre-release version.

Deterministic benchmark builds

BenchmarkDotNet is now always enforcing Deterministic builds (#1489) and Optimizations enabled (#1494) which is a must-have if you are using custom build configurations. MSBuild enforces optimizations only for configurations that are named Release (the comparison is case-insensitive).

<ItemGroup Condition=" '$(Configuration)' == 'X' ">
   <PackageReference Include="SomeLibThatYouWantToBenchmark" Version="1.0.0" />
</ItemGroup>
<ItemGroup Condition=" '$(Configuration)' == 'Y' ">
   <PackageReference Include="SomeLibThatYouWantToBenchmark" Version="2.0.0" />
</ItemGroup>

var config = DefaultConfig.Instance
  .AddJob(Job.Default.WithCustomBuildConfiguration("X").WithId("X").AsBaseline())
  .AddJob(Job.Default.WithCustomBuildConfiguration("Y").WithId("Y"));

#1489, #1494

Improved Hardware Counters support

BenchmarkDotNet is being used by the .NET Team to ensure that .NET is not regressing. More than three thousand benchmarks (they can be found here) are being executed multiple times a day on multiple hardware configs. Recently, .NET Team started to use InstructionsRetired to help to filter unstable benchmarks that report regressions despite not changing the number of instructions retired. This has exposed few bugs in Hardware Counters support in BenchmarkDotNet, which all got fixed by @adamsitnik in #1547 and #1550. Moreover, we have removed the old PmcDiagnoser and extended EtwProfiler with the hardware counters support. It's just much more accurate and futureproof. For details, please go to #1548.

How stable was PmcDiagnoser (same benchmarks run twice in a row on the same machine):

Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 845,746
Burgers_0 .NET Core 2.1 30,154,151
Burgers_0 .NET Framework 4.6.1 4,230,848
Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 34,154,524
Burgers_0 .NET Core 2.1 246,534,203
Burgers_0 .NET Framework 4.6.1 2,607,686

How stable is EtwProfiler:

Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 3,069,978,261
Burgers_0 .NET Core 2.1 3,676,000,000
Burgers_0 .NET Framework 4.6.1 3,468,866,667
Method Runtime InstructionRetired/Op
Burgers_0 .NET 5.0 3,066,810,000
Burgers_0 .NET Core 2.1 3,674,666,667
Burgers_0 .NET Framework 4.6.1 3,468,600,000

Moreover, in #1540 @WojciechNagorski has added the removal of temporary files created by EtwProfiler.

Improved Troubleshooting

We have the possibility to ask BDN to stop on the first error: --stopOnFirstError true.

The problem was when the users had not asked for that, tried to run n benchmarks, all of them failed to build, and BDN was printing the same build error n times.

In #1672 @adamsitnik has changed that, so when all the build fails, BDN stops after printing the first error.

Moreover, we have also changed the default behavior for the failed builds of the boilerplate code. If the build fails, we don't remove the files. Previously we have required the users to pass --keepFiles to keep them. See #1567 for more details and don't forget about the Troubleshooting docs!

Docs and Samples improvements

Big thanks to @lukasz-pyrzyk, @fleckert, @MarecekF, @joostas, @michalgalecki, @WojciechNagorski, @MendelMonteiro, @kevinsalimi, @cedric-cf, @YohDeadfall, @jeffhandley and @JohannesDeml who have improved our docs and samples!

#1463, #1465, #1508, #1518, #1554, #1568, #1601, #1633, #1645, #1647, #1657, #1675, #1676, #1690.

Template improvements

  • Projects created out of our official templates might have been unexpectedly packed when running dotnet pack on the entire solution. In #1584 @kendaleiv has explicitly disabled packing for the template.
  • The template had netcoreapp3.0 TFM hardcoded. This got fixed by @https://github.com/ExceptionCaught in #1630 and #1632.
  • In #1667 @YohDeadfall has changed the default debug type from portable to pdbonly (required by DisassemblyDiagnoser).

Bug fixes

  • Very long string [Arguments] and [Params] were causing BenchmarkDotNet to crash. Fixed by @adamsitnik in #1248 and #1545. So far trace file names were containing full benchmark names and arguments. Now if the name is too long, the trace file name is a hash (consistent for multiple runs of the same benchmark). The same goes for passing benchmark name by the host process to the benchmark process via command-line arguments.
  • LangVersion set to a non-numeric value like latest was crashing the build. Fixed by @martincostello in #1420.
  • Windows 10 November 2019 was being recognized as 2018. Fixed by @kapsiR in #1437.
  • Assemblies loaded via streams were not supported. Fixed by @jeremyosterhoudt in #1443.
  • NativeMemoryProfiler was detecting small leaks that were false positives. Fixed by @WojciechNagorski in #1451 and #1600.
  • DisassemblyDiagnoser was crashing on Linux. Fixed by @damageboy in #1459.
  • Target framework moniker was being printed as toolchain name for Full .NET Framework benchmarks. Fixed by @svick in #1471.
  • [ParamsSource] returning IEnumerable<object[]> was not working properly when combined with [Arguments]. Fixed by @adamsitnik in #1478.
  • NullReferenceException in MultimodalDistributionAnalyzer. Fixed by @suslovk in #1488.
  • NotSupportedException was being thrown if there was an encoding mismatch between host and benchmark process. Diagnosed by @ChristophLindemann in #1487, fixed by @lovettchris in #1491.
  • MissingMethodException was being thrown in projects that referenced a newer version of Iced. Fixed by @Symbai in #1497 and in #1502.
  • AppendTargetFrameworkToOutputPath set to false was not supported. Fixed by @adamsitnik in #1563
  • A locking finalizer could have hanged benchmark process which would just print // AfterAll and never quit. Fixed by @adamsitnik in #1571. To prevent other hangs from happening, a timeout of 250ms was added. If the process does not quit after running the benchmarks and waiting 250ms, it's being force killed.
  • In some cases, JsonExporter was reporting NaN for some of the Statistics. This was breaking non-.NET JSON deserializers. Fixed by @marcnet80 in #1581.
  • UnitType.Size metrics were not using the provided number format. Fixed by @jodydonetti in #1569 and #1618.
  • MaxColumnWidth setting was not used for type names. Fixed by @JohannesDeml in #1609.
  • Current culture was not respected when formatting Ratio column values. Fixed by @JohannesDeml in #1610.
  • BenchmarkDotNet was redirecting Standard Error of the benchmark process, which was causing deadlocks for benchmarks that were writing to it. Fixed by @adamstinik in #1631
  • DisassemblyDiagnoser was failing to disassemble multiline source code. @YohDeadfall fixed that in #1674.
  • In #1644 @adamstinik has fixed the inconsistency between benchmark filter and hint.

Removal of the dotnet global tool

In #1006 (0.11.4) we have introduced a new dotnet global tool.

By looking at the number of reported bugs we got to the conclusion that the tool has not passed the test of time.

Why? Because it was based entirely on dynamic assembly loading which is very hard to get right in .NET and the fact that we support all existing .NET Runtimes (.NET, .NET Core, Mono, CoreRT) made it even harder (if not impossible).

We have removed it and the old versions are no longer supported. For more details, please refer to #1572.

Milestone details

In the v0.13.0 scope, 53 issues were resolved and 94 pull requests were merged. This release includes 111 commits by 37 contributors.

Resolved issues (53)

  • #721 Add possibility to customize BaselinedScaledColumn with provided func (assignee: @AndreyAkinshin)
  • #1136 benchmark / beforeEverything fails when running in docker-win
  • #1242 bug JSON Exporter exports NaN for some properties. This fails most JSON parsers (assignee: @marcnet80)
  • #1247 Support benchmarks with very long string arguments (assignee: @adamsitnik)
  • #1288 Fix Hardware Counters diagnoser (assignee: @adamsitnik)
  • #1290 BenchmarkRunner should support parsing console line arguments (assignee: @chan18)
  • #1333 Getting "Unknown Processor" Again
  • #1427 NativeMemoryProfiler reports false positive leak (assignee: @WojciechNagorski)
  • #1431 System.IO.FileNotFoundException with EtwProfiler
  • #1433 Benchmarks work in .NET 4.7.2 but not .NET Core 3.1
  • #1436 Wrong OsBrandString for Windows 10 1909
  • #1448 NRE in MultimodalDistributionAnalyzer
  • #1452 Setting LangVersion to Latest Causes an Error
  • #1464 DisassemblyDiagnoserConfig.ExportDiff needs better description
  • #1472 Add support of Wasm to Benchmark Dotnet (assignee: @naricc)
  • #1474 docs: Getting started guide
  • #1480 BenchmarkRunner not working from VSTS agent (assignee: @lovettchris)
  • #1487 Benchmarks do not execute (as service process / VSTS agent) on non US systems
  • #1493 Using WithCustomBuildConfiguration leads to always running with RyuJIT Debug (assignee: @adamsitnik)
  • #1495 [GcServer(true)] is ignored with --corerun
  • #1496 System.MissingMethodException: Method not found: 'Iced.Intel.MasmFormatter.get_MasmOptions()'.
  • #1512 Template has hardcoded netcoreapp3.0 TFM (assignee: @ExceptionCaught)
  • #1532 Auto-generated project has invalid XML
  • #1535 Can't benchmark library that targets 'net5.0-windows' framework (assignee: @adamsitnik)
  • #1537 Failed to run benchmarks on .net 5 RC1 and Preview LangVersion
  • #1539 Inconsistency between benchmark filter and hint (assignee: @adamsitnik)
  • #1544 [EtwProfiler] Merge operation failed return code 0x3 (assignee: @adamsitnik)
  • #1546 Sometimes Hardware Counters per Op are reported as NaNs (assignee: @adamsitnik)
  • #1549 InstructionPointerExporter has been broken for a while (assignee: @adamsitnik)
  • #1559 [Docs] Update Console Args doc (assignee: @kevinsalimi)
  • #1561 NativeMemoryProfiler doesn't report allocations in v0.12.1.1432 Nightly (assignee: @WojciechNagorski)
  • #1564 error MSB4086: A numeric comparison was attempted on "$(LangVersion)" (assignee: @adamsitnik)
  • #1565 More consistent formatting of results
  • #1566 BDN should not delete temporary build directories for failed builds (assignee: @adamsitnik)
  • #1570 Benchmark runs failing when using .NET 5 RC2 SDK installed via snap (assignee: @adamsitnik)
  • #1576 Missing/misleading version number with corerun
  • #1585 Non-optimized dependency
  • #1591 Wasm Benchmark Runs Failing with Target Framework Error (assignee: @naricc)
  • #1598 VB Net Framework project throws exception from command line tool
  • #1605 CoreRT / NativeAOT version (assignee: @adamsitnik)
  • #1607 Exporter approval tests has recently become unstable (assignee: @marcnet80)
  • #1613 EventPipeProfiler generating invalid SpeedScope files.
  • #1616 BenchmarkDotNet fail in WPF project with .NET 5.0-windows target
  • #1620 NullReferenceException in v0.12.1
  • #1623 Can I run Full Framework benchmarks without having a Console App? (assignee: @adamsitnik)
  • #1628 Installation uses legacy/archaic dotnetcore 2.1.503 (assignee: @adamsitnik)
  • #1629 Writing to Console.Error in benchmarked code causes a deadlock (assignee: @adamsitnik)
  • #1654 Update for 0.12.2
  • #1670 dotnet benchmark cli tool errors with .net5.0 assemblies (assignee: @adamsitnik)
  • #1673 Source code provider incorrectly handles multiline source code
  • #1685 Support for SingleFile && SelfContained apps
  • #1692 Bug running wasm benchmarks - Broken Pipe in writeline
  • #1693 Estimate Date for Supporting .NET 6 (assignee: @adamsitnik)

Merged pull requests (94)

Commits (111)

Contributors (37)

Thank you very much!

Additional details

Date: May 19, 2021

Milestone: v0.13.0 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.12.1

Highlights

  • .NET 5 support
    As you probably know, .NET Core 5 was officially rebranded to .NET 5. The new version of BenchmarkDotNet supports the new runtime after rebranding.
    #1399 465ebf
  • Perfolizer adoption
    The internal statistical engine of BenchmarkDotNet became mature enough to be transformed into an independent project. Meet perfolizer — a toolkit for performance analysis! While BenchmarkDotNet focuses on obtaining reliable measurements, perfolizer focuses on the decent analysis of measured data. You still can use all the statistical algorithms from BenchmarkDotNet, but you can also install perfolizer as a standalone NuGet package. You can find more details in the official announcement.
    #1386 54a061
  • Cross-platform disassembler
    Now the DisassemblyDiagnoser is cross-platform! The disassembling logic was also improved, now it handles runtime helper methods and references to method tables properly. Internally, it uses the Iced library for formatting assembly code.
    Special thanks to @adamsitnik for the implementation and @0xd4d for Iced!
    #1332 #899 #1316 #1364 294320
  • EventPipe-based cross-platform profiler
    Now you can easily profiler your benchmarks on Windows, Linux, and macOS! Just mark your class with the [EventPipeProfiler(...)] attribute and get a .speedscope.json file that you can browse in SpeedScope.
    Special thanks to @WojciechNagorski for the implementation!
    #1321 #1315 c648ff
  • New fluent API
    We continue to improve our API and make it easier for reading and writing.
    Special thanks to @WojciechNagorski for the implementation!
    #1273 #1234 640d88
  • Ref readonly support
    Now you can use ref readonly in benchmark signatures.
    Special thanks to @adamsitnik for the implementation!
    #1389 #1388 9ac777

Cross-platform disassembler

Just mark your benchmark class with the [DisassemblyDiagnoser] attribute and you will get the disassembly listings for all the benchmarks. The formatting looks pretty nice thanks to Iced. It works like a charm on Windows, Linux, and macOS.

[DisassemblyDiagnoser]
public class IntroDisassembly
{
    private int[] field = Enumerable.Range(0, 100).ToArray();

    [Benchmark]
    public int SumLocal()
    {
        var local = field; // we use local variable that points to the field

        int sum = 0;
        for (int i = 0; i < local.Length; i++)
            sum += local[i];

        return sum;
    }

    [Benchmark]
    public int SumField()
    {
        int sum = 0;
        for (int i = 0; i < field.Length; i++)
            sum += field[i];

        return sum;
    }
}

.NET Core 2.1.16 (CoreCLR 4.6.28516.03, CoreFX 4.6.28516.10), X64 RyuJIT

; BenchmarkDotNet.Samples.IntroDisassembly.SumLocal()
       mov       rax,[rcx+8]
       xor       edx,edx
       xor       ecx,ecx
       mov       r8d,[rax+8]
       test      r8d,r8d
       jle       short M00_L01
M00_L00:
       movsxd    r9,ecx
       add       edx,[rax+r9*4+10]
       inc       ecx
       cmp       r8d,ecx
       jg        short M00_L00
M00_L01:
       mov       eax,edx
       ret
; Total bytes of code 35

.NET Core 2.1.16 (CoreCLR 4.6.28516.03, CoreFX 4.6.28516.10), X64 RyuJIT

; BenchmarkDotNet.Samples.IntroDisassembly.SumField()
       sub       rsp,28
       xor       eax,eax
       xor       edx,edx
       mov       rcx,[rcx+8]
       cmp       dword ptr [rcx+8],0
       jle       short M00_L01
M00_L00:
       mov       r8,rcx
       cmp       edx,[r8+8]
       jae       short M00_L02
       movsxd    r9,edx
       add       eax,[r8+r9*4+10]
       inc       edx
       cmp       [rcx+8],edx
       jg        short M00_L00
M00_L01:
       add       rsp,28
       ret
M00_L02:
       call      CORINFO_HELP_RNGCHKFAIL
       int       3
; Total bytes of code 53

Now we handle runtime helper methods and references to method tables properly. Example:

Before:

; MicroBenchmarks.WithCallsAfter.Benchmark(Int32)
       push    rsi
       sub     rsp,20h
       mov     rsi,rcx
       cmp     edx,7FFFFFFFh
       jne     M00_L00
       call    MicroBenchmarks.WithCallsAfter.Static()
       mov     rcx,rsi
       call    MicroBenchmarks.WithCallsAfter.Instance()
       mov     rcx,rsi
       call    MicroBenchmarks.WithCallsAfter.Recursive()
       mov     rcx,rsi
       mov     rax,qword ptr [rsi]
       mov     rax,qword ptr [rax+40h]
       call    qword ptr [rax+20h]
       mov     rcx,rsi
       mov     edx,1
       mov     rax,7FF8F4217050h
       add     rsp,20h
       pop     rsi
       jmp     rax
M00_L00:
       mov     rcx,offset System_Private_CoreLib+0xa31d48
       call    coreclr!MetaDataGetDispenser+0x322a0
       mov     rsi,rax
       mov     ecx,0ACFAh
       mov     rdx,7FF8F42F4680h
       call    coreclr!MetaDataGetDispenser+0x17140
       mov     rdx,rax
       mov     rcx,rsi
       call    System.InvalidOperationException..ctor(System.String)
       mov     rcx,rsi
       call    coreclr!coreclr_shutdown_2+0x39f0
       int     3
       add     byte ptr [rax],al
       sbb     dword ptr [00007ff9`26284e30],eax
       add     dword ptr [rax+40h],esp
       add     byte ptr [rax],al
       add     byte ptr [rax],al
       add     byte ptr [rax],al
       add     byte ptr [rax-70BC4CCh],ah
; Total bytes of code 157

After:

; BenchmarkDotNet.Samples.WithCallsAfter.Benchmark(Int32)
       push      rsi
       sub       rsp,20
       mov       rsi,rcx
       cmp       edx,7FFFFFFF
       jne       M00_L00 
       call      BenchmarkDotNet.Samples.WithCallsAfter.Static() 
       mov       rcx,rsi
       call      BenchmarkDotNet.Samples.WithCallsAfter.Instance() 
       mov       rcx,rsi
       call      BenchmarkDotNet.Samples.WithCallsAfter.Recursive() 
       mov       rcx,rsi
       mov       rax,[rsi]
       mov       rax,[rax+40]
       call      qword ptr [rax+20]
       mov       rcx,rsi
       mov       edx,1
       mov       rax BenchmarkDotNet.Samples.WithCallsAfter.Benchmark(Boolean) 
       add       rsp,20
       pop       rsi
       jmp       rax
M00_L00:
       mov       rcx MT_System.InvalidOperationException 
       call      CORINFO_HELP_NEWSFAST 
       mov       rsi,rax
       mov       ecx,12D
       mov       rdx,7FF954FF83F0
       call      CORINFO_HELP_STRCNS 
       mov       rdx,rax
       mov       rcx,rsi
       call      System.InvalidOperationException..ctor(System.String) 
       mov       rcx,rsi
       call      CORINFO_HELP_THROW 
       int       3
; Total bytes of code 134

See also: Cross-runtime .NET disassembly with BenchmarkDotNet.

Special thanks to @adamsitnik for the implementation and @0xd4d for Iced!

EventPipe-based cross-platform profiler

Now you can easily profiler your benchmarks on Windows, Linux, and macOS!

If you want to use the new profiler, you should just mark your benchmark class with the [EventPipeProfiler(...)] attribute:

[EventPipeProfiler(EventPipeProfile.CpuSampling)] // <-- Enables new profiler
public class IntroEventPipeProfiler
{
    [Benchmark]
    public void Sleep() => Thread.Sleep(2000);
}

Once the benchmark run is finished, you get a .speedscope.json file that can be opened in SpeedScope:

The new profiler supports several modes:

  • CpuSampling - Useful for tracking CPU usage and general .NET runtime information. This is the default option.
  • GcVerbose - Tracks GC collections and samples object allocations.
  • GcCollect - Tracks GC collections only at very low overhead.
  • Jit - Logging when Just in time (JIT) compilation occurs. Logging of the internal workings of the Just In Time compiler. This is fairly verbose. It details decisions about interesting optimization (like inlining and tail call)

Please see Wojciech Nagórski's blog post for all the details.

Special thanks to @WojciechNagorski for the implementation!

New fluent API

We continue to improve our API and make it easier for reading and writing. The old API is still existing, but it is marked as obsolete and will be removed in the further library versions. The most significant changes:

Changes in Job configuration

Changes in IConfig/ManualConfig

Full fluent API

Special thanks to @WojciechNagorski for the implementation!

Ref readonly support

Now you can use ref readonly in benchmark signatures. Here is an example:

public class RefReadonlyBenchmark
{
    static readonly int[] array = { 1 };

    [Benchmark]
    public ref readonly int RefReadonly() => ref RefReadonlyMethod();

    static ref readonly int RefReadonlyMethod() => ref array[0];
}

Special thanks to @adamsitnik for the implementation!

Milestone details

In the v0.12.1 scope, 31 issues were resolved and 42 pull requests were merged. This release includes 85 commits by 19 contributors.

Resolved issues (31)

  • #641 RPlotExporter hanging (assignee: @m-mccormick)
  • #899 Tiered compilation and disassembler (assignee: @adamsitnik)
  • #1023 Out of process benchmarks fail with ASP.NET Core SDK reference
  • #1211 Binding Redirect Issues When Using Xml Serializers
  • #1234 Strong type fluent API proposal (assignee: @WojciechNagorski)
  • #1238 RunAllJoined Causing Exception (assignee: @gsomix)
  • #1262 Params attribute doesn`t work in F# if you specify more than one enum value in constructor (assignee: @gsomix)
  • #1295 Custom format/culture for report output values (for CSV, and maybe HTML, MD)
  • #1305 Copy UserSecrets from benchmark project
  • #1311 Spelling nit (assignee: @AndreyAkinshin)
  • #1312 Add an option to pass environment variables to the default job
  • #1315 Implement cross platform EventPipeProfiler diagnoser (assignee: @WojciechNagorski)
  • #1316 Implement Unix Disassembler for .NET Core (assignee: @adamsitnik)
  • #1318 use of NugetReference[] causes System.MissingMethodException: No parameterless constructor defined for this object. (assignee: @adamsitnik)
  • #1323 DisassemblyDiagnoser index outside array bounds (assignee: @AndreyAkinshin)
  • #1325 Surface native code size benchmarked code (assignee: @adamsitnik)
  • #1326 BDN does not build using dotnet sdk from the command line in Linux
  • #1339 Generated code and StyleCop.Analyzers (assignee: @adamsitnik)
  • #1348 Different display text for arrays depending on a value source (assignee: @YohDeadfall)
  • #1350 Warn the user if command line arguments were not passed to the BenchmarkSwitcher
  • #1353 Show Length when param type is an array
  • #1361 SimpleJobAttribute with RunStrategy and RuntimeMoniker
  • #1363 Wrong assembly binding redirects for Microsoft.Data.SqlClient.resources ; using in Netcore3.0 project (assignee: @adamsitnik)
  • #1364 Bug: Benchmark class with Console.WriteLine(1) fails for DisassemblyDiagnoser with 'Sequence contains no matching element' (assignee: @adamsitnik)
  • #1369 Parameter column doesn't seem to respect culture info (assignee: @Tyrrrz)
  • #1379 Unix CI builds are red (assignee: @AndreyAkinshin)
  • #1385 Make BaselineCustomColumn.GetValue public
  • #1388 'ref readonly' return is not supported (assignee: @adamsitnik)
  • #1396 MacOS Azure Pipeline build is broken (assignee: @AndreyAkinshin)
  • #1413 Plot with only one "default" Job (assignee: @AndreyAkinshin)
  • #1416 EventPipeProfiler Documentation (assignee: @WojciechNagorski)

Merged pull requests (42)

Commits (85)

Contributors (19)

Thank you very much!

Additional details

Date: April 6, 2020

Milestone: v0.12.1 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.12.0

It's been several months since our last release, but we have been working hard and have some new features for you!

Highlights

  • Features and major improvements
    • Advanced multiple target frameworks support
      Now BenchmarkDotNet supports .NET Framework 4.8, .NET Core 3.1, and .NET Core 5.0. Also, we reworked our API that allows targeting several runtimes from the same config: the new API is more consistent, flexible, and powerful. For example, if you want to execute your benchmarking using .NET Framework 4.8 and .NET Core 3.1, you can use the SimpleJob(RuntimeMoniker.Net48), [SimpleJob(RuntimeMoniker.NetCoreApp31)] attributes or Job.Default.With(ClrRuntime.Net48), Job.Default.With(CoreRuntime.Core31) jobs in a manual config. You can find more details below. #1188, #1186, #1236
    • Official templates for BenchmarkDotNet-based projects
      With the help of the BenchmarkDotNet.Templates NuGet package, you can easily create new projects from the command line via dotnet new benchmark. This command has a lot of useful options, so you can customize your new project as you want. #1044
    • New NativeMemoryProfiler
      NativeMemoryProfiler measure the native memory traffic and adds the extra columns Allocated native memory and Native memory leak to the summary table. Internally, it uses EtwProfiler to profile the code using ETW. #457, #1131, #1208, #1214, #1218, #1219
    • New ThreadingDiagnoser
      ThreadingDiagnoser also adds two extra columns to the summary table: Completed Work Items (the number of work items that have been processed in ThreadPool per single operation) and Lock Contentions (the number of times there was contention upon trying to take a Monitor's lock per single operation). Internally, it uses new APIs exposed in .NET Core 3.0. #1154, #1227
    • Improved MemoryDiagnoser
      Now MemoryDiagnoser includes memory allocated by all threads that were live during benchmark execution: a new GC API was exposed in .NET Core 3.0 preview6+. It allows to get the number of allocated bytes for all threads. #1155, #1153, #723
    • LINQPad 6 support
      Now both LINQPad 5 and LINQPad 6 are supported! #1241, #1245
    • Fast documentation search
      We continue to improve the usability of our documentation. In this release, we improved the search experience in the documentation: now it works almost instantly with the help of Algolia engine! #1148, #1158
  • Minor summary and exporter improvements
    • Improved presentation of the current architecture in the environment information
      In the previous version of BenchmarkDotNet, the reports always contained "64bit" or "32bit" which did not tell if it was ARM or not. Now it prints the full architecture name (x64, x86, ARM, or ARM64). For example, instead of .NET Framework 4.8 (4.8.3815.0), 64bit RyuJIT you will get .NET Framework 4.8 (4.8.3815.0), X64 RyuJIT or .NET Framework 4.8 (4.8.3815.0), ARM64 RyuJIT. #1213
    • Simplified reports for Full .NET Framework version
      Previous version: .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3324.0. Current version: .NET Framework 4.7.2 (4.7.3362.0), 64bit RyuJIT. #1114, #1111
    • More reliable CPU info on Windows
      We added a workaround to for a bug in wmic that uses \r\r\n as a line separator. #1144, #1145
    • Better naming for generated plots
      When [RPlotExporter] is used, BenchmarkDotNet generates a lot of useful plots in the BenchmarkDotNet.Artifacts folder. The naming of the plot files was improved: benchmarks without Params doesn't include a double dash (--) in their names anymore. 1183, 1212
    • Better density plot precision The previous version of BenchmarkDotNet used the rule-of-thumb bandwidth selector in RPlotExporter density plots. It was fine for unimodal distributions, but sometimes it produced misleading plots for multimodal distributions. Now, RPlotExporter uses the Sheather&Jones bandwidth selector that significantly improves the presentation of the density plots for complex distributions. 58fde64
    • Better alignment in HtmlExporter
      Now BenchmarkDotNet aligns the content exported by HtmlExporter to the right. #1189 dfa074
    • Better precision calculation in SummaryTable
      4e9eb43
    • Better summary analysis
      BenchmarkDotNet warns the user when benchmark baseline value is too close to zero and the columns derived from BaselineCustomColumn cannot be computed. #1161, #600
    • Make log file datetime format 24-hour
      #1149
    • Improve AskUser prompt message
      The error messages will surround * by quotes on Linux and macOS. #1147
  • Minor API improvements
    • ED-PELT algorithm for changepoint detection is now available
      You can find details in this blog post. f89091
    • Improved OutlierMode API
      BenchmarkDotNet performs measurement postprocessing that may remove some of the outlier values (it can be useful to remove upper outliers that we get because of the natural CPU noise). In the previous version, naming for the OutlierMode values was pretty confusing: None/OnlyUpper/OnlyLower/All. Now, these values were renamed to DontRemove/RemoveUpper/RemoveLower/RemoveAll. For example, if you want to remove all the outliers, you can annotate your benchmark with the [Outliers(OutlierMode.RemoveAll)] attribute. The old names still exist (to make sure that the changes are backward compatible), but they are marked as obsolete, and they will be removed in the future versions of the library. #1199, 0e4b8e
    • Add the possibility to pass Config to BenchmarkSwitcher.RunAll and RunAllJoined
      #1194, ae23bd
    • Improved command line experience
      When user uses --packages $path, the $path will be sent to the dotnet build command as well. 1187
    • Extend the list of supported power plans. Now it supports "ultimate", "balanced", and "power saver" plans. #1132, #1139
    • Make it possible to not enforce power plan on Windows. 1578c5c
    • Guid support in benchmark arguments
      Now you can use Guid instances as benchmark arguments. 04ec20b
    • Make ArgumentsSource support IEnumerable<object[]> for benchmarks accepting a single argument to mimic MemberData behaviour.
      ec296dc
    • Make FullNameProvider public
      So it can be reused by the dotnet/performance repository. 6d71308
    • Extend Summary with LogFilePath #1135, 6e6559
    • Allow namespace filtering for InliningDiagnoser
      #1106, #1130
    • Option to configure MaxParameterColumnWidth
      #1269, 4ec888
  • Other improvements
    • Misc improvements in the documentation
      #1175, #1173, #1180, #1203, #1204, #1206, #1209, #1219, #1225, #1279
    • Copy PreserveCompilationContext MSBuild setting from the project that defines benchmarks
      #1152, 063d1a
    • Add System.Buffers.ArrayPoolEventSource to the list of default .NET Providers of EtwProfiler
      #1179, a106b1
    • Consume CoreRT from the new NuGet feed
      Because CoreRT no longer publishes to MyGet. #1129
  • Breaking changes:
    • The [ClrJob], [CoreJob] and [CoreRtJob] attributes got obsoleted and replaced by a [SimpleJob] which requires the user to provide target framework moniker in an explicit way. (See the "Advanced multiple target frameworks support" section for details.) #1188, #1182, #1115, #1056, #993,
    • The old InProcessToolchain is now obsolete. It's recommended to use InProcessEmitToolchain. If you want to use the old one on purpose, you have to use InProcessNoEmitToolchain. #1123
  • Bug fixes:
    • Invalid arg passing in StreamLogger constructor. The append arg was not passed to the StreamWriter .ctor. #1185
    • Improve the output path of .etl files produced by EtwProfiler. EtwProfiler was throwing NRE for users who were using [ClrJob] and [CoreJob] attributes. #1156, #1072
    • Flush custom loggers at the end of benchmark session. #1134
    • Make ids for tag columns unique - when using multiple TagColumns only one TagColumn was printed in the results. #1146

Advanced multiple target frameworks support

Now BenchmarkDotNet supports .NET Framework 4.8, .NET Core 3.1, and .NET Core 5.0. Also, we reworked our API that allows targeting several runtimes from the same config: the new API is more consistent, flexible, and powerful. For example, if you want to execute your benchmarking using .NET Framework 4.8 and .NET Core 3.1, you can use the SimpleJob(RuntimeMoniker.Net48), [SimpleJob(RuntimeMoniker.NetCoreApp31)] attributes or Job.Default.With(ClrRuntime.Net48), Job.Default.With(CoreRuntime.Core31) jobs in a manual config.

Now let's discuss how to use it in detail. If you want to test multiple frameworks, your project file MUST target all of them and you MUST install the corresponding SDKs:

<TargetFrameworks>netcoreapp3.0;netcoreapp2.1;net48</TargetFrameworks>

If you run your benchmarks without specifying any custom settings, BenchmarkDotNet is going to run the benchmarks using the same framework as the host process (it corresponds to RuntimeMoniker.HostProcess):

dotnet run -c Release -f netcoreapp2.1 # is going to run the benchmarks using .NET Core 2.1
dotnet run -c Release -f netcoreapp3.0 # is going to run the benchmarks using .NET Core 3.0
dotnet run -c Release -f net48         # is going to run the benchmarks using .NET 4.8
mono $pathToExe                        # is going to run the benchmarks using Mono from your PATH

To run the benchmarks for multiple runtimes with a single command from the command line, you need to specify the runtime moniker names via --runtimes|-r console argument:

dotnet run -c Release -f netcoreapp2.1 --runtimes netcoreapp2.1 netcoreapp3.0 # is going to run the benchmarks using .NET Core 2.1 and .NET Core 3.0
dotnet run -c Release -f netcoreapp2.1 --runtimes netcoreapp2.1 net48         # is going to run the benchmarks using .NET Core 2.1 and .NET 4.8

What is going to happen if you provide multiple Full .NET Framework monikers? Let's say:

dotnet run -c Release -f net461 net472 net48

Full .NET Framework always runs every .NET executable using the latest .NET Framework available on a given machine. If you try to run the benchmarks for a few .NET TFMs, they are all going to be executed using the latest .NET Framework from your machine. The only difference is that they are all going to have different features enabled depending on the target version they were compiled for. You can read more about this here and here. This is .NET Framework behavior which can not be controlled by BenchmarkDotNet or any other tool.

Note: Console arguments support works only if you pass the args to BenchmarkSwitcher:

class Program
{
    static void Main(string[] args) 
        => BenchmarkSwitcher
            .FromAssembly(typeof(Program).Assembly)
            .Run(args); // crucial to make it work
}

You can achieve the same thing using [SimpleJobAttribute]:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;

namespace BenchmarkDotNet.Samples
{
    [SimpleJob(RuntimeMoniker.Net48)]
    [SimpleJob(RuntimeMoniker.Mono)]
    [SimpleJob(RuntimeMoniker.NetCoreApp21)]
    [SimpleJob(RuntimeMoniker.NetCoreApp30)]
    public class TheClassWithBenchmarks

Or using a custom config:

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Environments;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

namespace BenchmarkDotNet.Samples
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = DefaultConfig.Instance
                .With(Job.Default.With(CoreRuntime.Core21))
                .With(Job.Default.With(CoreRuntime.Core30))
                .With(Job.Default.With(ClrRuntime.Net48))
                .With(Job.Default.With(MonoRuntime.Default));

            BenchmarkSwitcher
                .FromAssembly(typeof(Program).Assembly)
                .Run(args, config);
        }
    }
}

The recommended way of running the benchmarks for multiple runtimes is to use the --runtimes console line argument. By using the console line argument, you don't need to edit the source code anytime you want to change the list of runtimes. Moreover, if you share the source code of the benchmark, other people can run it even if they don't have the exact same framework version installed.

Official templates for BenchmarkDotNet-based projects

Since v0.12.0, BenchmarkDotNet provides project templates to setup your benchmarks easily. The template exists for each major .NET language (C#, F# and VB) with equivalent features and structure. The templates require the .NET Core SDK. Once installed, run the following command to install the templates:

dotnet new -i BenchmarkDotNet.Templates

If you want to uninstall all BenchmarkDotNet templates:

dotnet new -u BenchmarkDotNet.Templates

The template is a NuGet package distributed over nuget.org: BenchmarkDotNet.Templates. To create a new C# benchmark library project from the template, run:

dotnet new benchmark

If you'd like to create F# or VB project, you can specify project language with -lang option:

dotnet new benchmark -lang F#
dotnet new benchmark -lang VB

The template projects have five additional options - all of them are optional. By default, a class library project targeting netstandard2.0 is created. You can specify -f or --frameworks to change target to one or more frameworks:

dotnet new benchmark -f netstandard2.0;net472

The option --console-app creates a console app project targeting netcoreapp3.0 with an entry point:

dotnet new benchmark --console-app

This lets you run the benchmarks from a console (dotnet run) or from your favorite IDE. The option -f or --frameworks will be ignored when --console-app is set. The option -b or --benchmarkName sets the name of the benchmark class:

dotnet new benchmark -b Md5VsSha256

BenchmarkDotNet lets you create a dedicated configuration class (see Configs) to customize the execution of your benchmarks. To create a benchmark project with a configuration class, use the option -c or --config:

dotnet new benchmark -c

The option --no-restore if specified, skips the automatic NuGet restore after the project is created:

dotnet new benchmark --no-restore

Use the -h or --help option to display all possible arguments with a description and the default values:

dotnet new benchmark --help

The version of the template NuGet package is synced with the BenchmarkDotNet package. For instance, the template version 0.12.0 is referencing BenchmarkDotnet 0.12.0 - there is no floating version behavior. For more info about the dotnet new CLI, please read the documentation.

New NativeMemoryProfiler

NativeMemoryProfiler measure the native memory traffic and adds the extra columns Allocated native memory and Native memory leak to the summary table. Internally, it uses EtwProfiler to profile the code using ETW.

Consider the following benchmark:

[ShortRunJob]
[NativeMemoryProfiler]
[MemoryDiagnoser]
public class IntroNativeMemory
{
    [Benchmark]
    public void BitmapWithLeaks()
    {
        var flag = new Bitmap(200, 100);
        var graphics = Graphics.FromImage(flag);
        var blackPen = new Pen(Color.Black, 3);
        graphics.DrawLine(blackPen, 100, 100, 500, 100);
    }

    [Benchmark]
    public void Bitmap()
    {
        using (var flag = new Bitmap(200, 100))
        {
            using (var graphics = Graphics.FromImage(flag))
            {
                using (var blackPen = new Pen(Color.Black, 3))
                {
                    graphics.DrawLine(blackPen, 100, 100, 500, 100);
                }
            }
        }
    }

    private const int Size = 20; // Greater value could cause System.OutOfMemoryException for test with memory leaks.
    private int ArraySize = Size * Marshal.SizeOf(typeof(int));

    [Benchmark]
    public unsafe void AllocHGlobal()
    {
        IntPtr unmanagedHandle = Marshal.AllocHGlobal(ArraySize);
        Span<byte> unmanaged = new Span<byte>(unmanagedHandle.ToPointer(), ArraySize);
        Marshal.FreeHGlobal(unmanagedHandle);
    }

    [Benchmark]
    public unsafe void AllocHGlobalWithLeaks()
    {
        IntPtr unmanagedHandle = Marshal.AllocHGlobal(ArraySize);
        Span<byte> unmanaged = new Span<byte>(unmanagedHandle.ToPointer(), ArraySize);
    }
}

It will produce the summary table like this one:

Method Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated Allocated native memory Native memory leak
BitmapWithLeaks 73,456.43 ns 22,498.10 ns 1,233.197 ns - - - 177 B 13183 B 11615 B
Bitmap 91,590.08 ns 101,468.12 ns 5,561.810 ns - - - 180 B 12624 B -
AllocHGlobal 79.91 ns 43.93 ns 2.408 ns - - - - 80 B -
AllocHGlobalWithLeaks 103.50 ns 153.21 ns 8.398 ns - - - - 80 B 80 B

As you can see, we have two additional columns Allocated native memory and Native memory leak that contain some very useful numbers!

New ThreadingDiagnoser

ThreadingDiagnoser also adds two extra columns to the summary table:

  • Completed Work Items: The number of work items that have been processed in ThreadPool (per single operation)
  • Lock Contentions: The number of times there was contention upon trying to take a Monitor's lock (per single operation)

Internally, it uses new APIs exposed in .NET Core 3.0.

It can be activated with the help of the [ThreadingDiagnoser] attribute:

[ThreadingDiagnoser]
public class IntroThreadingDiagnoser
{
    [Benchmark]
    public void CompleteOneWorkItem()
    {
        ManualResetEvent done = new ManualResetEvent(initialState: false);
        ThreadPool.QueueUserWorkItem(m => (m as ManualResetEvent).Set(), done);
        done.WaitOne();
    }
}

The above example will print a summary table like this one:

Method Mean StdDev Median Completed Work Items Lock Contentions
CompleteOneWorkItem 8,073.5519 ns 69.7261 ns 8,111.6074 ns 1.0000 -

LINQPad 6 support

Now both LINQPad 5 and LINQPad 6 are supported:

We continue to improve the usability of our documentation. In this release, we improved the search experience in the documentation: now it works almost instantly with the help of Algolia engine! That's how it looks:

Milestone details

In the v0.12.0 scope, 44 issues were resolved and 56 pull requests were merged. This release includes 110 commits by 25 contributors.

Resolved issues (44)

  • #198 [Feature request] No logger for benchmark run? (assignee: @CodeTherapist)
  • #311 How to debug benchmarks that fail with exception on file system access operations (assignee: @adamsitnik)
  • #457 Track Native Memory Allocations and more informations with our ETW Memory Diagnoser
  • #600 Scaling issue
  • #723 MemoryDiagnoser should include memory allocated by all Threads that were live during benchmark execution (assignee: @adamsitnik)
  • #995 Running benchmark fails when targeting netcoreapp2.2 (assignee: @adamsitnik)
  • #1028 Add new template for "dotnet new benchmark" (assignee: @CodeTherapist)
  • #1072 EtwProfiler exports trace file only for a single runtime when Runtimes are controlled via attributes (assignee: @adamsitnik)
  • #1106 Allow user defined namespace filter for InliningDiagnoser
  • #1111 Change the format of printed Full .NET Framework Version (assignee: @adamsitnik)
  • #1115 Running using dotnet benchmark uses wrong core runtime
  • #1132 The power management feature extension
  • #1134 StreamLogger is not properly flushed on shutdown (assignee: @AndreyAkinshin)
  • #1135 The default file logger and summary title are out of sync (assignee: @adamsitnik)
  • #1137 [Discussion] Improve search experience in the documentation
  • #1144 Incorrect CPU info for .NET Core applications
  • #1146 Only the first of multiple custom columns is included in the summary table (assignee: @AndreyAkinshin)
  • #1147 Update benchmark switcher instructions to work on Linux (assignee: @AndreyAkinshin)
  • #1149 Ambiguous hour component in log file name timestamp (assignee: @AndreyAkinshin)
  • #1152 Failed to test Roslyn. (assignee: @adamsitnik)
  • #1153 Use GC.GetTotalAllocatedBytes when available in MemoryDiagnoser (assignee: @adamsitnik)
  • #1154 Add a ConcurrencyDiagnoser? (assignee: @adamsitnik)
  • #1156 Crash when BenchmarkDotNet.Diagnostics.Windows.Session.GetFilePath throws NRE (assignee: @adamsitnik)
  • #1158 🔍 Improving search on docs with Algolia's DocSearch
  • #1162 Incorrect value of BenchmarkDotNet.Toolchains.DotNetCli.NetCoreAppSettings.Default
  • #1168 Consider using default value instead of hardcoded '-' in MetricColumn.GetValue()
  • #1179 Add System.Buffers.ArrayPoolEventSource to the list of default .NET Providers of EtwProfiler (assignee: @adamsitnik)
  • #1181 Log shows a wrong name for plot images
  • #1182 Benchingmarking .NET 4.8 Causes Errors
  • #1183 Plots of benchmarks without params have a double dash (--) in the name
  • #1186 Add support for --runtimes net48 (assignee: @adamsitnik)
  • #1187 When user uses --packages $path, the $path should be sent to dotnet build command as well (assignee: @adamsitnik)
  • #1194 RunAll with ToolChains (assignee: @adamsitnik)
  • #1195 LatestCoreRtVersionIsSupported fails on Mac Os
  • #1202 BenchmarkDotNet Not Recognizing CPU
  • #1220 [Docs] RScript / R_HOME setup
  • #1235 NativeMemoryProfiler exception (assignee: @WojciechNagorski)
  • #1236 Rework new API for target runtimes (assignee: @adamsitnik)
  • #1241 Can BenchmarkDotNet be enabled for LINQPad 6? (assignee: @adamsitnik)
  • #1269 Unable to show full param string in the report (assignee: @adamsitnik)
  • #1280 Improvement in memory statistics (assignee: @WojciechNagorski)
  • #1285 Issue with .Net Core version 3.0
  • #1289 How to config to not save .log files?
  • #1291 MemoryDiagnoser reports weird results for .NET Core 3.0

Merged pull requests (56)

Commits (110)

Contributors (25)

Thank you very much!

Additional details

Date: October 24, 2019

Milestone: v0.12.0 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.11.5

Highlights

  • Features and noticeable improvements
    • Power plan management
      Now BenchmarkDotNet executes all benchmarks with enabled High-Performance power plan (configurable, Windows-only). You can find some details below. #68 #952
    • Better Environment Variables API
      Now we have some additional extension methods which allow defining environment variables in user jobs. In the previous version, users always had to set an array of environment variables like this: job.With(new[] { new EnvironmentVariable("a", "b") }). Now it's possible to define an environment variable like job.With(new EnvironmentVariable("a", "b")) or job.WithEnvironmentVariable("a", "b"). Also, it's possible to clear the list of environment variables via job.WithoutEnvironmentVariables(). #1069 #1080
    • Better outlier messages
      The previous version of BenchmarkDotNet printed information about detected or removed outliers like this: "3 outliers were detected". It was nice, but it didn't provide additional information about these outliers (users had to read the full log to find the outliers values). Now BenchmarkDotNet prints additional information about outlier values like this: "3 outliers were detected (2.50 us..2.70 us)". e983cd31
    • Support modern CPU architecture names
      In the environment information section, BenchmarkDotNet prints not only the processor brand string, but also its architecture (e.g., "Intel Core i7-4770K CPU 3.50GHz (Haswell)"). However, it failed to recognize some recent processors. Now it's able to detect the architecture for modern Intel processors correctly (Kaby Lake, Kaby Lake R, Kaby Lake G, Amber Lake Y, Coffee Lake, Cannon Lake, Whiskey Lake). 995e053d
    • Introduce BenchmarkDotNet.Annotations
      Currently, BenchmarkDotNet targets .NET Standard 2.0. It makes some users unhappy because they want to define benchmarks in projects with lower target framework. We decided to start working on the BenchmarkDotNet.Annotations NuGet package which targets .NET Standard 1.0 and contains classes that users need to define their benchmarks. However, it's not easy to refactor the full source code base and move all relevant public APIs to this package. In v0.11.5, we did the first step and moved some of these APIs to BenchmarkDotNet.Annotations. We want to continue moving classes to this package and get full-featured annotation package in the future. #1084 #1096
    • Use InProcessEmitToolchain by default in InProcess benchmarks
      In BenchmarkDotNet 0.11.4, we introduced InProcessEmitToolchain. It's a new, full-featured InProcess toolchain which allows executing benchmarks in the current process without spawning additional process per benchmark. It supports [Arguments], [ArgumentsSource], passing the arguments by out, ref and returning stack-only types like Span<T>. However, in v0.11.4, it can be activated only if InProcessEmitToolchain is declared explicitly. Now it's enabled by default when [InProcessAttribute] is used. #1093
    • Introduce an option which prevents overwriting results
      Currently, BenchmarkDotNet overwrites results each time when the benchmarks are executed. It allows avoiding tons of obsolete files in the BenchmarkDotNet.Artifacts folder. However, the behavior doesn't fit all use cases: sometimes users want to keep results for old benchmark runs. Now we have a special option for it. The option can be activated via --noOverwrite console line argument or DontOverwriteResults extension method for IConfig #1074 #1083
  • Other improvements and bug fixes
    • Diagnostics and validation
      • Better benchmark declaration error processing
        In the previous version, BenchmarkDotNet threw an exception when some benchmark methods had an invalid declaration (e.g., invalid signature or invalid access modifiers). Now it prints a nice error message without ugly stack traces. #1107
      • Better error message for users who want to debug benchmarks
        #1073
      • Don't show the same validation error multiple times
        Now each error will be printed only once. #1079
      • Restrict MemoryDiagnoserAttribute usage to class
        Now it's impossible to accidentally mark a method with this attribute. #1119 #1122
    • Export
      • Better indentation in disassembly listings
        Now DissassemblyDiagnoser correctly process source code which contains tab as the indentation symbol #1110
      • Fix incorrect indentation for StackOverflow exporter
        Previously, StackOverflow exporter doesn't have a proper indent for job runtimes in the environment information. Now it's fixed. #826 #1104
      • Fix StackOverflowException in XmlExporter.Full
        #1086 #1090
      • Shortify MemoryDiagnoser column titles
        Now we use the following column titles: "Allocated" instead of "Allocated Memory/Op", "Gen 0" instead of "Gen 0/1k Op". The full description of each column can be found in the legend section below the summary table. #1081
    • Benchmark generation and execution
      • Fixed broken Orderers
        The previous version has a nasty bug with custom orderers. Now it's fixed. #1070 #1109
      • Better overhead evaluation
        In the previous version, BenchmarkDotNet evaluated the benchmark overhead as a mean value of all overhead iteration. It was fine in most cases, but in some cases, the mean value can be spoiled by outliers. Now BenchmarkDotNet uses the median value. #1116
      • Respect CopyLocalLockFileAssemblies
        Now BenchmarkDotNet respect the CopyLocalLockFileAssemblies value and copies it to the generated benchmark project. #1068 #1108
      • Disable CodeAnalysisRuleSet for generated benchmarks
        Previously, generated benchmarks may fail if the CodeAnalysisRuleSet is defined in Directory.Build.Props. #1082
      • Supported undefined enum values
        #1020 #1071
    • Other minor improvements and bug fixes

Power plans

In #952, power plan management was implemented. It resolves a pretty old issue #68 which was created more than three years ago. Now BenchmarkDotNet forces OS to execute a benchmark on the High-Performance power plan. You can disable this feature by modifying PowerPlanMode property. Here is an example where we are playing with this value:

[Config(typeof(Config))]
public class IntroPowerPlan
{
    private class Config : ManualConfig
    {
        public Config()
        {
            Add(Job.MediumRun.WithPowerPlan(PowerPlan.HighPerformance));
            Add(Job.MediumRun.WithPowerPlan(PowerPlan.UserPowerPlan));
        }
    }

    [Benchmark]
    public int IterationTest()
    {
        int j = 0;
        for (int i = 0; i < short.MaxValue; ++i)
            j = i;
        return j;
    }

    [Benchmark]
    public int SplitJoin()
        => string.Join(",", new string[1000]).Split(',').Length;
}

And here is an example of the summary table on plugged-off laptop:

        Method |       PowerPlan |     Mean |     Error |    StdDev |
-------------- |---------------- |---------:|----------:|----------:|
 IterationTest | HighPerformance | 40.80 us | 0.4168 us | 0.6109 us |
     SplitJoin | HighPerformance | 13.24 us | 0.2514 us | 0.3763 us |
 IterationTest |   UserPowerPlan | 79.72 us | 2.5623 us | 3.8352 us |
     SplitJoin |   UserPowerPlan | 24.54 us | 2.1062 us | 3.1525 us |

As you can see, the power plan produces a noticeable effect on the benchmark results.

This feature is available on Windows only.

Milestone details

In the v0.11.5 scope, 16 issues were resolved and 16 pull requests were merged. This release includes 44 commits by 12 contributors.

Resolved issues (16)

  • #68 Power management
  • #826 MarkdownExporter.StackOverflow fails to indent jobs' runtime descriptions (assignee: @alinasmirnova)
  • #976 System.NotSupportedException: Line must start with GC (assignee: @adamsitnik)
  • #1020 Errors using undefined enum values as benchmark arguments (assignee: @adamsitnik)
  • #1068 The csproj setting CopyLocalLockFileAssemblies is ignored
  • #1070 System.InvalidOperationException: Sequence contains more than one matching element after 0.11.4 (assignee: @AndreyAkinshin)
  • #1071 Enum flags results into compiler errors (assignee: @adamsitnik)
  • #1073 The error message for users who want to Debug benchmarks is not clear (assignee: @adamsitnik)
  • #1074 Results should be exported to a file with unique name
  • #1079 Dont display the same Validation Error many times (assignee: @adamsitnik)
  • #1086 XmlExporter.Full fails with StackOverflowException
  • #1107 Unhandled Exception: System.InvalidOperationException: Benchmark method '' has incorrect signature. Method shouldn't have any arguments. (assignee: @AndreyAkinshin)
  • #1109 Issue with DefaultOrderer
  • #1110 DisassemblyDiagnoser assumes indentation uses spaces (assignee: @AndreyAkinshin)
  • #1116 Use Median instead of Mean whe deducing Overhead (assignee: @AndreyAkinshin)
  • #1119 MemoryDiagnoserAttribute on methods (assignee: @Rizzen)

Merged pull requests (16)

Commits (44)

Contributors (12)

Thank you very much!

Additional details

Date: April 2, 2019

Milestone: v0.11.5 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.11.4

It's been few months since our last release, but we have been working hard and have some new features for you!

Highlights

  • Features
    • BenchmarkDotNet as a global tool: a new global tool which allows you to run benchmarks from given library. Now you can run benchmarks from the command line via dotnet benchmark. You can find more information about it in the documentation #1006 #213
    • InProcessEmitToolchain: new, full-featured InProcess toolchain which allows executing benchmarks in the current process without spawning additional process per benchmark. It supports [Arguments], [ArgumentsSource], passing the arguments by out, ref and returning stack-only types like Span<T>. #919, #921 #843
    • ARM support: BenchmarkDotNet supports now ARM and ARM64. #780, #979 #385
    • Mono AOT support: a new toolchain which allows running benchmarks using AOT version of Mono #940
    • NuGet symbol server support: BenchmarkDotNet publishes now the symbols to NuGet.org symbol server and you can easily debug it. #967 #968
    • Experimental support for .NET Core 3.0 WPF benchmarks #1066 For a working example please go to https://github.com/dotMorten/WPFBenchmarkTests
  • Improvements:
    • CoreRT Toolchain improvements - thanks to help from CoreRT Team we were able to make the CoreRT Toolchain work with the latest version of CoreRT #1001, #1057
    • Display the number of benchmarks to run: we now display how many benchmarks are going to be executed before running them and how many remained after running each of them #1048
    • Better list of suggested benchmarks for wrong filter #834 #957
    • Invalid assembly binding redirects generated by VS were a pain to many of our users, we have now implemented an approach that tries to work around this issue. #895, #667, #896, #942
    • Handling duplicates in IConfig #912, #938, #360, #463
    • Disassembly diagnoser should be kept in a separate directory to avoid dependency conflicts #1059
    • Give a warning when the [Benchmark] method is static - we now produce an error when users fail into this common issue #983 #985
    • C# keywords are prohibited as benchmark names #849
    • File names should be consistent across all OSes - < and > are valid on Unix, but not on Windows. We have unified that and now files produced on Unix and Windows have the same names. #981
    • Improve restore, build and publish projects #1002, #1013
    • Make it possible to disable OptimizationsValidator #988
    • Sort enum parameters by value instead of name #977
    • Detect .NET Core benchmark failures from LINQPad #980
    • Improved error logging #1008
    • Improved disassembly diff #1022
    • Using invariant culture for Roslyn Toolchain error messages #1042
    • Use only full names in the auto-generated code to avoid any possible conflicts with user code #1007, #1009 #1010
    • Write the GitHub table format to the console by default #1062
    • Proper cleanup on Ctrl+C/console Window exit #1061
    • Introduce StoppingCriteria - the first step to writing your own heuristic that determines when benchmarking should be stopped #984
  • Breaking changes:
    • .NET Standard 2.0 only - BenchmarkDotNet has a single target now, which should help with some assembly resolving issues. We had to drop .NET 4.6 support because of that and .NET 4.6.1 is now the oldest supported .NET Framework. #1032
    • CustomCoreClrToolchain has been removed, it's recommended to use CoreRunToolchain instead #928
  • Bug fixes:
    • NRE in Summary ctor #986 #987
    • ArgumentNullException when running benchmarks from published .NET Core app #1018
    • Dry jobs can eat iteration failures #1045
    • NullReferenceException in BenchmarkDotNet.Reports.SummaryTable after iteration failure #1046
    • Running the example throws NullReference #1049
    • Fix race condition in process output reader #1051 #1053
    • Fix a rare but really annoying bug where for some reason we were sometimes setting ForegroundColor to the same color as BackgroundColor and some parts of the logged output were invisible commit
    • StopOnFirstError must be respected commit

Milestone details

In the v0.11.4 scope, 42 issues were resolved and 41 pull requests were merged. This release includes 99 commits by 18 contributors.

Resolved issues (42)

  • #213 Add a "benchmark" cmd to dotnet
  • #343 FileNotFoundException on mono (assignee: @AndreyAkinshin)
  • #360 Duplicates handling for IConfig (assignee: @adamsitnik)
  • #385 Consider using S.R.InteropServices.RuntimeInformation.ProcessArchitecture instead pointer based detection of platform (assignee: @adamsitnik)
  • #387 Add a mode to BenchmarkSwitcher that allows to run a method inline for profiling (assignee: @adamsitnik)
  • #463 Review interface IConfig (assignee: @adamsitnik)
  • #660 [Params] should not change the order of provided values
  • #667 Does BenchMarkDotnet supports 4.7.1 Dotnet framework (assignee: @adamsitnik)
  • #687 Implement [Arguments] support for InProcessToolchain (assignee: @ig-sinicyn)
  • #714 Test BenchmarkDotNet against unstable/multimodal benchmarks from CoreCLR/CoreFX repo (assignee: @adamsitnik)
  • #780 ARM support (assignee: @adamsitnik)
  • #834 Better list of suggested benchmarks for wrong filter (assignee: @morgan-kn)
  • #843 Exception when returning a stackonly structure in a benchmark case using in-process toolchain (assignee: @ig-sinicyn)
  • #849 C# keywords are prohibited as benchmark names (assignee: @adamsitnik)
  • #895 Could not load file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies (assignee: @adamsitnik)
  • #896 .NET 4.7.1 console app tries to use BenchmarkRunner, gets "Could not load file or assembly 'System.Runtime, Version=4.1.2.0" (assignee: @adamsitnik)
  • #919 Feature proposal: full-featured inprocess toolchain (assignee: @ig-sinicyn)
  • #928 Remove CustomCoreClrToolchain (assignee: @adamsitnik)
  • #938 Run benchmark with DisasemblyDiagnoser with --disasam option from console (assignee: @adamsitnik)
  • #942 System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.1.0.0' after adding BenchmarkDotNet.Diagnostics.Windows (assignee: @adamsitnik)
  • #967 Publish a snupkg to the NuGet.org symbol server (assignee: @WojciechNagorski)
  • #970 False alarm bug report
  • #981 File names should be consistent across all OSes (assignee: @adamsitnik)
  • #982 Invalid string representaiton of CPU Affinity on a machine with more than 32 cores on ARM64 (assignee: @adamsitnik)
  • #983 Give a warning when the [Benchmark] method is static (assignee: @Rizzen)
  • #986 NRE in Summary ctor
  • #988 Make it possible to disable OptimizationsValidator (assignee: @adamsitnik)
  • #998 Missing images in docs (assignee: @AndreyAkinshin)
  • #1002 Multiple build/publish failure with --coreRun toolchain (assignee: @WojciechNagorski)
  • #1007 benchmark cannot have type Action (assignee: @adamsitnik)
  • #1010 Write unit tests which check that BenchmarkProgram.txt doesn't contain usings (assignee: @adamsitnik)
  • #1018 ArgumentNullException when running benchmarks from published .NET Core app
  • #1039 Some tests are broken on Net 461 (culture-dependent thing) (assignee: @ig-sinicyn)
  • #1045 Dry jobs can eat iteration failures (assignee: @adamsitnik)
  • #1046 NullReferenceException in BenchmarkDotNet.Reports.SummaryTable after iteration failure (assignee: @adamsitnik)
  • #1048 Display the number of benchmarks to run (assignee: @adamsitnik)
  • #1049 Running the example throws NullReference (assignee: @adamsitnik)
  • #1051 Fix race condition in process output reader (assignee: @adamsitnik)
  • #1056 Fails to build when targeting .NET Core 3.0 and .NET Framework (assignee: @adamsitnik)
  • #1059 Disassembly diagnoser should be kept in a separate directory to avoid dependency conflicts (assignee: @adamsitnik)
  • #1062 Write the GitHub table format to the console by default (assignee: @adamsitnik)
  • #1065 Allow benchmarking .NET Core Desktop apps (assignee: @adamsitnik)

Merged pull requests (41)

Commits (99)

Contributors (18)

Thank you very much!

Additional details

Date: February 15, 2019

Milestone: v0.11.4 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.11.3

Highlights

This release is focused mainly on bug fixes that were affecting user experience. But don't worry, we have some new features too!

  • Diagnosers
    • ConcurrencyVisualizerProfiler (allows profiling benchmarks on Windows and exporting the data to a trace file which can be opened with Concurrency Visualizer)
  • Command-line:
    • --stopOnFirstError: Stops the benchmarks execution on first error. #947
    • --statisticalTest: Performs a Mann–Whitney Statistical Test for identifying regressions and improvements. #960
  • Bug fixes:
    • Dry mode doesn't work because of the ZeroMeasurementHelper #943
    • MannWhitneyTest fails when comparing statistics of different sample size #948 and #950
    • Improve the dynamic loading of Diagnostics package #955
    • BenchmarkRunner.RunUrl throws NRE when Config is not provided #961
    • Don't require the users to do manual installation of TraceEvent when using Diagnostics package #962
    • Stop benchmark after closing application + Flush log after stopping benchmark #963

Diagnosers

ConcurrencyVisualizerProfiler

ConcurrencyVisualizerProfiler allows to profile the benchmarked .NET code on Windows and exports the data to a CVTrace file which can be opened with Concurrency Visualizer.

ConcurrencyVisualizerProfiler uses EtwProfiler to get a .etl file which still can be opened with PerfView or Windows Performance Analyzer. The difference is that it also enables all Task and Thread related ETW Providers and exports a simple xml which can be opened with Visual Studio if you install Concurrency Visualizer plugin

open trace

Utilization

Threads

Command-line

In this release, we have some new command-line arguments!

--stopOnFirstError: Stops the benchmarks execution on first error

When provided, BenchmarkDotNet is going to stop the benchmarks execution on first error.

  • #947 Add option to stop running when the first benchmark fails (by @wojtpl2)

--statisticalTest: Statistical Test

To perform a Mann–Whitney U Test and display the results in a dedicated column you need to provide the Threshold via --statisticalTest. Examples: 5%, 10ms, 100ns, 1s.

Example: run Mann–Whitney U test with relative ratio of 1% for all benchmarks for .NET 4.6 (base), .NET Core 2.0 and .NET Core 2.1.

class Program
{
    static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}

public class MySample
{
    [Benchmark]
    public void Sleep()
    {
#if NETFRAMEWORK
        Thread.Sleep(50);
#elif NETCOREAPP2_0
        Thread.Sleep(45);
#elif NETCOREAPP2_1
        Thread.Sleep(55);
#endif
    }
    
    [Benchmark]
    public void Same() => Thread.Sleep(50);
}

dotnet run -c Release -f netcoreapp2.1 --filter * --runtimes net46 netcoreapp2.0 netcoreapp2.1 --statisticalTest 1%

Note: .NET 4.6 will be our baseline because it was provided as first on the runtimes list.

Method Runtime Toolchain Mean Error StdDev Ratio MannWhitney(1%)
Sleep Clr net46 50.51 ms 0.1833 ms 0.1714 ms 1.00 Base
Sleep Core netcoreapp2.0 45.53 ms 0.1262 ms 0.1181 ms 0.90 Faster
Sleep Core netcoreapp2.1 55.50 ms 0.1217 ms 0.1138 ms 1.10 Slower
Same Clr net46 50.47 ms 0.1795 ms 0.1679 ms 1.00 Base
Same Core netcoreapp2.0 50.55 ms 0.1873 ms 0.1752 ms 1.00 Same
Same Core netcoreapp2.1 50.55 ms 0.2162 ms 0.2022 ms 1.00 Same

Milestone details

In the v0.11.3 scope, 10 issues were resolved and 10 pull requests were merged. This release includes 26 commits by 6 contributors.

Resolved issues (10)

  • #870 Error after adding OperationsPerInvoke (assignee: @AndreyAkinshin)
  • #885 Closing application dot't stop benchmark (assignee: @WojciechNagorski)
  • #933 Investigate hanging SingleBenchmarkCanBeExecutedForMultipleRuntimes test (assignee: @adamsitnik)
  • #939 We need an option to stop running when the first benchmark fails. (assignee: @WojciechNagorski)
  • #943 Dry mode doesn't work because of the ZeroMeasurementHelper (assignee: @AndreyAkinshin)
  • #948 BenchmarkDotNet.Mathematics.StatisticalTesting.MannWhitneyTest.PValueForSmallN(int n, int m, double u) (assignee: @AndreyAkinshin)
  • #950 MannWhitneyTest fails when comparing statistics of different sample size (assignee: @AndreyAkinshin)
  • #955 Improve the dynamic loading of Diagnostics package (assignee: @WojciechNagorski)
  • #961 BenchmarkRunner.RunUrl throws NRE when Config is not provided
  • #964 Concurrency Visualizer Profiler (assignee: @adamsitnik)

Merged pull requests (10)

Commits (26)

Contributors (6)

Thank you very much!

Additional details

Date: November 20, 2018

Milestone: v0.11.3 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.11.2

This release includes many PRs from the Hacktoberfest. We are very grateful to all the contributors who spent their time to help us make BenchmarkDotNet even better!

Highlights

In this release, we have many improvements in different areas:

  • Diagnosers
    • EtwProfiler (allows profiling benchmarks on Windows and exporting the data to a trace file)
  • Execution:
    • Comparing NuGet packages (now it's possible to compare different versions of the same package)
    • .NET Core 3.0 support
    • Deferred Execution Validator
  • Command-line:
    • --list: List of benchmarks
    • --info: Print environment info
    • --runtimes: Choosing execution runtimes (--runtimes net472 netcoreapp2.1 will executed a benchmark on .NET 4.7.2 and .NET Core 2.1)
    • Options for number of invocations and iterations
    • Custom default settings for console argument parser
    • Case-insensitive filter
    • Benchmarking with different CoreRun instances
    • Hardware counters command-line support
  • Exporters:
    • Markdown output for DisassemblyDiagnoser
    • Diff view for disassembler output
    • Improved LINQPad support (colored monospaced logs)
    • Better CPU brand strings
  • Attributes:
    • Async [GlobalSetup] and [GlobalCleanup] support
    • Introduced [ParamsAllValues]
    • Selecting Baseline across Methods and Jobs
  • Statistics:
    • Better statistical tests (Welch's t-test and Mann-Whitney U-test)
    • ZeroMeasurementAnalyser
    • RatioColumn
  • Other:
    • Azure Pipelines support for internal builds
    • Many minor bug fixes
    • Improved documentation
    • New tests

Diagnosers

EtwProfiler

EtwProfiler allows to profile the benchmarked .NET code on Windows and exports the data to a trace file which can be opened with PerfView or Windows Performance Analyzer.

EtwProfiler uses TraceEvent library which internally uses Event Tracing for Windows (ETW) to capture stack traces and important .NET Runtime events. Before the process with benchmarked code is started, EtwProfiler starts User and Kernel ETW sessions. Every session writes data to it's own file and captures different data. User session listens for the .NET Runtime events (GC, JIT etc) while the Kernel session gets CPU stacks and Hardware Counter events. After this, the process with benchmarked code is started. During the benchmark execution all the data is captured and written to a trace file. Moreover, BenchmarkDotNet Engine emits it's own events to be able to differentiate jitting, warmup, pilot and actual workload when analyzing the trace file. When the benchmarking is over, both sessions are closed and the two trace files are merged into one.

You can find more details in the documentation and in the blog post by Adam Sitnik.

Execution

Comparing NuGet packages

Now it's possible to compare performance of several versions of the same NuGet package. An example:

[Config(typeof(Config))]
public class IntroNuGet
{
  // Specify jobs with different versions of the same NuGet package to benchmark.
  // The NuGet versions referenced on these jobs must be greater or equal to the
  // same NuGet version referenced in this benchmark project.
  // Example: This benchmark project references Newtonsoft.Json 9.0.1
  private class Config : ManualConfig
  {
    public Config()
    {
      var baseJob = Job.MediumRun.With(CsProjCoreToolchain.Current.Value);
      Add(baseJob.WithNuGet("Newtonsoft.Json", "11.0.2").WithId("11.0.2"));
      Add(baseJob.WithNuGet("Newtonsoft.Json", "11.0.1").WithId("11.0.1"));
      Add(baseJob.WithNuGet("Newtonsoft.Json", "10.0.3").WithId("10.0.3"));
      Add(baseJob.WithNuGet("Newtonsoft.Json", "10.0.2").WithId("10.0.2"));
      Add(baseJob.WithNuGet("Newtonsoft.Json", "10.0.1").WithId("10.0.1"));
      Add(baseJob.WithNuGet("Newtonsoft.Json", "9.0.1").WithId("9.0.1"));
    }
  }
  
  [Benchmark]
  public void SerializeAnonymousObject()
    => JsonConvert.SerializeObject(
      new { hello = "world", price = 1.99, now = DateTime.UtcNow });
}

See also: BenchmarkDotNet.Samples.IntroNuGet

  • #290 Question: Any official way to benchmark same method between different assembly versions?
  • #931 Same NuGet version used when benchmarking different packages
  • #922 Enables benchmarking betweeen different Nuget packages (by @Shazwazza)
  • #932 Partition benchmark run info based on added nuget packages (by @blairconrad)
  • 92a786 Enables benchmarking betweeen different Nuget packages (#922) fixes #290 (by @Shazwazza)
  • 510685 Partition benchmark run info based on added nuget packages (#932) (by @blairconrad)
  • cf84a4 NuGet casing fix (by @AndreyAkinshin)

.NET Core 3.0 support

Now it's possible to run benchmarks on preview versions of .NET Core 3.0.

Deferred Execution Validator

In LINQ, execution of a query is usually deferred until the moment when you actually request the data. If your benchmark just returns IEnumerable or IQueryable it's not measuring the execution of the query, just the creation.

This is why we decided to warn you about this issue whenever it happens:

Benchmark IntroDeferredExecution.Wrong returns a deferred execution result (IEnumerable<Int32>). You need to either change the method declaration to return a materialized result or consume it on your own. You can use .Consume() extension method to do that.

Don't worry! We are also providing you with a Consume extension method which can execute given IEnumerable or IQueryable and consume its results. All you need to do is to create a Consumer instance, preferably store it in a field (to exclude the cost of creating Consumer from the benchmark itself) and pass it to Consume extension method.

Do not call .ToArray() because it's an expensive operation and it might dominate given benchmark!

See also: BenchmarkDotNet.Samples.IntroDeferredExecution

  • #858 Should the Engine iterate over and consume IEnumerable and IQueryable results? (assignee: @adamsitnik)
  • cebe2a Deferred Execution Validator, fixes #858 (by @adamsitnik)

Command-line

In this release, we have tons of improvements for command-line experience.

--list: List of benchmarks

The --list allows you to print all of the available benchmark names. Available options are:

  • flat - prints list of the available benchmarks: --list flat
BenchmarkDotNet.Samples.Algo_Md5VsSha256.Md5
BenchmarkDotNet.Samples.Algo_Md5VsSha256.Sha256
BenchmarkDotNet.Samples.IntroArguments.Benchmark
BenchmarkDotNet.Samples.IntroArgumentsSource.SingleArgument
BenchmarkDotNet.Samples.IntroArgumentsSource.ManyArguments
BenchmarkDotNet.Samples.IntroArrayParam.ArrayIndexOf
BenchmarkDotNet.Samples.IntroArrayParam.ManualIndexOf
BenchmarkDotNet.Samples.IntroBasic.Sleep
[...]
  • tree - prints tree of the available benchmarks: --list tree
BenchmarkDotNet
 └─Samples
    ├─Algo_Md5VsSha256
    │  ├─Md5
    │  └─Sha256
    ├─IntroArguments
    │  └─Benchmark
    ├─IntroArgumentsSource
    │  ├─SingleArgument
    │  └─ManyArguments
    ├─IntroArrayParam
    │  ├─ArrayIndexOf
    │  └─ManualIndexOf
    ├─IntroBasic
    │  ├─Sleep
[...]

The --list option works with the --filter option. Examples:

  • --list flat --filter *IntroSetupCleanup* prints:
BenchmarkDotNet.Samples.IntroSetupCleanupGlobal.Logic
BenchmarkDotNet.Samples.IntroSetupCleanupIteration.Benchmark
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkA
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkB
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkC
BenchmarkDotNet.Samples.IntroSetupCleanupTarget.BenchmarkD
  • --list tree --filter *IntroSetupCleanup* prints:
BenchmarkDotNet
 └─Samples
    ├─IntroSetupCleanupGlobal
    │  └─Logic
    ├─IntroSetupCleanupIteration
    │  └─Benchmark
    └─IntroSetupCleanupTarget
       ├─BenchmarkA
       ├─BenchmarkB
       ├─BenchmarkC
       └─BenchmarkD
  • #905 Implement --list
  • #914 Implement --list - fixes #905 (by @wojtpl2)
  • #916 Update console-args.md - add information about --list option (by @wojtpl2)
  • 330f66 Implement --list - fixes #905 (#914) (by @wojtpl2)
  • 6c7521 Update console-args.md - add information about --list option (#916) (by @wojtpl2)

--info: Print environment info

Some of our users really like the info we print about hardware and OS. Now we have the --info console line argument which does not run the benchmarks, but simply prints the info.

BenchmarkDotNet=v0.11.1.786-nightly, OS=Windows 10.0.17134.285 (1803/April2018Update/Redstone4)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
Frequency=3507500 Hz, Resolution=285.1033 ns, Timer=TSC
.NET Core SDK=3.0.100-alpha1-009642
  [Host] : .NET Core 3.0.0-preview1-27004-04 (CoreCLR 4.6.27003.04, CoreFX 4.6.27003.02), 64bit RyuJIT

--runtimes: Choosing execution runtimes

The --runtimes or just -r allows you to run the benchmarks for selected Runtimes. Available options are: Mono, CoreRT, Core, Clr net46, net461, net462, net47, net471, net472, netcoreapp2.0, netcoreapp2.1, netcoreapp2.2, netcoreapp3.0.

Example: run the benchmarks for .NET 4.7.2 and .NET Core 2.1:

dotnet run -c Release -- --runtimes net472 netcoreapp2.1

Options for number of invocations and iterations

  • --launchCount - how many times we should launch process with target benchmark. The default is 1.
  • --warmupCount - how many warmup iterations should be performed. If you set it, the minWarmupCount and maxWarmupCount are ignored. By default calculated by the heuristic.
  • --minWarmupCount - minimum count of warmup iterations that should be performed. The default is 6.
  • --maxWarmupCount - maximum count of warmup iterations that should be performed. The default is 50.
  • --iterationTime - desired time of execution of an iteration. Used by Pilot stage to estimate the number of invocations per iteration. 500ms by default.
  • --iterationCount - how many target iterations should be performed. By default calculated by the heuristic.
  • --minIterationCount - minimum number of iterations to run. The default is 15.
  • --maxIterationCount - maximum number of iterations to run. The default is 100.
  • --invocationCount - invocation count in a single iteration. By default calculated by the heuristic.
  • --unrollFactor - how many times the benchmark method will be invoked per one iteration of a generated loop. 16 by default
  • --runOncePerIteration - run the benchmark exactly once per iteration. False by default.

Example: run single warmup iteration, from 9 to 12 actual workload iterations.

dotnet run -c Release -- --warmupCount 1 --minIterationCount 9 --maxIterationCount 12

Custom default settings for console argument parser

If you want to have a possibility to specify custom default Job settings programmatically and optionally overwrite it with console line arguments, then you should create a global config with single job marked as .AsDefault and pass it to BenchmarkSwitcher together with the console line arguments.

Example: run single warmup iteration by default.

static void Main(string[] args)
    => BenchmarkSwitcher
        .FromAssembly(typeof(Program).Assembly)
        .Run(args, GetGlobalConfig());

static IConfig GetGlobalConfig()
    => DefaultConfig.Instance
        .With(Job.Default
            .WithWarmupCount(1)
            .AsDefault()); // the KEY to get it working

Now, the default settings are: WarmupCount=1 but you might still overwrite it from console args like in the example below:

dotnet run -c Release -- --warmupCount 2

Case-insensitive filter

The --filter or just -f allows you to filter the benchmarks by their full name (namespace.typeName.methodName) using glob patterns.

Examples:

  1. Run all benchmarks from System.Memory namespace: -f System.Memory*
  2. Run all benchmarks: -f *
  3. Run all benchmarks from ClassA and ClassB -f *ClassA* *ClassB*

Now this filter expression is case-insensitive.

Benchmarking with different CoreRun instances

CoreRun is a simpler version of dotnet run, used for developing CoreCLR and CoreFX.

Typically when working on the performance of .NET Core a developer has more than 1 copy of CoreRun. Example: CoreRun before my changes, and after my changes. This change allows to simply run same benchmark for few different CoreRuns to compare the perf in easy way.

Sample usage:

dotnet run -c Release -f netcoreapp2.1 -- -f *Empty.method --job dry --coreRun
C:\Projects\coreclr_upstream\bin\tests\Windows_NT.x64.Release\Tests\Core_Root\CoreRun.exe
C:\Projects\coreclr_upstream\bin\tests\Windows_NT.x64.Release\Tests\Core_Root_beforeMyChanges\CoreRun.exe

Sample output:

image

  • #925 Make it possible to run the benchmark with multiple CoreRun.exe (assignee: @adamsitnik)
  • 901616 when user provides CoreRun path and runtime in explicit way, we should use th... (by @adamsitnik)
  • 46bebf allow the users to run the same benchmarks using few different CoreRun.exe, f... (by @adamsitnik)

Hardware counters command-line support

--counters CacheMisses+InstructionRetired
  • 1e3df7 make it possible to specify hardware counters from command line (by @adamsitnik)
  • a4f91a better handling of edge cases for parsing hardware counters from the console ... (by @adamsitnik)

Exporters

Markdown output for DisassemblyDiagnoser

Now DisassemblyDiagnoser generates markdown version of the assembly listing.

  • #560 Suggestion: markdown output for DisassemblyDiagnoser (assignee: @adamsitnik)
  • 1e6235 github markdown exporter for Disassembler, fixes #560 (by @adamsitnik)

Diff view for disassembler output

Now we have PrettyGithubMarkdownDiffDisassemblyExporter which can generates nice diffs between assembly listings. This mode can be activated via the --disasmDiff command line argument or the printDiff: true argument of DisassemblyDiagnoserConfig. An output example (Diff between SumLocal and SumField on .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT)

-; BenchmarkDotNet.Samples.IntroDisassemblyRyuJit.SumLocal()
-                   var local = field; // we use local variable that points to the field
-            ^^^^^^^^^^^^^^^^^^
-       mov     rax,qword ptr [rcx+8]
+; BenchmarkDotNet.Samples.IntroDisassemblyRyuJit.SumField()
                    int sum = 0;
             ^^^^^^^^^^^^
-       xor     edx,edx
-                   for (int i = 0; i < local.Length; i++)
+       xor     eax,eax
+                   for (int i = 0; i < field.Length; i++)
                  ^^^^^^^^^
-       xor     ecx,ecx
-                   for (int i = 0; i < local.Length; i++)
+       xor     edx,edx
+                   for (int i = 0; i < field.Length; i++)
                             ^^^^^^^^^^^^^^^^
-       mov     r8d,dword ptr [rax+8]
-       test    r8d,r8d
+       mov     rcx,qword ptr [rcx+8]
+       cmp     dword ptr [rcx+8],0
        jle     M00_L01
-                       sum += local[i];
+                       sum += field[i];
                 ^^^^^^^^^^^^^^^^
 M00_L00:
-       movsxd  r9,ecx
-       add     edx,dword ptr [rax+r9*4+10h]
-                   for (int i = 0; i < local.Length; i++)
+       mov     r8,rcx
+       cmp     edx,dword ptr [r8+8]
+       jae     00007ff9`0c412c1f
+       movsxd  r9,edx
+       add     eax,dword ptr [r8+r9*4+10h]
+                   for (int i = 0; i < field.Length; i++)
                                               ^^^
-       inc     ecx
-       cmp     r8d,ecx
+       inc     edx
+       cmp     dword ptr [rcx+8],edx
        jg      M00_L00
                    return sum;
             ^^^^^^^^^^^
 M00_L01:
-       mov     eax,edx
-; Total bytes of code 34
+       add     rsp,28h
+; Total bytes of code 42
  • #544 Diff view for disassembler output (assignee: @wojtpl2)
  • #927 Improve Disassembly exporters and add PrettyGithubMarkdownDiffDisassemblyExporter (by @wojtpl2)
  • #936 Producing the asm diff reports on demand
  • #937 Producing the asm diff reports on demand - fix for #936 (by @wojtpl2)
  • 1903a1 Improve Disassembly exporters and add PrettyGithubMarkdownDiffDisassemblyExpo... (by @wojtpl2)
  • dd103b Producing the asm diff reports on demand - fixes #936 (#937) (by @wojtpl2)

Improved LINQPad support

If you run BenchmarkDotNet v0.11.2+ in LINQPad, your logs will be colored and monospaced:

linqpad

Better CPU brand strings

We did a lot of changes which improve the presentation form of the CPU brand string. Here is an example of such string in the previous version of BenchmarkDotNet:

AMD Ryzen 7 2700X Eight-Core Processor (Max: 4.10GHz), 1 CPU, 16 logical and 8 physical cores

Now it becomes:

AMD Ryzen 7 2700X 4.10GHz, 1 CPU, 16 logical and 8 physical cores

As you can see, "Eight-Core Processor" was removed (because we already have "8 physical cores"); "(Max: 4.10GHz)" was replaced by 4.10GHz (because the original CPU brand string doesn't contain the nominal frequency).

  • #859 Strange max frequency values on Windows (assignee: @Rizzen)
  • #909 Improve CPU Brand Strings without frequency
  • #860 Fix strange CPU Frequency values (by @Rizzen)
  • #910 Simplify AMD Ryzen CPU brand info (by @lahma)
  • a78b38 Fix strange CPU Frequency values (#860) (by @Rizzen)
  • 5df1e6 Simplify AMD Ryzen CPU brand info (#910) (by @lahma)

Attributes

Async GlobalSetup and GlobalCleanup

Now GlobalSetup and GlobalCleanup methods can be async.

See also: docs.setup-and-cleanup

Introduced ParamsAllValues

If you want to use all possible values of an enum or another type with a small number of values, you can use the [ParamsAllValues] attribute, instead of listing all the values by hand. The types supported by the attribute are:

  • bool
  • any enum that is not marked with [Flags]
  • Nullable<T>, where T is an enum or boolean

An example:

public class IntroParamsAllValues
{
    public enum CustomEnum
    {
        A,
        BB,
        CCC
    }

    [ParamsAllValues]
    public CustomEnum E { get; set; }

    [ParamsAllValues]
    public bool? B { get; set; }

    [Benchmark]
    public void Benchmark()
    {
        Thread.Sleep(
            E.ToString().Length * 100 +
            (B == true ? 20 : B == false ? 10 : 0));
    }
}

Output:

    Method |   E |     B |     Mean | Error |
---------- |---- |------ |---------:|------:|
 Benchmark |   A |     ? | 101.9 ms |    NA |
 Benchmark |   A | False | 111.9 ms |    NA |
 Benchmark |   A |  True | 122.3 ms |    NA |
 Benchmark |  BB |     ? | 201.5 ms |    NA |
 Benchmark |  BB | False | 211.8 ms |    NA |
 Benchmark |  BB |  True | 221.4 ms |    NA |
 Benchmark | CCC |     ? | 301.8 ms |    NA |
 Benchmark | CCC | False | 312.3 ms |    NA |
 Benchmark | CCC |  True | 322.2 ms |    NA |

// * Legends *
  E     : Value of the 'E' parameter
  B     : Value of the 'B' parameter

Selecting Baseline across Methods and Jobs

Now it's possible to mark a method and a job as baselines at the same time:

public class TheBaselines
{
    [Benchmark(Baseline = true)]
    public void Sleep100ms() => Thread.Sleep(TimeSpan.FromMilliseconds(100));
    
    [Benchmark]
    public void Sleep50ms() => Thread.Sleep(TimeSpan.FromMilliseconds(50));
}

static void Main(string[] args)
    => BenchmarkSwitcher
        .FromTypes(new[] { typeof(TheBaselines) })
        .Run(args,
                DefaultConfig.Instance
                    .With(Job.Core.AsBaseline())
                    .With(Job.Clr.WithId("CLR 4.7.2")));

Statistics

Statistical tests

In this release, statistical testing was significantly improved. Now it's possible to compare all benchmarks against baseline with the help of Welch's t-test or Mann–Whitney U test.

An example:

[StatisticalTestColumn(
    StatisticalTestKind.Welch, ThresholdUnit.Microseconds, 1, true)]
[StatisticalTestColumn(
    StatisticalTestKind.MannWhitney, ThresholdUnit.Microseconds, 1, true)]
[StatisticalTestColumn(
    StatisticalTestKind.Welch, ThresholdUnit.Ratio, 0.03, true)]
[StatisticalTestColumn(
    StatisticalTestKind.MannWhitney, ThresholdUnit.Ratio, 0.03, true)]
[SimpleJob(warmupCount: 0, targetCount: 5)]
public class IntroStatisticalTesting
{
    [Benchmark] public void Sleep50() => Thread.Sleep(50);
    [Benchmark] public void Sleep97() => Thread.Sleep(97);
    [Benchmark] public void Sleep99() => Thread.Sleep(99);
    [Benchmark(Baseline = true)] public void Sleep100() => Thread.Sleep(100);
    [Benchmark] public void Sleep101() => Thread.Sleep(101);
    [Benchmark] public void Sleep103() => Thread.Sleep(103);
    [Benchmark] public void Sleep150() => Thread.Sleep(150);
}

Output:

Method Mean Error StdDev Ratio Welch(1us)/p-values Welch(3%)/p-values MannWhitney(1us)/p-values MannWhitney(3%)/p-values
Sleep50 53.13 ms 0.5901 ms 0.1532 ms 0.51 Faster: 1.0000/0.0000 Faster: 1.0000/0.0000 Faster: 1.0000/0.0040 Faster: 1.0000/0.0040
Sleep97 100.07 ms 0.9093 ms 0.2361 ms 0.97 Faster: 1.0000/0.0000 Same: 1.0000/0.1290 Faster: 1.0000/0.0040 Same: 1.0000/0.1111
Sleep99 102.23 ms 2.4462 ms 0.6353 ms 0.99 Faster: 0.9928/0.0072 Same: 1.0000/0.9994 Faster: 0.9960/0.0079 Same: 1.0000/1.0000
Sleep100 103.34 ms 0.8180 ms 0.2124 ms 1.00 Base: 0.5029/0.5029 Base: 1.0000/1.0000 Base: 0.7262/0.7262 Base: 1.0000/1.0000
Sleep101 103.73 ms 2.1591 ms 0.5607 ms 1.00 Same: 0.1041/0.8969 Same: 0.9999/1.0000 Same: 0.1111/0.9246 Same: 1.0000/1.0000
Sleep103 106.21 ms 1.2511 ms 0.3249 ms 1.03 Slower: 0.0000/1.0000 Same: 0.9447/1.0000 Slower: 0.0040/1.0000 Same: 0.9246/1.0000
Sleep150 153.16 ms 3.4929 ms 0.9071 ms 1.48 Slower: 0.0000/1.0000 Slower: 0.0000/1.0000 Slower: 0.0040/1.0000 Slower: 0.0040/1.0000 
// * Legends *
  Mean                      : Arithmetic mean of all measurements
  Error                     : Half of 99.9% confidence interval
  StdDev                    : Standard deviation of all measurements
  Ratio                     : Mean of the ratio distribution ([Current]/[Baseline])
  Welch(1us)/p-values       : Welch-based TOST equivalence test with 1 us threshold. Format: 'Result: p-value(Slower)|p-value(Faster)'
  Welch(3%)/p-values        : Welch-based TOST equivalence test with 3% threshold. Format: 'Result: p-value(Slower)|p-value(Faster)'
  MannWhitney(1us)/p-values : MannWhitney-based TOST equivalence test with 1 us threshold. Format: 'Result: p-value(Slower)|p-value(Faster)'
  MannWhitney(3%)/p-values  : MannWhitney-based TOST equivalence test with 3% threshold. Format: 'Result: p-value(Slower)|p-value(Faster)'
  1 ms                      : 1 Millisecond (0.001 sec)

The statistical testing is a work-in-progress feature. In future versions of BenchmarkDotNet, we are planning to improve API, fill missed docs, and introduce more parameters for customization.

See also: BenchmarkDotNet.Samples.IntroStatisticalTesting

ZeroMeasurementAnalyser

When you have an empty benchmark like this

[Benchmark]
public void Empty() { }

The expected duration of this method is zero. However, you can get the mean value like 0.0023ns because of the natural noise. It's a pretty confusing result for many developers. Since v0.11.2, we have ZeroMeasurementAnalyser which warn you about such methods. By default, BenchmarkDotNet automatically evaluate overhead. In this case, ZeroMeasurementAnalyser runs Welch's t-test and compare actual and overhead measurements. If the overhead evaluation is disabled, it runs one-sample Student's t-test against a half of CPU cycle.

RatioColumn

The Ratio column was formerly known as Scaled. The old title was a source of misunderstanding and confusion because many developers interpreted it as the ratio of means (e.g., 50.46/100.39 for Time50). The ratio of distribution means and the mean of the ratio distribution are pretty close to each other in most cases, but they are not equal.

See also: BenchmarkDotNet.Samples.IntroBenchmarkBaseline, BenchmarkDotNet.Samples.IntroRatioSD, Benchmark and Job Baselines.

Milestone details

In the v0.11.2 scope, 28 issues were resolved and 33 pull requests were merged. This release includes 84 commits by 20 contributors.

Resolved issues (28)

  • #221 Investigate why CanEnableServerGcMode test fails for Core on appveyor
  • #290 Question: Any official way to benchmark same method between different assembly versions?
  • #447 Implement ColoredLogger for LinqPad
  • #521 Support async Setup/Cleanup
  • #544 Diff view for disassembler output (assignee: @WojciechNagorski)
  • #560 Suggestion: markdown output for DisassemblyDiagnoser (assignee: @adamsitnik)
  • #601 Surprising results
  • #658 [Params] for enums should include all values by default
  • #731 Add constant folding analyser
  • #788 Detect correct version of .NET Core in Docket container
  • #842 Benchmark filter: wildcards on *nix CLI (assignee: @adamsitnik)
  • #858 Should the Engine iterate over and consume IEnumerable and IQueryable results? (assignee: @adamsitnik)
  • #859 Strange max frequency values on Windows (assignee: @Rizzen)
  • #862 Don't print parse errors to the output (assignee: @adamsitnik)
  • #863 Make it easier to understand which process belongs to which benchmark (assignee: @adamsitnik)
  • #864 Make the filter case insensitive (assignee: @adamsitnik)
  • #874 .NET Core 3.0 support (assignee: @adamsitnik)
  • #879 Benchmark attributed with "HardwareCounters" throws an exception (assignee: @adamsitnik)
  • #880 Select Baseline across Methods and Jobs (assignee: @AndreyAkinshin)
  • #889 ArgumentsSource doesn't work if method takes 1 arg (assignee: @adamsitnik)
  • #891 Add docs about debugging BDN issues (assignee: @adamsitnik)
  • #904 Implement --info
  • #905 Implement --list
  • #909 Improve CPU Brand Strings without frequency
  • #911 Excluding specific namespaces from disassembly (assignee: @adamsitnik)
  • #925 Make it possible to run the benchmark with multiple CoreRun.exe (assignee: @adamsitnik)
  • #931 Same NuGet version used when benchmarking different packages
  • #936 Producing the asm diff reports on demand

Merged pull requests (33)

Commits (84)

Contributors (20)

Thank you very much!

Additional details

Date: November 1, 2018

Milestone: v0.11.2 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.11.1

This release includes some minor improvements and bug fixes:

  • Fixed RPlotExporter (#852, #855). In v0.11.0, the plot generation was broken because of the huge renaming in #787.
  • ArgumentsSource now supports additional types like Type (#840), BigInteger (#850), DateTime (#853), and special double values like double.NaN (#851)
  • Generated projects ignore Directory.Build.props and Directory.Build.targets files #854
  • Now it's possible to run benchmarks with CoreRun (de152c, #857)

Milestone details

In the v0.11.1 scope, 7 issues were resolved and 2 pull requests were merged. This release includes 29 commits by 4 contributors.

Resolved issues (7)

Merged pull requests (2)

  • #839 Small Typo in changelog (by @Tornhoof)
  • #854 Exclude Directory.Build.props/targets from generated csproj files (by @agocke)

Commits (29)

Contributors (4)

Thank you very much!

Additional details

Date: August 22, 2018

Milestone: v0.11.1 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.11.0

This is one of the biggest releases of BenchmarkDotNet ever. There are so many improvements. We have new documentation, many performance improvements, Job Mutators, better user experience, correct Ctrl+C handling, better generic benchmarks support, more scenarios for passing arguments to benchmarks, awesome support of console arguments, unicode support, LLVM support in MonoDisassembler, and many-many other improvements and bug fixes!

A big part of the features and bug fixes were implemented to meet the enterprise requirements of Microsoft to make it possible to port CoreCLR, CoreFX, and CoreFXLab to BenchmarkDotNet.

The release would not be possible without many contributions from amazing community members. This release is a combined effort. We build BenchmarkDotNet together to make benchmarking .NET code easy and available to everyone for free!

New documentation

We have many improvements in our documentation! The new docs include:

  • DocFX under the hood
  • Detailed changelogs which includes all commits, merged pull requests and resolved issues
  • API references
  • Code samples for main features: we generate it automatically based on the BenchmarkDotNet.Samples project; it means that all samples can always be compiled (no more samples with outdated API)
  • Better UI
  • Documentation versioning: now it's possible to look at the documentation for recent BenchmarkDotNet versions

Performance improvements

BenchmarkDotNet needs to be capable of running few thousands of CoreFX and CoreCLR benchmarks in an acceptable amount of time. The code itself was already optimized so we needed architectural and design changes to meet this requirement.

Generate one executable per runtime settings

To ensure that the side effects of one benchmark run does not affect another benchmark run BenchmarkDotNet generates, builds and runs every benchmark in a dedicated process. So far we were generating and building one executable per benchmark, now we generate and build one executable per runtime settings. So if you want to run ten thousands of benchmarks for .NET Core 2.1 we are going to generate and build single executable, not ten thousand. If you target multiple runtimes the build is going to be executed in parallel. Moreover, if one of the parallel builds fail it's going to be repeated in a sequential way.

Previously the time to generate and build 650 benchmarks from our Samples project was one hour. Now it's something around 13 seconds which means 276 X improvement for this particular scenario. You can see the changes here.

Don't execute long operations more than once per iteration

BenchmarkDotNet was designed to allow for very accurate and stable micro-benchmarking. One of the techniques that we use is manual loop unrolling. In practice, it meant that for every iteration we were executing the benchmark at least 16 times (the default UnrollFactor value). It was of course not desired for the very time-consuming benchmarks.

So far this feature was always enabled by default and users would need to configure UnrollFactor=1 to disable it. Now BenchmarkDotNet is going to discover such scenario and don't perform manual loop unrolling for the very time-consuming benchmarks. BenchmarkDotNet uses Job.IterationTime setting (the default is 0.5s) in the Pilot Experiment stage to determine how many times given benchmark should be executed per iteration.

Example:

public class Program
{
    static void Main() => BenchmarkRunner.Run<Program>();

    [Benchmark]
    public void Sleep1s() => Thread.Sleep(TimeSpan.FromSeconds(1));
}

Time to run with the previous version: 374 seconds. With 0.11.0 it's 27 seconds which gives us almost 14 X improvement. A good example of benchmarks that are going to benefit from this change are computer game benchmarks and ML.NET benchmarks. You can see the changes here and here.

Exposing more configuration settings

The default settings were configured to work well with every scenario. Before running the benchmark, BenchmarkDotNet does not know anything about it. This is why it performs many warmup iterations before running the benchmarks.

When you author benchmarks and run them many times you can come up with custom settings that produce similar results but in a shorter manner of time. To allow you to do that we have exposed:

  • Job.MinIterationCount (default value is 15)
  • Job.MaxIterationCount (default value is 100)
  • Job.MinWarmupIterationCount (default value is 6)
  • Job.MaxWarmupIterationCount (default value is 50)

User Experience

One of the biggest success factors of BenchmarkDotNet is a great user experience. The tool just works as expected and makes your life easy. We want to make it even better!

.NET Standard 2.0

We have ported BenchmarkDotNet to .NET Standard 2.0 and thanks to that we were able to not only simplify our code and build process but also merge BenchmarkDotNet.Core.dll and BenchmarkDotNet.Toolchains.Roslyn.dll into BenchmarkDotNet.dll. We still support .NET 4.6 but we have dropped .NET Core 1.1 support. More information and full discussion can be found here.

Note: Our BenchmarkDotNet.Diagnostics.Windows package which uses EventTrace to implement ETW-based diagnosers was also ported to .NET Standard 2.0 and you can now use all the ETW diagnosers with .NET Core on Windows. We plan to add EventPipe support and make this page fully cross-platform and Unix compatible soon.

Using complex types as benchmark arguments

So far we have required the users to implement IParam interface to make the custom complex types work as benchmark arguments/parameters. This has changed, now the users can use any complex types as arguments and it will just work (more).

public class Program
{
    static void Main(string[] args) => BenchmarkRunner.Run<Program>();

    public IEnumerable<object> Arguments()
    {
        yield return new Point2D(10, 200);
    }

    [Benchmark]
    [ArgumentsSource(nameof(Arguments))]
    public int WithArgument(Point2D point) => point.X + point.Y;
}

public class Point2D
{
    public int X, Y;

    public Point2D(int x, int y)
    {
        X = x;
        Y = y;
    }

    public override string ToString() => $"[{X},{Y}]";
}

Note: If you want to control what will be displayed in the summary you should override ToString.

If IterationSetup is provided run benchmark once per iteration

When Stephen Toub says that something is buggy, it most probably is. BenchmarkDotNet performs multiple invocations of benchmark per every iteration. When we have exposed the [IterationSetup] attribute many users were expecting that the IterationSetup is going to be invoked before every benchmark execution.

It was invoked before every iteration, and iteration was more than one benchmark call if the user did not configure that explicitly. We have changed that and now if you provide an [IterationSetup] method it is going to be executed before every iteration and iteration will invoke the benchmark just once.

public class Test
{
    public static void Main() => BenchmarkRunner.Run<Test>();

    [IterationSetup]
    public void MySetup() => Console.WriteLine("MySetup");

    [Benchmark]
    public void MyBenchmark() => Console.WriteLine("MyBenchmark");
}

Before:

MySetup
MyBenchmark
MyBenchmark
MyBenchmark
MyBenchmark
(...)

After:

MySetup
MyBenchmark
MySetup
MyBenchmark
MySetup
MyBenchmark
(...)

Note: If you want to configure how many times benchmark should be invoked per iteration you can use the new [InvocationCountAttribute].

Job Mutators

Job represents a set of settings to run the benchmarks. We run every benchmark for every job defined by the user. The problem was that so far many jobs were just added to the config instead of being merged with other jobs.

An example:

[ClrJob, CoreJob]
[GcServer(true)]
public class MyBenchmarkClass

Resulted in 3 jobs and 3 benchmark executions: ClrJob, CoreJob and GcServer(true) for current runtime.

Now all Jobs and their corresponding attributes marked as mutators are going to be applied to other jobs, not just added to the config. So in this particular scenario, the benchmarks from MyBenchmarkClass are going to be executed for .NET with Server GC enabled and .NET Core with Server GC enabled.

Mutators are great when you want to have a single, global config for all benchmarks and apply given settings only to selected types. You can find out more about mutators here.

Ctrl+C

When the user:

  • presses Ctrl+C
  • presses Ctrl+Break
  • logs off
  • closes console window

We are now going to close any existing ETW session created by BenchmarkDotNet and restore console colors (read more).

Handle OutOfMemoryException more gracefully

When our benchmark hits OutOfMemoryException we print some nice explanation:

public class Program
{
    static void Main(string[] args) => BenchmarkRunner.Run<Program>();

    private List<object> list = new List<object>();

    [Benchmark]
    public void AntiPattern() => list.Add(new int[int.MaxValue / 2]);
}

OutOfMemoryException!
BenchmarkDotNet continues to run additional iterations until desired accuracy level is achieved. It's possible only if the benchmark method doesn't have any side-effects.
If your benchmark allocates memory and keeps it alive, you are creating a memory leak.
You should redesign your benchmark and remove the side-effects. You can use `OperationsPerInvoke`, `IterationSetup` and `IterationCleanup` to do that.

Trimming long strings

We used to display the values "as is" which was bad for long strings. Now the values are trimmed (more).

public class Long
{
    [Params("text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7")]
    public string Text;

    [Benchmark]
    public int HashCode() => Text.GetHashCode();
}
Method Text
HashCode text/(...)q=0.7 [86]

More features

Generic benchmarks

BenchmarkDotNet supports generic benchmarks, all you need to do is to tell it which types should be used as generic arguments (read more).

[GenericTypeArguments(typeof(int))]
[GenericTypeArguments(typeof(char))]
public class IntroGenericTypeArguments<T>
{
    [Benchmark] public T Create() => Activator.CreateInstance<T>();
}

Arguments

We now support more scenarios for passing arguments to benchmarks:

  • passing arguments to asynchronous benchmarks (more)
  • passing generic types
  • passing arguments by reference
  • passing jagged arrays (more)
  • types with implicit cast operator to stack only types can be passed as given stack-only types to Benchmarks (more)

Example:

public class WithStringToReadOnlySpan
{
    [Benchmark]
    [Arguments("some string")]
    public void AcceptsReadOnlySpan(ReadOnlySpan<char> notString)
}

Console Arguments

BenchmarkSwitcher supports various console arguments (PR), to make it work you need to pass the args to switcher:

class Program
{
    static void Main(string[] args) 
        => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}

Note: to get the most up-to-date info about supported console arguments run the benchmarks with --help.

Filter

The --filter or just -f allows you to filter the benchmarks by their full name (namespace.typeName.methodName) using glob patterns.

Examples:

  1. Run all benchmarks from System.Memory namespace: -f System.Memory*
  2. Run all benchmarks: -f *
  3. Run all benchmarks from ClassA and ClassB -f *ClassA* *ClassB*

Note: If you would like to join all the results into a single summary, you need to use --join.

Categories

You can also filter the benchmarks by categories:

  • --anyCategories - runs all benchmarks that belong to any of the provided categories
  • --allCategories- runs all benchmarks that belong to all provided categories

Diagnosers

  • -m, --memory - enables MemoryDiagnoser and prints memory statistics
  • -d, --disassm- enables DisassemblyDiagnoser and exports diassembly of benchmarked code

Runtimes

The --runtimes or just -r allows you to run the benchmarks for selected Runtimes. Available options are: Clr, Mono, Core and CoreRT.

Example: run the benchmarks for .NET and .NET Core:

dotnet run -c Release -- --runtimes clr core

More arguments

  • -j, --job (Default: Default) Dry/Short/Medium/Long or Default
  • -e, --exporters GitHub/StackOverflow/RPlot/CSV/JSON/HTML/XML
  • -i, --inProcess (Default: false) Run benchmarks in Process
  • -a, --artifacts Valid path to accessible directory
  • --outliers (Default: OnlyUpper) None/OnlyUpper/OnlyLower/All
  • --affinity Affinity mask to set for the benchmark process
  • --allStats (Default: false) Displays all statistics (min, max & more)
  • --attribute Run all methods with given attribute (applied to class or method)

Other small improvements

  • Unicode support: now you can enable support of Unicode symbols like μ or ± with [EncodingAttribute.Unicode], an example: BenchmarkDotNet.Samples.IntroEncoding (see #735)
  • Better benchmark validation (see #693, #737)
  • Improve .NET Framework version detection: now we support .NET Framework 4.7.2 (see #743)
  • OutlierModes: now it's possible to control how to process outliers, an example BenchmarkDotNet.Samples.IntroOutliers (see #766)
  • LLVM support in MonoDisassembler (see a7426e)
  • Grand API renaming we try not to change public API, but sometimes it's necessary because we want to get a consistent and understandable API in v1.0.0. (see #787)
  • Many-many small improvements and bug fixes

Milestone details

In the v0.11.0 scope, 65 issues were resolved and 34 pull requests were merged. This release includes 214 commits by 11 contributors.

Resolved issues (65)

  • #136 Fastcheck for correctness of benchmark implementations
  • #175 Add .NET Core support for Diagnostics package (assignee: @adamsitnik)
  • #368 Memory leak and crash with [Setup] (assignee: @adamsitnik)
  • #420 Make BenchmarkDotNet.Core runtime independent (assignee: @adamsitnik)
  • #464 Iteration setup / cleanup should not be called for Idle() (assignee: @adamsitnik)
  • #484 Broken HTTPS on site (assignee: @jongalloway)
  • #487 Please consider using 'µs' instead of 'us'
  • #551 List of structs and OutOfMemoryException
  • #583 BenchmarkDotNet.Samples refactoring (assignee: @AndreyAkinshin)
  • #586 IParam interface improvement (assignee: @adamsitnik)
  • #638 Config with ryujit but it doesnt actually use ryujit? (assignee: @morgan-kn)
  • #649 Searching docs leads to 404 page (assignee: @AndreyAkinshin)
  • #665 Handle OutOfMemoryException more gracefully (assignee: @adamsitnik)
  • #671 Why does BenchmarkRunner generate an isolated project per each benchmark method/job/params? (assignee: @adamsitnik)
  • #698 Port to .NET Standard 2.0, drop .NET Core 1.1 support (assignee: @adamsitnik)
  • #699 Generate one executable per runtime settings (assignee: @adamsitnik)
  • #700 Improve local CoreCLR support (assignee: @adamsitnik)
  • #701 Extend exported json file with FullName using xunit naming convention for integration purpose (assignee: @adamsitnik)
  • #710 Use DocFX as a documentation generator (assignee: @AndreyAkinshin)
  • #712 [Params] with arrays as params throws System.Reflection.TargetInvocationException (assignee: @adamsitnik)
  • #713 How to specify the invocation/launch count per type when using Config for multiple runtimes? (assignee: @adamsitnik)
  • #718 CoreRT support (assignee: @adamsitnik)
  • #719 If fail to build in Parallel due to file access issues, try to build sequentially (assignee: @adamsitnik)
  • #720 Add SummaryOrderPolicy.Declared
  • #724 Allocated Memory results are not scaled with OperationPerInvoke (assignee: @adamsitnik)
  • #726 Improve building guideline
  • #729 Handle Ctrl+C/Break (assignee: @adamsitnik)
  • #730 IterationSetup is not running before each benchmark invocation (assignee: @adamsitnik)
  • #733 IOException when running in OneDrive Folder (assignee: @adamsitnik)
  • #734 Handle missing Mono runtime more gracefully (assignee: @adamsitnik)
  • #736 Reduce number of initial pilot ops to 1 or make it configurable (assignee: @adamsitnik)
  • #738 Params string containing characters like quotes is not being escaped properly (assignee: @adamsitnik)
  • #741 Give users nice warning when T in generic benchmark is not public
  • #745 It should be possible to specify the generic arguments by using attributes
  • #747 Better docs that explain what is target/launch/iteration/invocation count (assignee: @adamsitnik)
  • #748 Very long string params/arguments should be trimmed (assignee: @adamsitnik)
  • #749 WithId(…) is ignored unless it’s at the end of the fluent calls chain. (assignee: @adamsitnik)
  • #763 Make MaxIterationCount configurable, keep current value as default (assignee: @adamsitnik)
  • #765 Add .NET Core 2.2 support (assignee: @adamsitnik)
  • #769 ArgumentsSource does not support Jagged Arrays (assignee: @adamsitnik)
  • #774 Make it possible to use Span and other ByRefLike types with implicit cast operators as benchmark argument (assignee: @adamsitnik)
  • #778 CS0104: 'Job' is an ambiguous reference between 'BenchmarkDotNet.Jobs.Job' and 'Nest.Job' (assignee: @adamsitnik)
  • #779 StackOnlyTypesWithImplicitCastOperatorAreSupportedAsArguments doesn't work on .NET Core 2.0 (assignee: @adamsitnik)
  • #787 Grand renaming
  • #793 job=core for BenchmarkSwitcher (assignee: @adamsitnik)
  • #794 Don't exclude allocation quantum side effects for .NET Core 2.0+ (assignee: @adamsitnik)
  • #795 Broken BenchmarkSwitcher (assignee: @adamsitnik)
  • #797 Allocated is not divided by OperationsPerInvoke (assignee: @adamsitnik)
  • #802 AdaptiveHistogramBuilder.BuildWithFixedBinSize error when running benchmarks (assignee: @AndreyAkinshin)
  • #804 What is the point of BuildScriptFilePath ? (assignee: @adamsitnik)
  • #809 Make it possible to configure Min and Max Warmup Iteration Count (assignee: @adamsitnik)
  • #810 handle new *Ansi events to make Inlining and TailCall Diagnosers work with .NET Core 2.2 (assignee: @adamsitnik)
  • #811 Question/Suggestion is GcStats forcing a GC.Collect when it doesn't need to (assignee: @adamsitnik)
  • #812 When will the next release be available on NuGet? (assignee: @adamsitnik)
  • #813 Problems with MemoryDiagnoserTests on Mono and .NET Core 2.0 (assignee: @adamsitnik)
  • #814 For type arguments we should display simple, not-trimmed name (assignee: @adamsitnik)
  • #816 BenchmarkDotNet.Autogenerated.csproj is not working on .NET Core 2.1 (assignee: @adamsitnik)
  • #817 Autogenerated project is missing dependencies (assignee: @adamsitnik)
  • #818 Arguments should be passed to asynchronous benchmarks (assignee: @adamsitnik)
  • #820 set DOTNET_MULTILEVEL_LOOKUP=0 when custom dotnet cli path is provided (assignee: @adamsitnik)
  • #821 ArgumentsAttribute causes an error when used with a string containing quotes (assignee: @adamsitnik)
  • #823 Allow to set multiple Setup/Cleanup targets without string concatenation (assignee: @adamsitnik)
  • #827 An easy way to run a specific benchmark class via command line (assignee: @adamsitnik)
  • #829 Error message for wrong command line filter (assignee: @adamsitnik)
  • #832 Compilation Error CS0119 with ParamsSource (assignee: @adamsitnik)

Merged pull requests (34)

Commits (214)

Contributors (11)

Thank you very much!

Additional details

Date: July 23, 2018

Milestone: v0.11.0 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.10.14

  • Per-method parameterization (Read more)
  • Console histograms and multimodal disribution detection
  • Many improvements for Mono disassembly support on Windows (Read more)
  • Many bugfixes

Milestone details

In the v0.10.14 scope, 8 issues were resolved and 11 pull requests were merged. This release includes 47 commits by 8 contributors.

Resolved issues (8)

Merged pull requests (11)

Commits (47)

Contributors (8)

Thank you very much!

Additional details

Date: April 09, 2018

Milestone: v0.10.14 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.10.13

Overview post: BenchmarkDotNet v0.10.13

Milestone details

In the v0.10.13 scope, 15 issues were resolved and 9 pull requests were merged. This release includes 50 commits by 9 contributors.

Resolved issues (15)

  • #541 Mono Support for DisassemblyDiagnoser (assignee: @morgan-kn)
  • #614 Build fails with "'Microsoft.NETCore.App', version '1.1.2' was not found" probably due to 1.1.4 runtime not being available
  • #626 Support Visual Basic project files (.vbroj) targeting .NET Core (assignee: @adamsitnik)
  • #630 Bug: Statistics.DivMean - NullReferenceException (assignee: @AndreyAkinshin)
  • #631 Bug: Generic benchmark class fails for DisassemblyDiagnoser with "Sequence contains no matching element" (assignee: @adamsitnik)
  • #632 ParamsSource no longer sorted in results (assignee: @AndreyAkinshin)
  • #634 Extend SummaryOrderPolicy (assignee: @adamsitnik)
  • #636 Unable to run Runner.exe --method MethodName (assignee: @adamsitnik)
  • #640 Disassembler fails with generic instance (assignee: @adamsitnik)
  • #643 BenchmarkDotNet should respect LangVersion project setting (assignee: @adamsitnik)
  • #644 BenchmarkDotNet.Mathematics.RankHelper again. (assignee: @AndreyAkinshin)
  • #648 BenchmarkDotNet requires dotnet cli toolchain to be installed (assignee: @adamsitnik)
  • #651 Support ANY CoreFX and CoreCLR builds (assignee: @adamsitnik)
  • #652 BenchmarkSwitcher should support generic types with parameterless public ctors (assignee: @adamsitnik)
  • #653 Proper way to run BenchmarkDotNet on macOS/Linux (assignee: @adamsitnik)

Merged pull requests (9)

  • #624 Upgrade build tools (by @Ky7m)
  • #625 Fix xunit warnings connected to usage of Assert.Equal() to check for Null (by @Ky7m)
  • #633 HostEnvironmentInfo: remove LogicalCoreCount (by @morgan-kn)
  • #637 Mono Support for DisassemblyDiagnoser #541 (by @morgan-kn)
  • #639 Portability.Cpu tests improvements (by @morgan-kn)
  • #642 sync DataContracts to CopiedDataContracts (by @morgan-kn)
  • #645 Fixing --help display for options (by @ENikS)
  • #646 Allow sorting by the Method name in DefaultOrderProvider and OrderProviderAttribute (by @ENikS)
  • #666 Plots...Examples...Added A config example in F# (by @ScottHutchinson)

Commits (50)

Contributors (9)

Thank you very much!

Additional details

Date: March 02, 2018

Milestone: v0.10.13 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.10.12

Overview post: BenchmarkDotNet v0.10.12

Highlights

  • Improved DisassemblyDiagnoser: BenchmarkDotNet contains an embedded disassembler so that it can print assembly code for all benchmarks; it's not easy, but the disassembler evolves in every release.
  • Improved MemoryDiagnoser: it has a better precision level, and it takes less time to evaluate memory allocations in a benchmark.
  • New TailCallDiagnoser: now you get notifications when JIT applies the tail call optimizations to your methods.
  • Better environment info: when your share performance results, it's very important to share information about your environment. The library generates the environment summary for you by default. Now it contains information about the amount of physical CPU, physical cores, and logic cores. If you run a benchmark on a virtual machine, you will get the name of the hypervisor (e.g., Hyper-V, VMware, or VirtualBox).
  • Better summary table: one of the greatest features of BenchmarkDotNet is the summary table. It shows all important information about results in a compact and understandable form. Now it has better customization options: you can display relative performance of different environments (e.g., compare .NET Framework and .NET Core) and group benchmarks by categories.

Milestone details

In the v0.10.12 scope, 15 issues were resolved and 10 pull requests were merged. This release includes 42 commits by 9 contributors.

Resolved issues (15)

Merged pull requests (10)

Commits (42)

Contributors (9)

Thank you very much!

Additional details

Date: January 15, 2018

Milestone: v0.10.12 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.10.11

Highlights

  • ByRef and Stack-only support (#492, sample)
  • .NET Core 2.1 support (#587)
  • Improved LINQPad support
  • Smart logic for precision in ScaledColumn (#509, #590)
  • Better macOS version detection (15d72388)
  • Minor fixes and improvements

Milestone details

In the v0.10.11 scope, 6 issues were resolved and 8 pull requests were merged. This release includes 18 commits by 8 contributors.

Resolved issues (6)

  • #509 Better formatting for the Scaled column
  • #579 Improve error message about non-optimized dependencies (assignee: @adamsitnik)
  • #580 How to get benchmarks running from LINQPad? (assignee: @adamsitnik)
  • #587 Support netcoreapp2.1 (assignee: @eerhardt)
  • #588 Broken appveyor build
  • #593 BenchmarkDotNet is not working with LinqPad (assignee: @adamsitnik)

Merged pull requests (8)

Commits (18)

Contributors (8)

Thank you very much!

Additional details

Date: December 01, 2017

Milestone: v0.10.11 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.10.10

Highlights:

  • Disassembly Diagnoser (read more here: Disassembling .NET Code with BenchmarkDotNet)
  • ParamsSources
  • .NET Core x86 support
  • Environment variables and Mono args support
  • Better environment description
  • More: additional sections in the documentation, bug fixes, build script improvements, internal refactoring.

Overview post: BenchmarkDotNet v0.10.10

Milestone details

In the v0.10.10 scope, 34 issues were resolved and 18 pull requests were merged. This release includes 95 commits by 12 contributors.

Resolved issues (34)

  • #160 Make ClrMd Source diagnoser working with new ClrMD api (assignee: @adamsitnik)
  • #167 Detect virtual machine environment (assignee: @lukasz-pyrzyk)
  • #262 Runtime knobs (assignee: @adamsitnik)
  • #310 Support 32bit benchmarks for .NET Core (assignee: @adamsitnik)
  • #350 ParamsSource (assignee: @adamsitnik)
  • #437 Add DisassemblyDiagnoser for outputting disassembled JITed code. (assignee: @adamsitnik)
  • #466 MSBuild parameters are not passed to generated benchmark project (assignee: @adamsitnik)
  • #495 Attributes put on base methods are not considered in derived class (assignee: @lukasz-pyrzyk)
  • #500 Borken compilation for net46 projects when .NET Framework 4.7 is installed (assignee: @adamsitnik)
  • #505 JsonExporterBase doesn't include MemoryDiagnoser stats in output
  • #511 [bug] Bug in GetTargetedMatchingMethod() logic
  • #513 IterationSetup not run in Job.InProcess
  • #516 Get a compilation error "CS1009: Unrecognized escape sequence" when using verbatim strings
  • #519 BenchmarkSwitcher.RunAllJoined throws InvalidOperationException (assignee: @AndreyAkinshin)
  • #526 Remove project.json support (assignee: @adamsitnik)
  • #529 No namespace in export filenames can lead to data loss
  • #530 Build error on Appveyor with recent changes.
  • #533 When I clone, build, and run BenchmarkDotNet.Samples I get an error
  • #534 Allow the users to compare 32 vs 64 RyuJit for .NET Core (assignee: @adamsitnik)
  • #535 No way to set RuntimeFrameworkVersion in multiple-version config (assignee: @adamsitnik)
  • #536 Strange disassembly ordering/truncation (assignee: @adamsitnik)
  • #537 Can't benchmark a netstandard2.0 project (assignee: @adamsitnik)
  • #538 Duplicate using causing benchmark not to work (assignee: @adamsitnik)
  • #539 Target .NET Core 2.0 to take advantage of the new APIs (assignee: @adamsitnik)
  • #540 Artifacts for disassembler projects (assignee: @adamsitnik)
  • #542 Problems with Disassembler + Job.Dry (assignee: @adamsitnik)
  • #555 Test "CanDisassembleAllMethodCalls" fails on Ubuntu (assignee: @adamsitnik)
  • #556 Table in report is broken in VSCode markdown viewer (assignee: @adamsitnik)
  • #558 Warn the users when running Benchmarks from xUnit with shadow copy enabled (assignee: @adamsitnik)
  • #559 DissassemblyDiagnoser jit/arch info seems to be wrong (assignee: @adamsitnik)
  • #561 Strange behaviour when benchmark project is build in debug mode (assignee: @adamsitnik)
  • #562 DisassemblyDiagnoser crashes on overloaded benchmark (assignee: @adamsitnik)
  • #564 [Bug] Benchmarking a method doesn't run global setup when filter is applied (assignee: @adamsitnik)
  • #571 Allow users to use non compile-time constants as Parameters (assignee: @adamsitnik)

Merged pull requests (18)

Commits (95)

Contributors (12)

Thank you very much!

Additional details

Date: November 03, 2017

Milestone: v0.10.10 (List of commits)

NuGet Packages:

BenchmarkDotNet v0.10.9

Milestone details

In the v0.10.9 scope, 13 issues were resolved and 14 pull requests were merged. This release includes 37 commits by 10 contributors.

Resolved issues (13)

  • #380 Problem running benchmark due to "could not copy" during build (assignee: @adamsitnik)
  • #390 Crashing benchmark (assignee: @adamsitnik)
  • #426 Migrate from custom build scripts to Cake (C# Make)
  • #448 Detect correct version of .NET Core (assignee: @AndreyAkinshin)
  • #453 MemoryDiagnoser and JsonExporter
  • #469 [Suggestion] Specify Setup per benchmark (assignee: @AndreyAkinshin)
  • #474 PlatformNotSupportedException when reading ProcessorAffinity on non-Windows platforms (assignee: @AndreyAkinshin)
  • #479 Invalid C# code generated for valid F# identifiers (assignee: @adamsitnik)
  • #481 Iteration cleanup runs before the benchmark (assignee: @AndreyAkinshin)
  • #482 Benchmark seems to hang when no logger is defined (assignee: @adamsitnik)
  • #490 BDN.Generated.exe is locking files when killed with ctrl+c (assignee: @adamsitnik)
  • #491 UnauthorizedAccessException preventing report to be written (assignee: @adamsitnik)
  • #499 Opting into app-compat switches in a benchmark doesn't work (assignee: @adamsitnik)

Merged pull requests (14)

Commits (37)

Contributors (10)

Thank you very much!

Additional details

Date: July 28, 2017

Milestone: v0.10.9

NuGet Packages:

BenchmarkDotNet v0.10.8

Milestone details

In the v0.10.8 scope, 5 issues were resolved and 3 pull requests were merged. This release includes 8 commits by 4 contributors.

Resolved issues (5)

  • #157 Implement export to xml
  • #349 What the report title and value means? (assignee: @AndreyAkinshin)
  • #450 [Minor feature request] Please make GcStats.AllocationQuantum public
  • #459 [Question] What does the unit of measurement us stand for
  • #461 .NET Framework 4.7 support (assignee: @adamsitnik)

Merged pull requests (3)

Commits (8)

Contributors (4)

Thank you very much!

Additional details

Date: June 09, 2017

Milestone: v0.10.8

NuGet Packages:

BenchmarkDotNet v0.10.7

  • LINQPad support (5.22.05+) (#66, #445)
  • Benchmark filters and categories (#248)
  • Updated setup/cleanup attributes: [GlobalSetup], [GlobalCleanup], [IterationSetup], [IterationCleanup] (#270, #274, #325, #456)
  • Better Value Types support (afa803d0)
  • Building Sources on Linux: it's possible to build the solution (with unloaded F#/VB projects), run samples (for both net46/netcoreapp1.1), run unit tests (for netcoreapp1.1 only)
  • Fix minor bugs in JsonExporter (#451)

Milestone details

In the v0.10.7 scope, 6 issues were resolved and 1 pull requests were merged. This release includes 24 commits by 4 contributors.

Resolved issues (6)

  • #66 Friendliness to LinqPad (assignee: @adamsitnik)
  • #248 Support a "category" attribute for selecting benchmarks (assignee: @AndreyAkinshin)
  • #270 Add support for Cleanup and Setup between benchmarks (assignee: @AndreyAkinshin)
  • #274 Support for run-once Setup and Clean-up with Parameters available (assignee: @AndreyAkinshin)
  • #325 Setup & Cleanup versions of attribute which would run before/after each benchmark iteration (assignee: @AndreyAkinshin)
  • #445 Missing reference to Microsoft.CodeAnalysis.CSharp when using BenchmarkDotNet in Linqpad (assignee: @adamsitnik)

Merged pull requests (1)

Commits (24)

Contributors (4)

Thank you very much!

Additional details

Date: June 05, 2017

Milestone: v0.10.7

Overview post: https://aakinshin.net/posts/bdn-v0_10_7/

NuGet Packages:

BenchmarkDotNet v0.10.6

  • Removed buggy allocation from Engine which was spoiling the results of MemoryDiagnoser for micro benchmarks. This part of the code is now guarded with very strict integration tests, it should never happen again. We now also exclude the side effects of the Allocation Quantum. This bug was serious, you must update to 0.10.6 (#439)
  • Support of the PackageTargetFallback setting which allows to reference components that target old framework monikers (like dotnet5.4 or portable-net45+win8) (#438)
  • Added InstructionRetiredPerCycleColumn which shows up automatically when HardwareCounter.InstructionRetired and HardwareCounter.TotalCycles are used.
  • Support benchmark classes without namespace (#446)
  • Fix problem with RPlotExporter and quoted directories in %PATH% (#446)
  • Show Windows brand version in summary

Milestone details

In the v0.10.6 scope, 3 issues were resolved and 1 pull requests were merged. This release includes 11 commits by 3 contributors.

Resolved issues (3)

Merged pull requests (1)

Commits (11)

Contributors (3)

Thank you very much!

Additional details

Date: May 12, 2017

Milestone: v0.10.6

NuGet Packages:

BenchmarkDotNet v0.10.5

  • Fixed SizeUnit presentation in the summary table (#434)
  • In MemoryDiagnoser, now 1kB = 1024B (instead of 1000 in v0.10.4) (#434)
  • Fix false allocations detection (#436 9b44de70)
  • Hide ScaledSD column for small values (da857ad7)
  • Autoselecting amount of digits after the decimal point (#404)

Milestone details

In the v0.10.5 scope, 3 issues were resolved and 2 pull requests were merged. This release includes 16 commits by 4 contributors.

Resolved issues (3)

Merged pull requests (2)

Commits (16)

Contributors (4)

Thank you very much!

Additional details

Date: April 26, 2017

Milestone: v0.10.5

NuGet Packages:

BenchmarkDotNet v0.10.4

  • New logo
  • Update to Roslyn 2.0, drop .NET 4.5 support (#303)
  • Initial support of HardwareCounters (Windows only)
  • Initial experimental support of in-process benchmarks
  • Optional configs for BenchmarkSwitcher (#391, #392)
  • Host API interface (#356)
  • Improved measurements for async benchmarks (#415)
  • Improved precision level (MinIterationTimes is 500ms instead of 200ms; introduced AccuracyMode.MaxAbsoluteError and AccuracyMode.MaxRelativeError instead of AccuracyMode.MaxStdErrRelative; logic which select amount of iterations uses confidence intervals instead of standard errors; the Error column (half of CI99.9%) is shown by default instead of StdErr)
  • Introduced ISummaryStyle, raw data in CSV reports (#118, #146, #396)
  • Handle cases when report files are existed and locked (#414, #416)
  • MarkdownExporter right-justifies numeric columns (#421)
  • Better colors for console output (#376)
  • Column legends
  • Add information about CPU microarchitecture for well-known processors to summary
  • Fix AssemblyInformationalVersionAttribute (#382)
  • Fix incorrect method filtering in BenchmarkSwitcher (#365)
  • Fix OS Version in Summary for Windows 10 (#351)
  • Fix OS Version on Mono
  • Fix --class and --method filtering (#249)
  • Fix --exporters option (#189)
  • Fix escaping logic in CsvExporter (#294, #409)
  • Fix MacOS detection
  • Minor bugfixes and API improvements

Milestone details

In the v0.10.4 scope, 23 issues were resolved and 14 pull requests were merged. This release includes 103 commits by 9 contributors.

Resolved issues (23)

  • #118 Raw data in CSV reports (assignee: @AmadeusW)
  • #146 Ability to specify units / easier comparison (assignee: @AmadeusW)
  • #159 Warn user if no Columns were defined (assignee: @adamsitnik)
  • #189 --exporters option appears not to be working (assignee: @adamsitnik)
  • #249 --class and --method should combine as "AND" filtering (assignee: @adamsitnik)
  • #294 [Suggestion] CSVHelper.Escape() method should check for actual separator value (assignee: @alinasmirnova)
  • #303 Update to Roslyn 2.0 when RTM is shipped to nuget.org (assignee: @adamsitnik)
  • #351 Fix OS Version in Summary for Windows 10 (assignee: @adamsitnik)
  • #352 Troubles with CoreJob on Linux (assignee: @adamsitnik)
  • #365 [Minor bug] Benchmark switcher: incorrect method filtering (assignee: @adamsitnik)
  • #376 Pick better background colors for output (assignee: @AndreyAkinshin)
  • #382 AssemblyInformationalVersion doesn't work (assignee: @AndreyAkinshin)
  • #388 Precise Machine Counter Diagnoser (assignee: @adamsitnik)
  • #391 BenchmarkSwitcher should take an optinal IConfig
  • #393 Troubles with ClrJob in .NET Core applications (assignee: @adamsitnik)
  • #395 Could not load file or assembly 'System.Reflection.Metadata' (assignee: @adamsitnik)
  • #401 Exceptions in Roslyn.Builder (assignee: @adamsitnik)
  • #406 BenchmarkDotNet with netcoreapp2.0 requires using RuntimeFrameworkVersion directly in the project file (assignee: @adamsitnik)
  • #410 Troubles with Classic applications on nightly BenchmarkDotNet (assignee: @adamsitnik)
  • #412 HardwareCounter.InstructionRetired failing with ArgumentNullException. Build 82 (assignee: @adamsitnik)
  • #415 Allocations for async methods measures BenchmarkDotNet (assignee: @adamsitnik)
  • #419 Suspicious warnings about MemoryMappedFiles (assignee: @adamsitnik)
  • #424 Make InliningDiagnoser filtering more flexible (assignee: @adamsitnik)

Merged pull requests (14)

Commits (103)

Contributors (9)

Thank you very much!

Additional details

Date: April 21, 2017

Milestone: v0.10.4

NuGet Packages:

BenchmarkDotNet v0.10.3

  • New .csprojs support for .NET Core. Also for F# (#366)!
  • New plots and RPlotExporter (density plots for each job; cumulative mean plots)
  • Fixed exporter order (now RPlotExporer uses the actual measurements instead of previous version)
  • Xplat improvments in RuntimeInformation
  • Introduced RunStrategy.Monitoring
  • Possibility to set custom path for Mono (#306)
  • Possibility to set any .NET Core version >= 1.1 (#336)
  • MemoryDiagnoser is now disabled by default (Breaking changes!!) (#369)

Milestone details

In the v0.10.3 scope, 10 issues were resolved and 2 pull requests were merged. This release includes 79 commits by 3 contributors.

Resolved issues (10)

  • #300 Switch back from project.json and xproj to csproj, support dotnet cli preview 3 (assignee: @adamsitnik)
  • #306 Custom path for mono (assignee: @adamsitnik)
  • #320 Results table should be a GitHub Flavored Markdown table (assignee: @alinasmirnova)
  • #322 First benchmark always fails when running on .NET Core with -c release (assignee: @adamsitnik)
  • #336 allow the users to choose the target .NET Core version (1.2, 2.0 etc) (assignee: @adamsitnik)
  • #366 Support the new .fsprojs targetting .NET Core (F# + .NET Core + MSBuild) (assignee: @adamsitnik)
  • #369 Consider to disable MemoryDiagnoser by default (assignee: @adamsitnik)
  • #372 Troubles with ClrJob from CoreCLR project (assignee: @adamsitnik)
  • #374 BenchmarkDotNet doesn't understand netcoreapp2.0 (assignee: @adamsitnik)
  • #375 Troubles with dotnet pack (assignee: @adamsitnik)

Merged pull requests (2)

Commits (79)

Contributors (3)

Thank you very much!

Additional details

Date: March 01, 2017

Milestone: v0.10.3

NuGet Packages:

BenchmarkDotNet v0.10.2

  • Closed #307: culture invariant statistics output
  • Closed #321: persist optimized, auto-generated dll compiled from url/plain code
  • Closed #322: always restore the console foreground color
  • Closed #337: Better detection of Rscript.exe in RPlotExporter
  • Closed #345: fix bug in WelchTTestPValueColumn for DryJob
  • VS 2017 compatibility fix
  • fix bold markup for Atlassian exporter
  • Improved precision of nanobenchmarks
  • Minor infrastructure changes and misc fixes

Milestone details

In the v0.10.2 scope, 10 issues were resolved and 3 pull requests were merged. This release includes 30 commits by 8 contributors.

Resolved issues (10)

  • #295 Fix CLS-compliant warnings in Diagnostics (assignee: @adamsitnik)
  • #307 Output: interpolated strings & culture (assignee: @alinasmirnova)
  • #319 [Request] some API to public? (assignee: @adamsitnik)
  • #321 BenchmarkRunner.RunUrl throws BenchmarkSystem.IO.FileNotFoundException (assignee: @adamsitnik)
  • #327 Unable to use ClassicToolchain in explicit way (assignee: @adamsitnik)
  • #332 default color of terminal is changed after the run is completed (assignee: @adamsitnik)
  • #335 Support benchmarking startup performance (assignee: @AndreyAkinshin)
  • #337 Problematic mechanism/docs for locating Rscript.exe
  • #340 [FeatureRequest] Enable Characteristic-based properties for non-job types. (assignee: @ig-sinicyn)
  • #345 Fail to run IntroAdvancedStats in dry mode (assignee: @AndreyAkinshin)

Merged pull requests (3)

Commits (30)

Contributors (8)

Thank you very much!

Additional details

Date: January 21, 2017

Milestone: v0.10.2

NuGet Packages:

BenchmarkDotNet v0.10.1

  • MemoryDiagnoser got improved. The changes:
    • Memory Diagnoser is now part of BenchmarkDotNet.Core.dll, and it's enabled by default
    • MemoryDiagnoser is 100% accurate about allocated memory when using default settings or Job.ShortRun or any longer job. (see #284)
    • Memory Diagnoser no longer includes allocations from Cleanup/Setup methods (see #186)
    • the results are now scaled so they are stable across the runs. (see #133)
  • .NET Core 1.1+ support, we no longer support 1.0, we target netcoreapp1.1 now. Reason: we wanted to use GC.GetAllocatedBytesForCurrentThread in MemoryDiagnoser which is available only in 1.1+
  • Improved information about environment in summary
  • Minor bugfixes

Milestone details

In the v0.10.1 scope, 9 issues were resolved and 2 pull requests were merged. This release includes 38 commits by 2 contributors.

Resolved issues (9)

Merged pull requests (2)

Commits (38)

Contributors (2)

Thank you very much!

Additional details

Date: December 04, 2016

Milestone: v0.10.1

NuGet Packages:

BenchmarkDotNet v0.10.0

  • Now BenchmarkDotNet is a part of .NET Foundation
  • Job and Column API refactoring (see new documentation)
  • Measurement engine improvements
  • Horology enhancement (see TimeInterval and Frequency)
  • Introduced RankColumn which is based on WelchTTest (see 157aabc3)
  • JsonExporters refactoring (see the Exporters/Json section in the documentation)
    • Renamed JsonExporters classed and attributes
    • JsonExporters with custom settings
    • JsonExporters now includes information about the target type namespace (see #246).
  • Add JetBrains.Annotations (see #253)
  • RFC 4180 support in CSV exporters (see #241)
  • Many bugfixes

Milestone details

In the v0.10.0 scope, 19 issues were resolved and 7 pull requests were merged. This release includes 85 commits by 7 contributors.

Resolved issues (19)

  • #30 Better information when we haven't got a valid measurement
  • #121 Strange # of "Launches" chosen with Count.Auto
  • #154 PathTooLong exception on custom config
  • #185 Report if difference between 2 benchmarks is statistically significance
  • #241 .csv results output does not play well with Excel or Google Sheets
  • #244 DefaultConfig StatisticColumn values
  • #246 No namespace information?
  • #265 Add ability to specify that benchmark requires STAThread
  • #266 Don't assume that TargetType has reference to BenchmarkDotNet (assignee: @adamsitnik)
  • #268 Print runtime of child processes in summary
  • #271 Params attribute does not handle nullable types
  • #272 [Setup] error when doing inheritance
  • #276 System.EntryPointNotFoundException
  • #280 Cannot run on OSX / Mono (System.Xml.XmlException: Root element is missing)
  • #281 Results are exported twice for single run
  • #288 IdleWarmup running off forever
  • #291 [Bug] Incorrect results for targetCount:Auto
  • #292 Support for Beta versions
  • #296 [BUG] NRE in OutliersAnalyser

Merged pull requests (7)

Commits (85)

Contributors (7)

Thank you very much!

Additional details

Date: November 10, 2016

Milestone: v0.10.0

NuGet Packages:

Online Documentation: https://dotnet.github.io/BenchmarkDotNet/

BenchmarkDotNet v0.9.9

  • Attribute config style (see #166)
  • Online documentation (see #219)
  • Mono LLVM support (see #226)
  • Async method support (see #236)
  • NuGet packages and repo layout restructuring (see #225, #228)
  • [Cleanup] attribute (see #215)
  • New statistics columns: Skewness, Kurtosis, WelchTTestPValue, Improved math for the Scaled column
  • Now current default branch is master
  • Minor improvements and bug fixes

Milestone details

In the v0.9.9 scope, 14 issues were resolved and 1 pull requests were merged. This release includes 46 commits by 4 contributors.

Resolved issues (14)

  • #166 Suggestion: Attribute Config style (assignee: @AndreyAkinshin)
  • #215 Allow a [Cleanup] method to be specified
  • #219 Online documentation and API reference
  • #223 Add support for System.Type in Params
  • #224 [Breaking change] BenchmarkAttribute become sealed.
  • #225 Refactoring the roslyn dependency into another package
  • #226 Support Mono/LLVM as a runtime/jit
  • #227 Suggestion: change default branch (assignee: @AndreyAkinshin)
  • #228 Suggestion: clean up the root folder
  • #231 Add references to default framework assemblies (System.Runtime etc)
  • #232 Make all tests use OutputLogger (assignee: @adamsitnik)
  • #235 Don't show non-error output of dotnet cli (assignee: @adamsitnik)
  • #236 Possibility to benchmark asynchronous methods (assignee: @adamsitnik)
  • #240 Total Time reports hour rounded up

Merged pull requests (1)

Commits (46)

Contributors (4)

Thank you very much!

Additional details

Date: August 18, 2016

Milestone: v0.9.9

NuGet Packages:

Online Documentation: https://perfdotnet.github.io/BenchmarkDotNet/

BenchmarkDotNet v0.9.8

  • CoreCLR RTM support (see #216). Breaking change: we have dropped dnx451 and dnxcore50 support.
  • Migration from MSBuild to Roslyn, which supports Mono on Linux and MacOS (see #149). Breaking change: we have dropped .NET 4.0 support.
  • Ability to manage GC mode: turn on/off the Server/Concurrent GC modes, extend to CPU groups, set gcAllowVeryLargeObjects and avoid BenchmarkDotNet from forcing GC.Collect (see #188, #76 and #211)
  • Support CopyToOutput (see #212). Now you can use native dependencies as well as custom files.
  • Copying custom settings from app.config (see #108). It means we support assembly binding redirects as well as custom connection strings etc.
  • AsciiDocExporter (see #169)
  • Framework setting in Job has been removed (see #194)
  • Minor bugfixes and improvements

Milestone details

In the v0.9.8 scope, 21 issues were resolved and 1 pull requests were merged. This release includes 69 commits by 5 contributors.

Resolved issues (21)

  • #57 Make Benchmark, Setup and Params attribute sealed in explicit way
  • #76 Allow users to set gcAllowVeryLargeObjects for Runtime Settings (assignee: @AndreyAkinshin)
  • #77 Implement a C.I build (assignee: @mattwarren)
  • #108 Copy custom setting from app.config
  • #131 [Mono] BenchmarkDotNet doesn't work on Mac OS
  • #149 Migrate from MSBuild to Roslyn
  • #174 NRE in Summary indexer property.
  • #176 Split and simplify printed summaries
  • #188 Ability to manage GC mode: turn on/off the Server/Concurrent GC modes and extend to CPU groups
  • #191 MSBuild dependency - best way of fixing?
  • #194 Framework settings in Jobs (assignee: @mattwarren)
  • #196 Allow specifying a gist url to RunUrl
  • #197 Regression in 0.9.7: --help option fails under dotnet run
  • #203 Third-party libraries must be explicitly included in test context to be loaded by runner
  • #209 Fix appveyor bug
  • #211 Possibility to turn off GC.Collect after each benchmark run (assignee: @adamsitnik)
  • #212 Support CopyToOutput
  • #214 Benchmark ignores binding redirects
  • #216 Update to .NET Core RTM (assignee: @adamsitnik)
  • #218 Errors in BenchmarkDotNet.Samples.FSharp.Core/projects.json
  • #222 A problem with System.Threading.Tasks

Merged pull requests (1)

Commits (69)

Contributors (5)

Thank you very much!

Additional details

Milestone: v0.9.8

Date: July 07, 2016

NuGet Packages:

BenchmarkDotNet v0.9.7

  • .NET Core RC2 support (see #187)
  • Bugfixes

Milestone details

In the v0.9.7 scope, 8 issues were resolved and 0 pull requests were merged. This release includes 27 commits by 3 contributors.

Resolved issues (8)

  • #168 Unable to run tests locally
  • #170 Error handling in the spawned Benchmark process (assignee: @mattwarren)
  • #172 Ensure CsvMeasurementsExporter is enabled when RPlotExporter is used (assignee: @mattwarren)
  • #179 Job.GetAllProperties(): old property names are used
  • #181 CompositeValidator: some validators will be skipped
  • #183 [Suggestion] Make ValidationError public
  • #187 .NET Core RC2 (assignee: @adamsitnik)
  • #192 Avoid creating .cs files at execution time

Merged pull requests (0)

Commits (27)

Contributors (3)

Thank you very much!

Additional details

Milestone: v0.9.7

Date: May 29, 2016

NuGet Packages:

BenchmarkDotNet v0.9.6

  • Added Percentiles (see #164)
  • Added support for Json export (see #84)
  • Bugfixes

Milestone details

In the v0.9.6 scope, 11 issues were resolved and 2 pull requests were merged. This release includes 40 commits by 5 contributors.

Resolved issues (11)

  • #100 Code generation doesn't support generic classes
  • #112 Generic benchmark classes are not supported
  • #140 Readd an ability to define and to use custom Toolchain
  • #141 [Request for comments] Assembly-level config attribute?
  • #151 Crash during benchmark with baseline
  • #152 MarkdownExporter.Default.ExportToLog fails with NRE for Summary with Critical Validation Errors
  • #153 ManualConfig.Add(IConfig config) does not add the validators.
  • #156 Cleanup benchmark folders
  • #158 BaselineDiffColumn: NullReferenceException if one of the benchmark methods was failed.
  • #161 Build warnings after update to 0.9.5
  • #171 Problem with ExceptionDispatchInfo (assignee: @AndreyAkinshin)

Merged pull requests (2)

  • #138 Percentiles added into Statistics, StatisticColumn, BaselineDiffColumn (by @ig-sinicyn)
  • #164 Percentiles added into Statistics, StatisticColumn, BaselineDiffColumn (by @ig-sinicyn)

Commits (40)

Contributors (5)

Thank you very much!

Additional details

Milestone: v0.9.6

Date: May 11, 2016

NuGet Packages:

BenchmarkDotNet v0.9.5

  • Added validators, JitOptimizationsValidator detects all non-optimzied dlls that were referenced #134
  • Strong naming #101
  • Add IOrderProvider #107
  • Putting all the generated artifacts in a separate folder: ./BenchmarkDotNet.Artifacts/results and ./BenchmarkDotNet.Artifacts/bin #94
  • Printing dotnet cli version for .NET Core and Dnx451, informing user when not installed. Closed #128
  • Supporting assembly redirects #67
  • Changed used msbuild version: 12 for .NET 4.5 (VS 2013), 14 for .NET 4.6 (VS 2015). Closed #132 and #137
  • Switched to new ‘dotnet’ target framework monikers (dotnet5.4 instead of dnxcore50), why
  • dnx452, dnx46, net462 support added
  • Executing single Benchmark for multiple Runtimes also with Diagnoser attached (see #117)
  • Misc minor changes

Milestone details

In the v0.9.5 scope, 13 issues were resolved and 0 pull requests were merged. This release includes 45 commits by 4 contributors.

Resolved issues (13)

  • #67 F# requiring assembly binding redirects for FSharp.Core
  • #94 Put all the generated artifacts in a separate folder
  • #101 Strong naming?
  • #107 Specify benchmark method order
  • #122 Reports: Move params columns next to Method column
  • #128 Print dotnet cli version in EnvironmentInfo
  • #129 Improve DnxAndCoreTests
  • #132 [Bug] FileNotFoundException On 0.9.4.
  • #134 [Feature request] Release builds only?
  • #137 Brand new machine with VS 2015 only b0rks
  • #142 [Suggestion] BenchmarkDotNet.Analyzers.IAnalyser - use same spelling for namespace and type?
  • #148 Crash on [Params] with a string value that contains an invalid path char
  • #150 Declaring nested enums and using them as parameter value make incorrectly generated code/

Merged pull requests (0)

Commits (45)

Contributors (4)

Thank you very much!

Additional details

Milestone: v0.9.5

Date: May 02, 2016

NuGet Packages:

BenchmarkDotNet v0.9.4

  • Improved messages about error in benchmarks, see #104
  • Natural sort order, see #92, #95, #97
  • Improved double/float/decimal/enum support for Params, see #96, #105, #116
  • Now environment info includes information about HardwareTimerKind and JitModules
  • Added DryConfig
  • Improved export performance, closed #119, merged #124
  • Better cmd-line discoverability (see #78), e.g. run Benchmark.exe --help and some useful information will be printed
  • Supporting all kinds of references for generated project (exact version, custom paths, GAC, sub-folders, dependent assemblies not copied), closed #41, #49, #72, #123, merged #125
  • Friendliness to LinqPad restored, closed #66, merged #125

Milestone details

In the v0.9.4 scope, 13 issues were resolved and 2 pull requests were merged. This release includes 55 commits by 4 contributors.

Resolved issues (13)

  • #41 Seems, not supported "sub-folder"
  • #49 Dependent assemblies are not copied or added to the project file.
  • #72 Referenced assembly dll-file (directly via file) not referenced in generated Program.csproj
  • #78 Better command line discoverability (assignee: @mattwarren)
  • #92 Results in the R graphs aren't displayed in a "Natural Sort Order"
  • #95 Results should preserve the order of param values definition
  • #96 Implement enums as valid Param for test
  • #97 Params changes an order
  • #104 System.InvalidOperationException: StatSummary: Sequence contains no elements
  • #105 Params Attribute bug with float type
  • #116 Issue when ParamAttribute decorated property is double and current system culture has comma as decimal separator
  • #119 For large benchmarks report exporting is very, very slow and has a massive Gen2 heap
  • #123 Can not run benchmark that references custom framework library (like WindowsBase)

Merged pull requests (2)

Commits (55)

Contributors (4)

Thank you very much!

Additional details

Milestone: v0.9.4

Date: March 24, 2016

NuGet Packages:

BenchmarkDotNet v0.9.3

  • CoreCLR support (Closed #52, Merged #113)

Milestone details

In the v0.9.3 scope, 2 issues were resolved and 1 pull requests were merged. This release includes 12 commits by 2 contributors.

Resolved issues (2)

  • #52 CoreCLR Compatibility
  • #114 Update NETStandard.Library dependency

Merged pull requests (1)

Commits (12)

Contributors (2)

Thank you very much!

Additional details

Milestone: v0.9.3

Date: March 13, 2016

NuGet Packages:

BenchmarkDotNet v0.9.2

  • Dnx451 support (Closed #51, Merged #87)

Milestone details

In the v0.9.2 scope, 1 issues were resolved and 1 pull requests were merged. This release includes 48 commits by 2 contributors.

Resolved issues (1)

  • #51 DNX Compatibility

Merged pull requests (1)

Commits (48)

  • f25427 road to DNX: part I: moving from csproj to xproj (by @adamsitnik)
  • 2fe5cf road to DNX: part II: added dnx451 target (DNX SDK running on .Net 4.5.1) (by @adamsitnik)
  • 6b4400 road to DNX: part II: the moment when Unit Test has shown up in VS! (by @adamsitnik)
  • c97792 road to DNX: part II: able to debug Samples (by @adamsitnik)
  • f901d6 road to DNX: part II: the moment when Integration Test has shown up in VS! (by @adamsitnik)
  • f46296 road to DNX: part II: including *.txt files as resources, excluding auto-gene... (by @adamsitnik)
  • cfc1b6 updated gitignore to exclude files created during integration tests run (by @adamsitnik)
  • 60b343 Road to DNX: benchmark execution : building project.json instead of .csproj (by @adamsitnik)
  • 0d41cb Road to DNX: benchmark execution : compiling project.json with Microsoft.Dnx.... (by @adamsitnik)
  • 31bc59 updated versions in .json files after sync with master, (by @adamsitnik)
  • c8e826 road to DNX: compilation: adding MetadataReferences for dlls required to comp... (by @adamsitnik)
  • 2ebe6c road to DNX: compilation: adding executing assembly as dependency to project.... (by @adamsitnik)
  • 76a74e Road to DNX: removing dependencies to BenchmarkDotNet.Diagnostics for DNX451 ... (by @adamsitnik)
  • 40419a road to DNX: handling "nuget-like" package versions that can contain text (as... (by @adamsitnik)
  • 8f06a7 it should have never happened but it does when debugging: DirectoryNotFoundEx... (by @adamsitnik)
  • 7c2a96 Road to DNX: reusing MS dnu to restore and build. Simple solution that just w... (by @adamsitnik)
  • e51d4d Road to DNX: alternative to MS dnu. reuses nuget and roslyn but as for now it... (by @adamsitnik)
  • ada7cb Road to DNX: replacing dnu build with dnx run. +putting files in a folder tha... (by @adamsitnik)
  • 18e969 road to DNX: referencing the right thing, bitness the same as hosting process (by @adamsitnik)
  • 8e870c road to DNX: setting the compiler EXPLICIT to finally make it WORKING (at lea... (by @adamsitnik)
  • 466d13 Road to DNX: removing failed PoC,dependencies cleanup, added some comments (by @adamsitnik)
  • 5b3675 Road to DNX: removing dependencies to MSBuild for DNX target, some project.js... (by @adamsitnik)
  • 3e65e8 road to DNU: reference project during development, but package when released (by @adamsitnik)
  • 617a61 road to DNX: logging output from dnu restore/dnx run + default timeout (by @adamsitnik)
  • 075cdc road to DNX: new value for toolchain enum: DNX451 (by @adamsitnik)
  • bd3fea road to DNX: being able to debug NET40 from VS (by @adamsitnik)
  • da5a9a road to DNX: copying all files that used to be copied in ".csproj times" (by @adamsitnik)
  • 5d8717 road to DNX: new debug profile with DNX trace mode ON, use when troubleshooti... (by @adamsitnik)
  • 8272f6 road to DNX: fixing tests (by @adamsitnik)
  • 96bcf8 road to DNX: make sure that our child process get the right priority and affi... (by @adamsitnik)
  • cd0ba8 road to DNX: respecting specified benchmark processor architecture (by @adamsitnik)
  • b1eb28 road to DNX: added all output files from integration tests to .gitignore (by @adamsitnik)
  • aa62d1 fix for 4.0 (was passing arguments in wrong order) (by @adamsitnik)
  • 0c1251 merge (by @adamsitnik)
  • 46cd5e F# support, limited to existing tools possibilities (can not run from VS, onl... (by @adamsitnik)
  • 74020c Merge remote-tracking branch 'upstream/master' (by @adamsitnik)
  • 3402f0 replacing dnx with dotnet cli (by @adamsitnik)
  • d634f4 road to DNX: final cleanup (by @adamsitnik)
  • 24cd3a removing project.lock.json files (by @adamsitnik)
  • 80becb removing all .csproj & packages.config file + keeping only single .sln file (by @adamsitnik)
  • 9633d5 removing nuspec (now auto-generated by VS based on project.json) + version in... (by @adamsitnik)
  • 617d82 running Classic Framework tests from console + minor cleanup (by @adamsitnik)
  • fc9d98 DNX: running once compiled assembly directly without dotnet cli (perf+abble t... (by @adamsitnik)
  • 8adad8 dnx: Diagnosers support. Currently only these which do not need umanaged libs... (by @adamsitnik)
  • 34c3c9 DNX: changed folder of the auto-generated files to benchmark-specific + samples (by @adamsitnik)
  • 9cf009 DNX: description for development (by @adamsitnik)
  • 43af33 update to dotnet cli changes: output path and exit codes , now we set output... (by @adamsitnik)
  • f59d3e Merge pull request #87 from adamsitnik/master (by @AndreyAkinshin)

Contributors (2)

Thank you very much!

Additional details

Milestone: v0.9.2

Date: March 5, 2016

NuGet Packages:

BenchmarkDotNet v0.9.1

Milestone details

In the v0.9.1 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 5 commits by 2 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (5)

Contributors (2)

Thank you very much!

Additional details

Date: February 10, 2016

NuGet Packages:

BenchmarkDotNet v0.9.0

  • New API
  • Autodetermination of amount iteration for warmup/target idle/main iterations, duration of iteration, amount of CLR launches.

Milestone details

In the v0.9.0 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 34 commits by 4 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (34)

Contributors (4)

Thank you very much!

Additional details

Date: February 9, 2016

NuGet Packages:

BenchmarkDotNet v0.8.2

  • Changes in the Summary table
    • Summary table now supports ResultExtenders that can add new column to the table
    • Now we use StandardError (aka Error) as the main accuracy metric
    • Columns op/s, StdDev are disabled by default (you can add it via ResultExtenders)
  • Statistic improvements, now you have detailed statistic in the console log like follows:
Mean = 118.5298 us, StdError = 1.2863 us (N = 30, StdDev = 7.0454 us)
Min = 109.1602 us, Median = 117.1794 us, Max = 132.5764 us
IQR = 10.1244 us, LowerFence = 98.0834 us, UpperFence = 138.5810 us
ConfidenceInterval = [116.0086 us; 121.0510 us] (CI 95%)
  • Added the Baseline feature, see #64
  • Export improvements, now you have files <BenchmarkName>-report-github.md, <BenchmarkName>-report-stackoverflow.md for easy publishing results on GitHub and StackOverflow.
  • Basic plotting. Added new BenchmarkRPlotExporter that creates BuildPlots.R in the bin directory. It is an R script that generates boxplot and barplot for your benchmarks (you should have installed R with defined R_HOME environment variable)
  • Updated environment info
    • Added Stopwatch Frequency and Resolution
    • Split common benchmark properties (like Mode, Platform, Runtime) in several lines (3 properties per line)
  • Log improvements: add total time, statistics, list of exported files
  • Bug fixes

Milestone details

In the v0.8.2 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 34 commits by 3 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (34)

Contributors (3)

Thank you very much!

Additional details

Date: January 19, 2016

NuGet Packages:

BenchmarkDotNet v0.8.1

Milestone details

In the v0.8.1 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 12 commits by 2 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (12)

Contributors (2)

Thank you very much!

BenchmarkDotNet v0.8.0

Milestone details

In the v0.8.0 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 67 commits by 5 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (67)

Contributors (5)

Thank you very much!

BenchmarkDotNet v0.7.8

Milestone details

In the v0.7.8 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 36 commits by 5 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (36)

Contributors (5)

Thank you very much!

BenchmarkDotNet v0.7.7

Milestone details

In the v0.7.7 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 14 commits by 2 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (14)

Contributors (2)

Thank you very much!

BenchmarkDotNet v0.7.6

Milestone details

In the v0.7.6 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 9 commits by 5 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (9)

Contributors (5)

Thank you very much!

BenchmarkDotNet v0.7.5

Milestone details

In the v0.7.5 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 4 commits by 1 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (4)

Contributors (1)

Thank you very much!

BenchmarkDotNet v0.7.4

Milestone details

In the v0.7.4 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 2 commits by 1 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (2)

Contributors (1)

Thank you very much!

BenchmarkDotNet v0.7.3

Milestone details

In the v0.7.3 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 1 commits by 1 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (1)

Contributors (1)

Thank you very much!

BenchmarkDotNet v0.7.2

Milestone details

In the v0.7.2 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 2 commits by 1 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (2)

Contributors (1)

Thank you very much!

BenchmarkDotNet v0.7.1

Milestone details

In the v0.7.1 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 4 commits by 1 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (4)

Contributors (1)

Thank you very much!

BenchmarkDotNet v0.7.0

Milestone details

In the v0.7.0 scope, 0 issues were resolved and 0 pull requests were merged. This release includes 59 commits by 2 contributors.

Resolved issues (0)

Merged pull requests (0)

Commits (59)

Contributors (2)

Thank you very much!