BenchmarkDotNet v0.11.4

It's been few months since our last release, but we have been working hard and have some new features for you!

Highlights

Features
- BenchmarkDotNet as a global tool: a new global tool which allows you to run benchmarks from given library. Now you can run benchmarks from the command line via dotnet benchmark. You can find more information about it in the documentation #1006 #213
- InProcessEmitToolchain: new, full-featured InProcess toolchain which allows executing benchmarks in the current process without spawning additional process per benchmark. It supports [Arguments], [ArgumentsSource], passing the arguments by out, ref and returning stack-only types like Span<T>. #919, #921 #843
- ARM support: BenchmarkDotNet supports now ARM and ARM64. #780, #979 #385
- Mono AOT support: a new toolchain which allows running benchmarks using AOT version of Mono #940
- NuGet symbol server support: BenchmarkDotNet publishes now the symbols to NuGet.org symbol server and you can easily debug it. #967 #968
- Experimental support for .NET Core 3.0 WPF benchmarks #1066 For a working example please go to https://github.com/dotMorten/WPFBenchmarkTests
Improvements:
- CoreRT Toolchain improvements - thanks to help from CoreRT Team we were able to make the CoreRT Toolchain work with the latest version of CoreRT #1001, #1057
- Display the number of benchmarks to run: we now display how many benchmarks are going to be executed before running them and how many remained after running each of them #1048
- Better list of suggested benchmarks for wrong filter #834 #957
- Invalid assembly binding redirects generated by VS were a pain to many of our users, we have now implemented an approach that tries to work around this issue. #895, #667, #896, #942
- Handling duplicates in IConfig #912, #938, #360, #463
- Disassembly diagnoser should be kept in a separate directory to avoid dependency conflicts #1059
- Give a warning when the [Benchmark] method is static - we now produce an error when users fail into this common issue #983 #985
- C# keywords are prohibited as benchmark names #849
- File names should be consistent across all OSes - < and > are valid on Unix, but not on Windows. We have unified that and now files produced on Unix and Windows have the same names. #981
- Improve restore, build and publish projects #1002, #1013
- Make it possible to disable OptimizationsValidator #988
- Sort enum parameters by value instead of name #977
- Detect .NET Core benchmark failures from LINQPad #980
- Improved error logging #1008
- Improved disassembly diff #1022
- Using invariant culture for Roslyn Toolchain error messages #1042
- Use only full names in the auto-generated code to avoid any possible conflicts with user code #1007, #1009 #1010
- Write the GitHub table format to the console by default #1062
- Proper cleanup on Ctrl+C/console Window exit #1061
- Introduce StoppingCriteria - the first step to writing your own heuristic that determines when benchmarking should be stopped #984
Breaking changes:
- .NET Standard 2.0 only - BenchmarkDotNet has a single target now, which should help with some assembly resolving issues. We had to drop .NET 4.6 support because of that and .NET 4.6.1 is now the oldest supported .NET Framework. #1032
- CustomCoreClrToolchain has been removed, it's recommended to use CoreRunToolchain instead #928
Bug fixes:
- NRE in Summary ctor #986 #987
- ArgumentNullException when running benchmarks from published .NET Core app #1018
- Dry jobs can eat iteration failures #1045
- NullReferenceException in BenchmarkDotNet.Reports.SummaryTable after iteration failure #1046
- Running the example throws NullReference #1049
- Fix race condition in process output reader #1051 #1053
- Fix a rare but really annoying bug where for some reason we were sometimes setting ForegroundColor to the same color as BackgroundColor and some parts of the logged output were invisible commit
- StopOnFirstError must be respected commit

Milestone details

In the v0.11.4 scope, 42 issues were resolved and 41 pull requests were merged. This release includes 99 commits by 18 contributors.

Resolved issues (42)

#213 Add a "benchmark" cmd to dotnet
#343 FileNotFoundException on mono (assignee: @AndreyAkinshin)
#360 Duplicates handling for IConfig (assignee: @adamsitnik)
#385 Consider using S.R.InteropServices.RuntimeInformation.ProcessArchitecture instead pointer based detection of platform (assignee: @adamsitnik)
#387 Add a mode to BenchmarkSwitcher that allows to run a method inline for profiling (assignee: @adamsitnik)
#463 Review interface IConfig (assignee: @adamsitnik)
#660 [Params] should not change the order of provided values
#667 Does BenchMarkDotnet supports 4.7.1 Dotnet framework (assignee: @adamsitnik)
#687 Implement [Arguments] support for InProcessToolchain (assignee: @ig-sinicyn)
#714 Test BenchmarkDotNet against unstable/multimodal benchmarks from CoreCLR/CoreFX repo (assignee: @adamsitnik)
#780 ARM support (assignee: @adamsitnik)
#834 Better list of suggested benchmarks for wrong filter (assignee: @morgan-kn)
#843 Exception when returning a stackonly structure in a benchmark case using in-process toolchain (assignee: @ig-sinicyn)
#849 C# keywords are prohibited as benchmark names (assignee: @adamsitnik)
#895 Could not load file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies (assignee: @adamsitnik)
#896 .NET 4.7.1 console app tries to use BenchmarkRunner, gets "Could not load file or assembly 'System.Runtime, Version=4.1.2.0" (assignee: @adamsitnik)
#919 Feature proposal: full-featured inprocess toolchain (assignee: @ig-sinicyn)
#928 Remove CustomCoreClrToolchain (assignee: @adamsitnik)
#938 Run benchmark with DisasemblyDiagnoser with --disasam option from console (assignee: @adamsitnik)
#942 System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.1.0.0' after adding BenchmarkDotNet.Diagnostics.Windows (assignee: @adamsitnik)
#967 Publish a snupkg to the NuGet.org symbol server (assignee: @WojciechNagorski)
#970 False alarm bug report
#981 File names should be consistent across all OSes (assignee: @adamsitnik)
#982 Invalid string representaiton of CPU Affinity on a machine with more than 32 cores on ARM64 (assignee: @adamsitnik)
#983 Give a warning when the [Benchmark] method is static (assignee: @Rizzen)
#986 NRE in Summary ctor
#988 Make it possible to disable OptimizationsValidator (assignee: @adamsitnik)
#998 Missing images in docs (assignee: @AndreyAkinshin)
#1002 Multiple build/publish failure with --coreRun toolchain (assignee: @WojciechNagorski)
#1007 benchmark cannot have type Action (assignee: @adamsitnik)
#1010 Write unit tests which check that BenchmarkProgram.txt doesn't contain usings (assignee: @adamsitnik)
#1018 ArgumentNullException when running benchmarks from published .NET Core app
#1039 Some tests are broken on Net 461 (culture-dependent thing) (assignee: @ig-sinicyn)
#1045 Dry jobs can eat iteration failures (assignee: @adamsitnik)
#1046 NullReferenceException in BenchmarkDotNet.Reports.SummaryTable after iteration failure (assignee: @adamsitnik)
#1048 Display the number of benchmarks to run (assignee: @adamsitnik)
#1049 Running the example throws NullReference (assignee: @adamsitnik)
#1051 Fix race condition in process output reader (assignee: @adamsitnik)
#1056 Fails to build when targeting .NET Core 3.0 and .NET Framework (assignee: @adamsitnik)
#1059 Disassembly diagnoser should be kept in a separate directory to avoid dependency conflicts (assignee: @adamsitnik)
#1062 Write the GitHub table format to the console by default (assignee: @adamsitnik)
#1065 Allow benchmarking .NET Core Desktop apps (assignee: @adamsitnik)

Merged pull requests (41)

#912 Duplicates handling for IConfig = big refactor, fixes #360 (by @adamsitnik)
#921 InProcessEmitToolchain (by @ig-sinicyn)
#940 Add support for mono AOT pass (by @alexanderkyte)
#957 Better list of suggested benchmarks for wrong filter #834 (by @morgan-kn)
#968 Support Nuget symbol server (by @WojciechNagorski)
#969 Disable batch mode and explicitly enable build trigger for master branch (by @Ky7m)
#977 sort enum parameters by value instead of name (by @kayle)
#979 ARM support (by @adamsitnik)
#980 Detect .NET Core benchmark failures from LINQPad (by @Turnerj)
#984 Introduce StoppingCriteria (by @AndreyAkinshin)
#985 Give a warning when the [Benchmark] method is static (by @Rizzen)
#987 Fix NRE in MetricColumn (#986) (by @qbit86)
#991 Fix typos (by @0x6a62)
#992 Use .NET Standard 2.0 CommandLineParser, Update to net461 for NS2.0 support (by @glennawatson)
#996 Rephrase Notes section and fix markdown (by @Maximusya)
#997 Remove obsolete info from the docs (by @Maximusya)
#999 Synchronize benchmark output with the code in docs (by @Maximusya)
#1001 CoreRT toolchain update (by @adamsitnik)
#1006 BenchmarkDotNet as global tool (by @CodeTherapist)
#1008 Improve error logging to diagnose unstable tests (by @adamsitnik)
#1009 Use only full names in the auto-generated code to avoid any possible conflicts with user code (by @adamsitnik)
#1012 Changed TargetCount to IterationCount in docs (by @Sitiritis)
#1013 Improve restore, build and publish projects - Fix for #1002 (by @WojciechNagorski)
#1014 Update IntroRatioSD.md (by @fredeil)
#1022 Improve diff disassembly (by @WojciechNagorski)
#1032 Target .NET Standard 2.0 only (by @adamsitnik)
#1033 BenchmarkDotNet as global tool (#1006), fixes #213 (by @adamsitnik)
#1035 Improve global tool (by @WojciechNagorski)
#1036 Remove InternalsVisibleTo for Samples application (by @WojciechNagorski)
#1038 Change the name of the global tool (by @WojciechNagorski)
#1040 Making the new InProcessEmitToolchain work after my recent refactor and .NET Standard 2.0 port (by @adamsitnik)
#1041 InProcessEmitToolchain (by @adamsitnik)
#1042 Use invariant culture for csc messages (by @ig-sinicyn)
#1043 minor InProcess fix: diff now checks for implementation flags (by @ig-sinicyn)
#1052 dotnet cli version update + reducing the number of long running tests (by @adamsitnik)
#1053 read the process output in a thread safe way, fixes #1051 (by @adamsitnik)
#1054 update Travis Ubuntu image from 14.04 to 16.04 (by @adamsitnik)
#1055 Allow reflecting on DebuggableAttribute on CoreRT (by @MichalStrehovsky)
#1057 CoreRT toolchain improvements (by @adamsitnik)
#1061 Proper cleanup on Ctrl+C/console Window exit (by @adamsitnik)
#1066 Add experimental support for .NET Core 3.0 WPF benchmarks (by @adamsitnik)

Commits (99)

1fac9b Postrelease update of v0.11.3 changelog (by @AndreyAkinshin)
40fae8 Support Nuget symbol server (#968) (by @WojciechNagorski)
69b01f remove batch and explicitly enable build for master (#969) (by @Ky7m)
01992c better error messages for lack of Cli and invalid CoreRun path (by @adamsitnik)
767e02 sort enum parameters by value instead of name (#977), fixes #660 (by @kayle)
128e11 Better list of suggested benchmarks for wrong filter #834 (#957) (by @morgan-kn)
c0910a Fix NRE in BaselineRatioColumn.GetRatioStatistics, fixes #970 (by @AndreyAkinshin)
67b675 Fix TimeSpan calculations in DotNetCliCommand.AddPackages (by @AndreyAkinshin)
847c27 Fix a few typos (by @AndreyAkinshin)
6fb830 ARM support (#979) (by @adamsitnik)
410d14 Detect .NET Core benchmark failures from LINQPad (#980), #975 (by @Turnerj)
138325 File names should be consistent across all OSes, fixes #981 (by @adamsitnik)
add308 test fix for #981 (by @adamsitnik)
77ed41 expose OriginalValues and SortedValues in the Statistics type so they get exp... (by @adamsitnik)
c0aac1 More tests for FolderNameHelper (by @AndreyAkinshin)
3497ae Better message in MinIterationTimeAnalyser (by @AndreyAkinshin)
91e16a Fix duplication of IsLinqPad check (by @AndreyAkinshin)
89255c Refactoring xUnit tests to avoid non-serializable objects in MemberData (by @AndreyAkinshin)
2cd06a Give a warning when the [Benchmark] method is static (#985) (by @Rizzen)
5070af Fix NRE in MetricColumn (#986) (#987) (by @qbit86)
17378d Fix typos (#991) (by @0x6a62)
2ce35a Fix path to logo in README (by @AndreyAkinshin)
a20e4b always print the path, args and working dir of the auto-generated executable,... (by @adamsitnik)
27dd87 CoreRunToolchain: when the file exists, overwrite it (by @adamsitnik)
1b01f3 Remove obsolete info from the docs (#997) (by @Maximusya)
299e1f Synchronized benchmark output with the code in docs (#999) (by @Maximusya)
0da14b Rephrase Notes section and fix markdown (#996) (by @Maximusya)
9e791f CoreRT toolchain update (#1001) (by @adamsitnik)
04747a Use only full names in the auto-generated code to avoid any possible conflict... (by @adamsitnik)
f35465 add unit test that prevents from adding using statements to the code, fixes #... (by @adamsitnik)
fd0b8c Improve build error logging, increase the default timeout (by @adamsitnik)
8276be Changed TargetCount to IterationCount in docs (#1012) (by @Sitiritis)
3c98da Update IntroRatioSD.md (#1014) (by @fredeil)
286996 Improve restore, build and publish projects - Fix for #1002 (#1013) (by @WojciechNagorski)
923b23 BenchmarkDotNet as global tool (#1006), fixes #213 (by @CodeTherapist)
7ef5f6 Introduce StoppingCriteria (#984) (by @AndreyAkinshin)
5e4428 make SortedValues internal property, don't export it! (by @adamsitnik)
5c519f Use .NET Standard 2.0 CommandLineParser, Update to net461 for NS2.0 support (... (by @glennawatson)
6ee21b if we fail to do the full build, we try with --no-dependencies (by @adamsitnik)
8d9714 Support machines without .NET DevPack, fix #1018 (by @AndreyAkinshin)
573566 Add support for mono AOT pass (#940) (by @alexanderkyte)
011c79 + InProcessEmitToolchain (by @ig-sinicyn)
862e6e InProcessEmitToolchain cleanup (by @ig-sinicyn)
252d7a Fix StringCanBePassedToBenchmarkAsReadOnlySpan (by @ig-sinicyn)
7ec2f3 Check if there's something wrong with RoslynToolchain on .Net Core (by @ig-sinicyn)
57acd6 No NOPs (thanks to @Warpten!) (by @ig-sinicyn)
bf362a Diff now compares nops (by @ig-sinicyn)
aa9ff8 + emit correct IL (by @ig-sinicyn)
351ca5 Ignore NOPs for ldarg too (by @ig-sinicyn)
fc530f Fix build after rebase (by @ig-sinicyn)
1577ba MonoAotToolchain refactoring, post #940 (by @adamsitnik)
6ccf45 Improve diff disassembly (#1022) (by @WojciechNagorski)
ef090d if we fail to do the full build, we try with --no-dependencies (for the publi... (by @adamsitnik)
c69934 Give a warning when the [Benchmark] method is static: handle some edge-cases,... (by @adamsitnik)
935ead use .NET 4.6.1 everywhere, we don't support 4.6 anymore. Cleanup after #992 (by @adamsitnik)
20a011 C# keywords are prohibited for benchmark names, print nice error message, fix... (by @adamsitnik)
2aec75 remove CustomCoreClrToolchain, it was causing too much trouble. We can run th... (by @adamsitnik)
f10752 Duplicates handling for IConfig = big refactor, fixes #360, closes #464 and f... (by @adamsitnik)
f9c8cc Target .NET Standard 2.0 (#1032), apply a workaround for assembly binding red... (by @adamsitnik)
aa1ded Merge branch 'master' into tools (by @adamsitnik)
ccee3e Merge pull request #1033 from dotnet/tools (by @adamsitnik)
1b9f9f almost no warnings ;) (by @adamsitnik)
3bd18f introduce ConfigOptions - an enum flag which make it easier to introduce new ... (by @adamsitnik)
66c264 reverting things that should not introduce problems but did... (by @adamsitnik)
33eaeb Merge branch 'master' into feature-inprocessemit (by @adamsitnik)
a1df27 Merge pull request #921 from ig-sinicyn/feature-inprocessemit (by @adamsitnik)
aac7a6 Improve global tool (#1035) (by @WojciechNagorski)
c01bc5 Remove InternalsVisibleTo for Samples application (#1036) (by @WojciechNagorski)
f6dbe4 Change the name of the global tool (#1038) (by @WojciechNagorski)
11d8cb Making the new InProcessEmitToolchain work after my recent refactor and .NET ... (by @adamsitnik)
e935b5 minor InProcess fix: diff now checks for implementation flags, (#1043) (by @ig-sinicyn)
da4128 Merge pull request #1041 from dotnet/newInProcess (by @adamsitnik)
82170f if global cleanup throws, we should report the problem but don't rethrow beca... (by @adamsitnik)
904ddd tests that ensure that when a benchmark throws the runner does not throw, fix... (by @adamsitnik)
cbdf7b Use invariant culture for csc messages (#1042) (by @ig-sinicyn)
5215ee Display the number of benchmarks to run, fixes #1048 (by @adamsitnik)
84cc3e dotnet cli version update + reducing the number of long running tests (#1052)... (by @adamsitnik)
1c431a read the process output in a thread safe way, fixes #1051 (#1053) (by @adamsitnik)
41a367 update Travis Ubuntu image from 14.04 to 16.04 (by @adamsitnik)
20744e allow the user to choose the strategy from console line arguments (by @adamsitnik)
190b9b Allow reflecting on DebuggableAttribute on CoreRT (#1055) (by @MichalStrehovsky)
9dd9e7 Ctlr+C: we should kill the entire process tree, not only for the benchmarks b... (by @adamsitnik)
d4249f Disassembly diagnoser should be kept in a separate directory to avoid depende... (by @adamsitnik)
0b83c9 CoreRT toolchain improvements (#1057) (by @adamsitnik)
9bc8f3 0.11.4 initial release notes (by @adamsitnik)
13bb97 if the user provide a custom value, we should use it, post #1057 (by @adamsitnik)
ea3036 fix a rare but really annoying bug where for some reason we were sometimes se... (by @adamsitnik)
cf3f8c Proper cleanup on Ctrl+C/console Window exit (#1061) (by @adamsitnik)
61f563 Write the GitHub table format to the console by default, fixes #1062 (by @adamsitnik)
87d281 StopOnFirstError must be respected (by @adamsitnik)
413d31 Add experimental support for .NET Core 3.0 WPF benchmarks (#1066), fixes #1065 (by @adamsitnik)
69a8aa copy UseWindowsForms too (thanks @onovotny for pointing this out), post #1066 (by @adamsitnik)
d3379e Bump DocFX version (by @AndreyAkinshin)
423204 Remove Version ComboBox in docs (by @AndreyAkinshin)
1a8826 Update build-and-pack.cmd (by @AndreyAkinshin)
9068d6 Update copyrights in docs (by @AndreyAkinshin)
c0c2bf Update v0.11.4 changelog (by @AndreyAkinshin)
e26441 Highlight the first column separator in the console summary table (by @AndreyAkinshin)
a704a4 Set library version: 0.11.4 (by @AndreyAkinshin)

Contributors (18)

Adam Sitnik (@adamsitnik)
Alexander Kyte (@alexanderkyte)
Andrey Akinshin (@AndreyAkinshin)
Code Therapist (@CodeTherapist)
Fredrik Eilertsen (@fredeil)
Glenn (@glennawatson)
ig-sinicyn (@ig-sinicyn)
Igor Fesenko (@Ky7m)
Irina Ananeva (@morgan-kn)
James Turner (@Turnerj)
Jeff B. (@0x6a62)
kayle (@kayle)
Maksim Yakimets (@Maximusya)
Mark Tkachenko (@Rizzen)
Michal Strehovský (@MichalStrehovsky)
Tymur Lysenko (@Sitiritis)
Viktor Ptitselov (@qbit86)
Wojciech Nagórski (@WojciechNagorski)

Thank you very much!

Additional details

Date: February 15, 2019

Milestone: v0.11.4 (List of commits)

NuGet Packages:

Table of Contents