BenchmarkDotNet v0.13.3

Highlights

• New supported technologies
  • Add arm64 disassembler #1422 #2127 #2107 #2123 #2070 #2118 #2119 #2234 #2222 #2212 9ee1/Capstone.NET#37
  • Initial .NET 8 support #2192
  • .NET 6/7 MonoVM support #2064 #2142 #2227 #2230
  • Armv6 and Ppc64le architectures support #2216 #2219
• Improved support
  • Improved WASM support #2201 #2099 #2154 #2112
  • Improved NativeAOT support #2095 #2221
  • Improved Android support #2231
  • 32-bit benchmarks can now handle addresses larger than 2GB with the help of LargeAddressAware #1469 #2145
  • Support 64bit affinity masks #2211 #2228
• New features
  • Add ExceptionDiagnoser #1736 #2169 #2182
  • Add PerfCollectProfiler #2117
  • Incremental benchmark execution with the help of --resume #1799 #2164
  • Taskbar progress #2102 #2158 #2140
  • Support --noForcedGCs to avoid forced GC between benchmark runs #2101
  • Added apples to apples comparison mode #2116 #2193
  • Communication between the host process and the benchmark process is now using pipes instead of output parsing #2092 #1933 #2189 #2207 #2200
• Dozens of bugfixes

Special Thanks

We would like to highlight some important contributors who helped us with this release:

1. OpenHack'22 (devexperts.com) hackathon sponsored by the DevExperts company. As part of this hackathon we have received following PRs:

• #2132 fix: include argument and param names in --filter (by @blouflashdb)
• #2140 Update console title with benchmark information (by @franciscomoloureiro)
• #2142 Issue 2064: Mono70 moniker (by @Serg046)
• #2148 adding validation errors when the benchmarks are unsupported (by @emanuel-v-r)
• #2160 Corrected logic to restore foreground color in ConsoleLogger.cs (by @farQtech)
• #2164 1799 adding resume arg (by @melias)
• #2169 Issue #1736: Add ExceptionDiagnoser (by @Serg046)
• #2161 add quiet logger (by @franciscomoloureiro) (not merged yet)
• #2171 Issue #1024: Calculate baseline by the fastest benchmark (by @Serg046) (not merged yet)

2. Jan Vorlicek helped to implement arm64 disassembler during an internal Microsoft Hackathon:

• #2107 Implement TryGetReferencedAddress for relative branches (by @janvorli)
• #2123 Added other arm64 constant form extraction plus other changes (by @janvorli)

3. Ahmed Garhy (maintainer of Capstone.NET) helped to improve Capstone.NET, which was need to implement arm64 disassembler:

• 9ee1/Capstone.NET#37 Sign Assembly with a Strong Name (by @9ee1)

Milestone details

In the v0.13.3 scope, 29 issues were resolved and 71 pull requests were merged. This release includes 87 commits by 22 contributors.

Resolved issues (29)

• #989 [Suggestion] add API for detecting benchmark run failures. (assignee: @emanuel-v-r)
• #1422 --disasm switch on ARM64 throws Exception (assignee: @adamsitnik)
• #1469 Host exe marked /largeaddressaware
• #1521 Iteration setup and cleanup causes job baseline error with multiple runtimes passed to BenchmarkSwitcher
• #1684 Getting System.FormatException when passing certain string as params. (assignee: @YegorStepanov)
• #1709 BenchmarkSwitcher executes all benchmarks that share a base class (assignee: @YegorStepanov)
• #1736 Consider adding an ExceptionDiagnoser (assignee: @Serg046)
• #1737 runtime knobs broken link (assignee: @YegorStepanov)
• #1799 Enable interactive, incremental runs through the terminal (assignee: @melias)
• #1839 Markdown output should escape the output
• #1867 Trailing newline characters in input value break summary table
• #1933 how to debug reflection error
• #2064 Running with .Net 6/7 Mono JIT (assignee: @Serg046)
• #2070 BenchmarkDotNet crashing on Linux with DisassemblyDiagnoser (assignee: @adamsitnik)
• #2088 Running on Linux leaves terminal colors changed (assignee: @farQtech)
• #2099 WASM is recognized as NativeAOT
• #2102 Add benchmarking progress to console title (assignee: @franciscomoloureiro)
• #2125 Tests Just Stop Running During Run (assignee: @adamsitnik)
• #2126 DotNet 6 - VB - Error: The type or namespace name 'DeserializingBenchmarks' could not be found in the global namespace (are you missing an assembly reference?)
• #2131 --filter should include argument/params names (assignee: @blouflashdb)
• #2146 Build warning MSB3245: Could not locate the assembly Mono.Posix
• #2167 Site: No vertical bar should be displayed for the Main page
• #2185 WarmupCount=0 doesn't work (assignee: @AndreyAkinshin)
• #2187 Reports for InProcess jobs don't include non-Result measurements (assignee: @AndreyAkinshin)
• #2189 broker.ProcessData() hangs if something wrong is the spawned process (assignee: @adamsitnik)
• #2210 Broken integration tests on Ubuntu 22.04 (assignee: @adamsitnik)
• #2211 Setting affinity does not work for Environment.ProcessorCount >= 32 (assignee: @Donis-)
• #2216 ppc64le architecture support required for running benchmarks on Power Systems (assignee: @adamsitnik)
• #2223 Disassembler fails to disassemble some methods on Linux when using recursive mode (assignee: @adamsitnik)

Merged pull requests (71)

• #2018 Port JetBrains' nullability annotations and clean-up code. (by @teo-tsirpanis)
• #2085 Use latest AzDO macOS pool (by @adamsitnik)
• #2087 Nit: fix broken link (by @YegorStepanov)
• #2092 Use Pipes for host and benchmark process communication (by @adamsitnik)
• #2095 [NativeAOT] use PublishAot and don't reference ILCompiler in explicit way (by @adamsitnik)
• #2096 ExecuteResult: Surface a Data property so the full output can be acce… (by @radical)
• #2101 Added a command line arg for not inducing any GCs while running Benchmarks (by @mrsharm)
• #2104 Fix invalid pre-requisites NativeAOT link in the 0.13.2 changelog (by @KeterSCP)
• #2107 Implement TryGetReferencedAddress for relative branches (by @janvorli)
• #2112 Disambiguate NativeAOT, and Wasm identification in BenchmarkDotNet.Po… (by @radical)
• #2116 Implement apples to apples comparison mode (by @adamsitnik)
• #2117 PerfCollect diagnoser (by @adamsitnik)
• #2118 Resolve indirect addresses in disassembly (by @adamsitnik)
• #2119 Initial version of the Arm64 instruction formatter (by @adamsitnik)
• #2120 Roslyn Toolchain does not support .NET Core (by @YegorStepanov)
• #2123 Added other arm64 constant form extraction plus other changes (by @janvorli)
• #2127 arm64 disassembler (by @adamsitnik)
• #2132 fix: include argument and param names in --filter (by @blouflashdb)
• #2133 ensure CompositeLogger is synchronized (by @adamsitnik)
• #2134 [Mono] Disable LLVM JIT (by @fanyang-mono)
• #2135 Escape Param data for the exporters (by @YegorStepanov)
• #2136 Pass non escaped strings to generated project (by @YegorStepanov)
• #2137 Update doc links and reduce redirects (by @YegorStepanov)
• #2140 Update console title with benchmark information (by @franciscomoloureiro)
• #2142 Issue 2064: Mono70 moniker (by @Serg046)
• #2143 throw an exception when multiple benchmark projects with the same name are found (by @adamsitnik)
• #2144 Automated spellcheck for docs via GitHub Actions (and address all raised issues) (by @SeanKilleen)
• #2145 Handle addresses larger than 2GB for 32-bit benchmarks #1469 (by @leonvandermeer)
• #2148 adding validation errors when the benchmarks are unsupported (by @emanuel-v-r)
• #2150 Update StaThread intro documentation (by @norepro)
• #2151 Fix user input matching (by @YegorStepanov)
• #2152 Make ParamsAllValues validator mandatory (by @YegorStepanov)
• #2154 remove dependency to Mono.Posix.NETStandard to unblock WASM benchmarks runs (by @adamsitnik)
• #2158 Add taskbar progress (by @timcassell)
• #2160 Corrected logic to restore foreground color in ConsoleLogger.cs (by @farQtech)
• #2164 1799 adding resume arg (by @melias)
• #2168 Fix bugs and typos (by @YegorStepanov)
• #2169 Issue #1736: Add ExceptionDiagnoser (by @Serg046)
• #2172 Cleanup NuGet.config (by @Youssef1313)
• #2174 Added the ability to not run with Evaluation Overhead (by @mrsharm)
• #2175 Rename TargetCount to IterationCount in SimpleJob attribute (by @johanvts)
• #2176 Fix #1521 (by @YegorStepanov)
• #2177 Simplify GetHashCode() (by @YegorStepanov)
• #2180 Fix race in AsyncProcessOutputReader (by @AndreyAkinshin)
• #2181 Fix #2167 - Give main page a title. Then the tab displays 'Home | Be…' (by @leonvandermeer)
• #2182 Fix a threading issue in ExceptionDiagnoser #1736 (by @leonvandermeer)
• #2183 Remove allowMultiple=true from column attributes (by @YegorStepanov)
• #2186 Fix EngineStage.Run for warmupCount=0 (fixes #2185) (by @AndreyAkinshin)
• #2188 Engine.Run() should return the full list of performed measurements (fixes #2187) (by @AndreyAkinshin)
• #2192 Add net8.0 support to all existing runtimes and toolchains (by @adamsitnik)
• #2193 use ImmutableConfig when doing apples-to-apples comparison (by @adamsitnik)
• #2196 Docs: Add note for ETW Profiling regarding Intel TDT (by @rbanks54)
• #2200 Fix resources leak (by @adamsitnik)
• #2201 fix WASM support (by @adamsitnik)
• #2202 Fix job filtering (by @YegorStepanov)
• #2204 Deprecate RunSource/RunUrl methods (by @YegorStepanov)
• #2207 Avoid hangs when starting benchmark process fails (by @adamsitnik)
• #2208 Fix Full Framework tests (by @adamsitnik)
• #2209 Update SDK to .NET 7, re-enable tests (by @adamsitnik)
• #2212 use the new, strong-name signed Capstone.NET (by @adamsitnik)
• #2217 Bump Newtonsoft.Json (by @YegorStepanov)
• #2219 add Armv6 and Ppc64le architectures support (by @adamsitnik)
• #2220 Remove duplicate reference (by @YegorStepanov)
• #2221 NuGet.org should be the default feed for NativeAOT 7.0 ILCompiler (by @adamsitnik)
• #2222 arm64 disassembler tests (by @adamsitnik)
• #2227 Add support for MonoVM to MemoryDiagnoser (by @adamsitnik)
• #2228 Increase max supported affinity from 31 to 64 (#2211) (by @Donis-)
• #2230 Add support for .NET SDK that uses Mono instead CLR as a default VM (by @adamsitnik)
• #2231 improve Android support (by @adamsitnik)
• #2234 Disassembler realiability fixes (by @adamsitnik)
• #2235 tests can't assume x64 hardware (by @adamsitnik)

Commits (87)

• 33b288 Use latest AzDO macOS pool (#2085) (by @adamsitnik)
• 83750b Postrelease v0.13.2 update (by @AndreyAkinshin)
• 188c11 Nit: fix broken link (#2087) (by @YegorStepanov)
• 3a18b1 use PublishAot and don't reference ILCompiler in explicit way (#2095) (by @adamsitnik)
• 8ed521 Added a command line arg for not inducing any GCs while running Benchmarks (#... (by @mrsharm)
• 8514b2 ExecuteResult: Surface a Data property so the full output can be accessed (#2... (by @radical)
• 58f2d1 Use Pipes for host and benchmark process communication (#2092) (by @adamsitnik)
• b525ba Fix invalid pre-requisites NativeAOT links in the docs (#2104) (by @KeterSCP)
• 0cee16 Disambiguate NativeAOT, and Wasm identification in BenchmarkDotNet.Portabilit... (by @radical)
• 21a294 Implement apples to apples comparison mode (#2116) (by @adamsitnik)
• 37d0cf Roslyn Toolchain does not support .NET Core (#2120) (by @YegorStepanov)
• dbccef Add workflows/docs-stable.yaml (by @AndreyAkinshin)
• d03287 PerfCollect diagnoser (#2117) (by @adamsitnik)
• de5cf4 Bump docfx 2.59.3->2.59.4 (by @AndreyAkinshin)
• 7e87e8 Update workflows/docs-stable.yaml (by @AndreyAkinshin)
• ff443a Update workflows/docs-stable.yaml (by @AndreyAkinshin)
• a78c2e Add write-all permissions to workflows/docs-stable.yaml (by @AndreyAkinshin)
• f4d99a arm64 disassembler (#2127) (by @adamsitnik)
• 97c2d1 ensure access to logger is synchronized for async output reader (#2133) (by @adamsitnik)
• 9938c3 include argument and param names in --filter (#2132) (by @blouflashdb)
• 7f9590 Update doc links and reduce redirects (#2137) (by @YegorStepanov)
• 95bb2a Escape Param data for the exporters (#2135) (by @YegorStepanov)
• ecb25b Pass non escaped strings to generated project (#2136) (by @YegorStepanov)
• 5ed46d [Mono] Disable LLVM JIT (#2134) (by @fanyang-mono)
• 1f5637 throw an exception when multiple benchmark projects with the same name are fo... (by @adamsitnik)
• 095975 Handle addresses larger than 2GB for 32-bit benchmarks #1469 (#2145) (by @leonvandermeer)
• 8d2379 Update console title with benchmark information (#2140) (by @franciscomoloureiro)
• f8e0a5 Automated spellcheck for docs via GitHub Actions (and address all raised issu... (by @SeanKilleen)
• db8f8d Fix spelling warning in docs/articles/configs.jobs.md (by @AndreyAkinshin)
• adf9d6 Fix BenchmarkSwitcher user input matching (#2151) (by @YegorStepanov)
• b758d2 Update StaThread intro documentation (#2150) (by @norepro)
• eda1a4 remove dependency to Mono.Posix.NETStandard to unblock WASM benchmarks runs (... (by @adamsitnik)
• 58d4ba Make ParamsAllValues validator mandatory (#2152) (by @YegorStepanov)
• 28bf21 adding validation errors when the benchmarks are unsupported (#2148) (by @emanuel-v-r)
• 1fb101 Corrected logic to restore foreground color in ConsoleLogger.cs (#2160) (by @farQtech)
• c02c3d Fix bugs and typos (#2168) (by @YegorStepanov)
• 0f7eb2 Implement --resume support (#2164), fixes #1799 (by @melias)
• 7d8375 Issue #1736: Add ExceptionDiagnoser (#2169) (by @Serg046)
• 64c3a3 Implement MonoVM toolchain for net6.0 and net7.0 monikers (#2142) fixes #2064 (by @Serg046)
• c69895 Add taskbar progress (#2158) (by @timcassell)
• 163899 Cleanup NuGet.config (#2172) (by @Youssef1313)
• f4c0a7 Added the ability to not run with Evaluation Overhead from command line (#2174) (by @mrsharm)
• ce1b74 Rename TargetCount to IterationCount in SimpleJob attribute (#2175) (by @johanvts)
• 18c6ff Fix #1521 (#2176) (by @YegorStepanov)
• 4eb6d3 Bump Build.csproj TFM to net6.0 (by @AndreyAkinshin)
• 279c96 Bump actions/checkout v2->v3 (by @AndreyAkinshin)
• a1c62c Bump actions/checkout v2->v3 for spellcheck.yml (by @AndreyAkinshin)
• 7ba989 Automatically generate redirects in documentation (by @AndreyAkinshin)
• d55c47 Remove old redirect files in documentation (by @AndreyAkinshin)
• b6d329 Bump System.Drawing.Common in BenchmarkDotNet.Samples.csproj: 4.5.1->4.7.2 (by @AndreyAkinshin)
• ee8125 Remove allowMultiple=true from column attributes (#2183) (by @YegorStepanov)
• ddd2f3 Fix a threading issue in ExceptionDiagnoser #1736 (#2182) (by @leonvandermeer)
• 276f1a Fix EngineStage.Run for warmupCount=0 (fixes #2185) (#2186) (by @AndreyAkinshin)
• 681a63 Fix #2167 - Give main page a title. Then the tab displays 'Home | BenchmarkDo... (by @leonvandermeer)
• 00f693 Fix race in AsyncProcessOutputReader (#2180) (by @AndreyAkinshin)
• 36e998 Disable MemoryDiagnoserSupportsNativeAOT on osx-arm64 (by @AndreyAkinshin)
• 9e759f Disable ThreadingDiagnoserTests with ILCompiler 6.0.0-rc.1.21420.1 on osx-arm64 (by @AndreyAkinshin)
• e75bdd Engine.Run() should return the full list of performed measurements (fixes #21... (by @AndreyAkinshin)
• 28a8e7 Simplify GetHashCode() (#2177) (by @YegorStepanov)
• 36f9e7 Set net6.0 as the first TFM for BenchmarkDotNet.Tests (by @AndreyAkinshin)
• 761590 Add net8.0 support to all existing runtimes and toolchains (#2192) (by @adamsitnik)
• 8b6159 use ImmutableConfig when doing apples-to-apples comparison (#2193) (by @adamsitnik)
• a4ab68 Docs: Add note for ETW Profiling regarding Intel TDT and Windows Defender (#2... (by @rbanks54)
• 9c32a8 fix resources leak (#2200) (by @adamsitnik)
• ad8e9b fix WASM support (#2201) (by @adamsitnik)
• ccbaf0 Remove duplicated jobs when creating immutable config (#2202) (by @YegorStepanov)
• 90c82b Deprecate methods (#2204) (by @YegorStepanov)
• 47b8b7 Avoid hangs when starting benchmark process fails (#2207) (by @adamsitnik)
• 7982b8 Fix Full Framework tests (#2208) (by @adamsitnik)
• 61b3c5 Update SDK to .NET 7, re-enable NativeAOT tests, fix some other tests (#2209) (by @adamsitnik)
• d38313 use the new, strong-name signed Capstone.NET (#2212) (by @adamsitnik)
• b40150 add Armv6 and Ppc64le architectures support (#2219) (by @adamsitnik)
• 5ce28b Bump Newtonsoft.Json (#2217) (by @YegorStepanov)
• 3531c1 Port JetBrains' nullability annotations and clean-up code. (#2018) (by @teo-tsirpanis)
• 5ef855 Remove duplicate reference (#2220) (by @YegorStepanov)
• ee24d7 NuGet.org should be the default feed for NativeAOT 7.0 ILCompiler package (#2... (by @adamsitnik)
• 530d00 arm64 disassembler tests (#2222) (by @adamsitnik)
• 699285 Add support for MonoVM to MemoryDiagnoser (#2227) (by @adamsitnik)
• 0c90af change something small just to triegger the CI (by @adamsitnik)
• 2cce42 improve Android support (#2231) (by @adamsitnik)
• a6ef73 don't run Long Running Test on AppVeyor, try to avoid 1h timeouts (by @adamsitnik)
• bf8b53 Increase max supported affinity from 31 to 64 (#2211) (#2228) (by @Donis-)
• 82f03f Add support for .NET SDK that uses Mono instead CLR as a default VM (#2230) (by @adamsitnik)
• 968448 Disassembler realiability fixes (#2234) (by @adamsitnik)
• a098bc tests can't assume x64 hardware (#2235) (by @adamsitnik)
• 2665ac Prepare v0.13.3 changelog (by @AndreyAkinshin)
• 0714f5 Set library version: 0.13.3 (by @AndreyAkinshin)

Contributors (22)

• Adam Sitnik (@adamsitnik)
• Andrey Akinshin (@AndreyAkinshin)
• Ankit Jain (@radical)
• blouflashdb (@blouflashdb)
• Donis- (@Donis-)
• Emanuel Ramos (@emanuel-v-r)
• Fan Yang (@fanyang-mono)
• farQtech (@farQtech)
• franciscomoloureiro (@franciscomoloureiro)
• Johan von Tangen Sivertsen (@johanvts)
• leonvandermeer (@leonvandermeer)
• Maykon Elias (@melias)
• Mukund Raghav Sharma (Moko) (@mrsharm)
• norepro (@norepro)
• Richard Banks (@rbanks54)
• Sean Killeen (@SeanKilleen)
• Sergey Aseev (@Serg046)
• Sergiusz Zalewski (@KeterSCP)
• Theodore Tsirpanis (@teo-tsirpanis)
• Tim Cassell (@timcassell)
• Yegor Stepanov (@YegorStepanov)
• Youssef Victor (@Youssef1313)

Thank you very much!

Additional details

Date: December 26, 2022

Milestone: v0.13.3 (List of commits)

NuGet Packages:

• https://www.nuget.org/packages/BenchmarkDotNet/0.13.3
• https://www.nuget.org/packages/BenchmarkDotNet.Diagnostics.Windows/0.13.3
• https://www.nuget.org/packages/BenchmarkDotNet.Annotations/0.13.3
• https://www.nuget.org/packages/BenchmarkDotNet.Templates/0.13.3