BenchmarkDotNet v0.14.0

Highlights

• Introduce BenchmarkDotNet.Diagnostics.dotMemory #2549: memory allocation profile of your benchmarks using dotMemory, see BenchmarkDotNet.Samples.IntroDotMemoryDiagnoser
• Introduce BenchmarkDotNet.Exporters.Plotting #2560: plotting via ScottPlot (initial version)
• Multiple bugfixes
• The default build toolchains have been updated to pass IntermediateOutputPath, OutputPath, and OutDir properties to the dotnet build command. This change forces all build outputs to be placed in a new directory generated by BenchmarkDotNet, and fixes many issues that have been reported with builds. You can also access these paths in your own .csproj and .props from those properties if you need to copy custom files to the output.

Bug fixes

• Fixed multiple build-related bugs including passing MsBuildArguments and .Net 8's UseArtifactsOutput.

Breaking Changes

• DotNetCliBuilder removed retryFailedBuildWithNoDeps constructor option.
• DotNetCliCommand removed RetryFailedBuildWithNoDeps property and BuildNoRestoreNoDependencies() and PublishNoBuildAndNoRestore() methods (replaced with PublishNoRestore()).

Milestone details

In the v0.14.0 scope, 17 issues were resolved and 28 pull requests were merged. This release includes 55 commits by 16 contributors.

Resolved issues (17)

• #946 Copy to $(OutDir) does not copy to benchmarkdotnet output folder (assignee: @timcassell)
• #1270 Docs for SummaryStyle
• #1377 Forward msbuild properties to project reference (assignee: @timcassell)
• #1457 Directory.Build.props hijacking generated csproj output path (assignee: @timcassell)
• #1595 ValueTask-returning benchmarks are not being called correctly (assignee: @timcassell)
• #1795 'BenchmarkDotNet.Autogenerated\project.assets.json' not found. (assignee: @timcassell)
• #2253 TaskbarProgressCom is not supported when benchmarking Native AOT
• #2310 Bug: Getting an error loading BenchmarkDotNet.Diagnostics.Windows.dll for a benchmark that is not windows specific
• #2366 default(object) takes more time than expected in net7.0
• #2466 Add support for true (assignee: @timcassell)
• #2477 MSBuild Params Not Applied to Benchamrk (assignee: @timcassell)
• #2524 Reference assemblies are searched in the wrong location
• #2540 Unable to use wasm or monoaotllvm as runtime (assignee: @LoopedBard3)
• #2542 InvalidMeasurementUnitException when using --statisticalTest argument when comparing private builds (assignee: @AndreyAkinshin)
• #2545 HideColumns + custom corerun = incorrect table markdown
• #2554 Question about visibility of DotTraceDiagnoser's second constructor (assignee: @AndreyAkinshin)
• #2570 BenchmarkDotNet test adapter not running tests with dotnet test

Merged pull requests (28)

• #2108 Fix consume ValueTask backed by IValueTaskSource (by @timcassell)
• #2255 Fix TaskbarProgress (by @timcassell)
• #2374 stackalloc in separate method (by @timcassell)
• #2393 Fix MsBuildArguments (by @timcassell)
• #2498 Enhance Diagnoser Loading Behavior for Non-Critical Assemblies (by @MattFromRVA)
• #2500 Mark test adapter entry points as autogenerated (by @molesmoke)
• #2503 Update MsBuildErrorMapper (by @timcassell)
• #2504 Fix some tests (by @timcassell)
• #2510 Docs for SummaryStyle (by @MattFromRVA)
• #2517 fix: Spelling mistake (by @BenjaminMichaelis)
• #2522 Fix benchmarks with same class but different namespace having same ID (by @caaavik-msft)
• #2523 .Net SDK Validation (by @MattFromRVA)
• #2525 Update toolchain docs (by @MattFromRVA)
• #2526 Bump Octokit version (by @adamsitnik)
• #2531 Fix wasm toolchain (by @timcassell)
• #2532 Add Integration Test for WASM (by @caaavik-msft)
• #2534 Update Microsoft.Diagnostics.Tracing.TraceEvent (by @sbomer)
• #2538 Add support for wasm and Mono AOT LLVM monikers to SDK Validator (by @caaavik-msft)
• #2539 Fix Mono AOT LLVM (by @caaavik-msft)
• #2541 Add Wasm and MonoAOTLLVM to DotNetSdkVersionValidator (by @LoopedBard3)
• #2549 DotMemoryDiagnoser implementation (by @martinothamar)
• #2560 Initial ScottPlotExporter with just Bar Plot and Unit Tests (by @FlatlinerDOA)
• #2565 Fix - incorrect table markdown #2545 (by @AumkarGorde)
• #2566 Fix names of instructions sets for PublishAot scenarios (by @MichalStrehovsky)
• #2571 Fix BenchmarkDotNet test adapter not running tests with dotnet test (by @sheffer21)
• #2572 Add Support for Warning Logging and Integrate with Visual Studio Test Logging (by @sheffer21)
• #2583 Updated readme to make it clear that experienced performance engineers are also welcome (by @iCodeSometime)
• #2605 fix(CI): Deprecation issues (by @workgroupengineering)

Commits (55)

• f8ac9c Set next BenchmarkDotNet version: 0.13.13 (by @AndreyAkinshin)
• 5dd1a9 [docs] Update cSpell.json (by @AndreyAkinshin)
• 7dacb1 [docs] Update cSpell.json (by @AndreyAkinshin)
• aa85d8 Added new Exception (by @MattFromRVA)
• e8cf48 Update copyright year (by @AndreyAkinshin)
• 688913 [build] Bump .NET SDK: 8.0.100 -> 8.0.101 (by @AndreyAkinshin)
• 3cf7b8 [build] Reduce verbosity of RestoreTask (by @AndreyAkinshin)
• 96d8f5 [docs] Update vstest.md (by @AndreyAkinshin)
• cf14d5 Mark test adapter entry points as autogenerated (by @jeremy-visionaid)
• 570eee [build] Bump Docfx.App: 2.74.1->2.75.0 (by @AndreyAkinshin)
• c4c204 Fixed some tests. (by @timcassell)
• f32166 Update MsBuildErrorMapper. (by @timcassell)
• d2f690 Update tests. (by @timcassell)
• 1d95e5 [build] Bump Docfx.App: 2.75.0->2.75.1 (by @AndreyAkinshin)
• e6fdc6 Pass OutputPath, IntermediateOutputPath, OutDir properties to dotnet ... (by @timcassell)
• 33e9e9 fix: Spelling mistake (by @BenjaminMichaelis)
• 109528 Fix benchmarks with same class but different namespace having same ID (#2522) (by @caaavik-msft)
• c20fd5 Update toolchain docs (#2525) (by @MattFromRVA)
• 0a41e1 Bump Octokit version (by @adamsitnik)
• 863165 Fix wasm toolchain (#2531) (by @timcassell)
• f4c39e Update Microsoft.Diagnostics.Tracing.TraceEvent (#2534) (by @sbomer)
• 59647c .Net SDK Validation (#2523) (by @MattFromRVA)
• f17d40 Add support for wasm and Mono AOT LLVM monikers to SDK Validator (#2538) (by @caaavik-msft)
• 443579 Add Integration Test for WASM (#2532) (by @caaavik-msft)
• b5fc05 [build] Make build.sh more portable (by @AndreyAkinshin)
• 63626b [tests] Add docs for WasmTests (by @AndreyAkinshin)
• 9a9d7e Fix Mono AOT LLVM (#2539) (by @caaavik-msft)
• 3305b3 Add System.Runtime.CompilerServices.Unsafe to BenchmarkDotNet.Tests (by @AndreyAkinshin)
• 306210 Bump Perfolizer: 0.2.1->0.3.16 (by @AndreyAkinshin)
• 036f46 Add Wasm and MonoAOTLLVM to DotNetSdkVersionValidator (#2541) (by @LoopedBard3)
• ad0240 Bump Perfolizer: 0.3.16->0.3.17 (by @AndreyAkinshin)
• 7a1c10 Fix bugs in statistical tests, fix #2542 (by @AndreyAkinshin)
• 0d3099 Fix StatisticalTestColumnTests.NoDifferenceIfValuesAreTheSame (by @AndreyAkinshin)
• 7306ee Fix consume ValueTask backed by IValueTaskSource (#2108) (by @timcassell)
• 4ab69b Moved stackalloc to separate method. (#2374) (by @timcassell)
• c7b7ab DotMemoryDiagnoser implementation (#2549) (by @martinothamar)
• 55ce92 Bump JetBrains.Profiler.SelfApi: 2.5.0->2.5.2 (by @AndreyAkinshin)
• a24d68 Fix DotTrace/DotMemory attribute ctors, fix #2554 (by @AndreyAkinshin)
• d98a1d Fix names of instructions sets for PublishAot scenarios (#2566) (by @MichalStrehovsky)
• 205ce6 Fix - incorrect table markdown #2545 (#2565) (by @AumkarGorde)
• 063d2f Fix BenchmarkDotNet test adapter not running tests with dotnet test (#2571) (by @sheffer21)
• 6a7244 Add warning support (#2572) (by @sheffer21)
• 01d9b7 Update README.md (#2583) (by @iCodeSometime)
• 20e2ee Docs for SummaryStyle (#2510) (by @MattFromRVA)
• e933bb Initial ScottPlotExporter with just Bar Plot and Unit Tests (#2560) (by @FlatlinerDOA)
• f8082a Fix IntroSummaryStyle compilation (by @AndreyAkinshin)
• 834417 Improve logging in ScottPlotExporterTests (by @AndreyAkinshin)
• 15200d [build] Add BenchmarkDotNet.Exporters.Plotting.Tests to unit-tests (by @AndreyAkinshin)
• 027564 Fixed crash from TaskbarProgress when BuiltInComInteropSupport is disabled. ... (by @timcassell)
• bf0a49 fix(CI): Deprecation issues (#2605) (by @workgroupengineering)
• 3d34ed Bump JetBrains.Profiler.SelfApi: 2.5.2->2.5.9 (by @AndreyAkinshin)
• 23e6c5 Fix InvalidOperationException in DotMemoryDiagnoser (by @AndreyAkinshin)
• b3fbe7 Set next BenchmarkDotNet version: 0.14.0 (by @AndreyAkinshin)
• 17cf3b [docs] Prepare v0.14.0 changelog (by @AndreyAkinshin)
• cf882d Add macOS Sequoia in OsBrandStringHelper (by @AndreyAkinshin)

Contributors (16)

• Adam Sitnik (@adamsitnik)
• Alon Sheffer (@sheffer21)
• Andrew Chisholm (@FlatlinerDOA)
• Andrey Akinshin (@AndreyAkinshin)
• Benjamin Michaelis (@BenjaminMichaelis)
• Cameron Aavik (@caaavik-msft)
• Jeremy Powell (@jeremy-visionaid)
• Kenneth Cochran (@iCodeSometime)
• Martin Othamar (@martinothamar)
• Matt Chaulklin (@MattFromRVA)
• Michal Strehovský (@MichalStrehovsky)
• Omkar Gorde (@AumkarGorde)
• Parker Bibus (@LoopedBard3)
• Sven Boemer (@sbomer)
• Tim Cassell (@timcassell)
• workgroupengineering (@workgroupengineering)

Thank you very much!

Additional details

Date: August 06, 2024

Milestone: v0.14.0 (List of commits)

NuGet Packages:

• https://www.nuget.org/packages/BenchmarkDotNet/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.Annotations/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.Diagnostics.dotMemory/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.Diagnostics.dotTrace/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.Diagnostics.Windows/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.Exporters.Plotting/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.Templates/0.14.0
• https://www.nuget.org/packages/BenchmarkDotNet.TestAdapter/0.14.0