Show / Hide Table of Contents

BenchmarkDotNet v0.11.5

Highlights

  • Features and noticeable improvements
    • Power plan management
      Now BenchmarkDotNet executes all benchmarks with enabled High-Performance power plan (configurable, Windows-only). You can find some details below. #68 #952
    • Better Environment Variables API
      Now we have some additional extension methods which allow defining environment variables in user jobs. In the previous version, users always had to set an array of environment variables like this: job.With(new[] { new EnvironmentVariable("a", "b") }). Now it's possible to define an environment variable like job.With(new EnvironmentVariable("a", "b")) or job.WithEnvironmentVariable("a", "b"). Also, it's possible to clear the list of environment variables via job.WithoutEnvironmentVariables(). #1069 #1080
    • Better outlier messages
      The previous version of BenchmarkDotNet printed information about detected or removed outliers like this: "3 outliers were detected". It was nice, but it didn't provide additional information about these outliers (users had to read the full log to find the outliers values). Now BenchmarkDotNet prints additional information about outlier values like this: "3 outliers were detected (2.50 us..2.70 us)". e983cd31
    • Support modern CPU architecture names
      In the environment information section, BenchmarkDotNet prints not only the processor brand string, but also its architecture (e.g., "Intel Core i7-4770K CPU 3.50GHz (Haswell)"). However, it failed to recognize some recent processors. Now it's able to detect the architecture for modern Intel processors correctly (Kaby Lake, Kaby Lake R, Kaby Lake G, Amber Lake Y, Coffee Lake, Cannon Lake, Whiskey Lake). 995e053d
    • Introduce BenchmarkDotNet.Annotations
      Currently, BenchmarkDotNet targets .NET Standard 2.0. It makes some users unhappy because they want to define benchmarks in projects with lower target framework. We decided to start working on the BenchmarkDotNet.Annotations NuGet package which targets .NET Standard 1.0 and contains classes that users need to define their benchmarks. However, it's not easy to refactor the full source code base and move all relevant public APIs to this package. In v0.11.5, we did the first step and moved some of these APIs to BenchmarkDotNet.Annotations. We want to continue moving classes to this package and get full-featured annotation package in the future. #1084 #1096
    • Use InProcessEmitToolchain by default in InProcess benchmarks
      In BenchmarkDotNet 0.11.4, we introduced InProcessEmitToolchain. It's a new, full-featured InProcess toolchain which allows executing benchmarks in the current process without spawning additional process per benchmark. It supports [Arguments], [ArgumentsSource], passing the arguments by out, ref and returning stack-only types like Span<T>. However, in v0.11.4, it can be activated only if InProcessEmitToolchain is declared explicitly. Now it's enabled by default when [InProcessAttribute] is used. #1093
    • Introduce an option which prevents overwriting results
      Currently, BenchmarkDotNet overwrites results each time when the benchmarks are executed. It allows avoiding tons of obsolete files in the BenchmarkDotNet.Artifacts folder. However, the behavior doesn't fit all use cases: sometimes users want to keep results for old benchmark runs. Now we have a special option for it. The option can be activated via --noOverwrite console line argument or DontOverwriteResults extension method for IConfig #1074 #1083
  • Other improvements and bug fixes
    • Diagnostics and validation
      • Better benchmark declaration error processing
        In the previous version, BenchmarkDotNet threw an exception when some benchmark methods had an invalid declaration (e.g., invalid signature or invalid access modifiers). Now it prints a nice error message without ugly stack traces. #1107
      • Better error message for users who want to debug benchmarks
        #1073
      • Don't show the same validation error multiple times
        Now each error will be printed only once. #1079
      • Restrict MemoryDiagnoserAttribute usage to class
        Now it's impossible to accidentally mark a method with this attribute. #1119 #1122
    • Export
      • Better indentation in disassembly listings
        Now DissassemblyDiagnoser correctly process source code which contains tab as the indentation symbol #1110
      • Fix incorrect indentation for StackOverflow exporter
        Previously, StackOverflow exporter doesn't have a proper indent for job runtimes in the environment information. Now it's fixed. #826 #1104
      • Fix StackOverflowException in XmlExporter.Full
        #1086 #1090
      • Shortify MemoryDiagnoser column titles
        Now we use the following column titles: "Allocated" instead of "Allocated Memory/Op", "Gen 0" instead of "Gen 0/1k Op". The full description of each column can be found in the legend section below the summary table. #1081
    • Benchmark generation and execution
      • Fixed broken Orderers
        The previous version has a nasty bug with custom orderers. Now it's fixed. #1070 #1109
      • Better overhead evaluation
        In the previous version, BenchmarkDotNet evaluated the benchmark overhead as a mean value of all overhead iteration. It was fine in most cases, but in some cases, the mean value can be spoiled by outliers. Now BenchmarkDotNet uses the median value. #1116
      • Respect CopyLocalLockFileAssemblies
        Now BenchmarkDotNet respect the CopyLocalLockFileAssemblies value and copies it to the generated benchmark project. #1068 #1108
      • Disable CodeAnalysisRuleSet for generated benchmarks
        Previously, generated benchmarks may fail if the CodeAnalysisRuleSet is defined in Directory.Build.Props. #1082
      • Supported undefined enum values
        #1020 #1071
    • Other minor improvements and bug fixes

Power plans

In #952, power plan management was implemented. It resolves a pretty old issue #68 which was created more than three years ago. Now BenchmarkDotNet forces OS to execute a benchmark on the High-Performance power plan. You can disable this feature by modifying PowerPlanMode property. Here is an example where we are playing with this value:

[Config(typeof(Config))]
public class IntroPowerPlan
{
    private class Config : ManualConfig
    {
        public Config()
        {
            Add(Job.MediumRun.WithPowerPlan(PowerPlan.HighPerformance));
            Add(Job.MediumRun.WithPowerPlan(PowerPlan.UserPowerPlan));
        }
    }

    [Benchmark]
    public int IterationTest()
    {
        int j = 0;
        for (int i = 0; i < short.MaxValue; ++i)
            j = i;
        return j;
    }

    [Benchmark]
    public int SplitJoin()
        => string.Join(",", new string[1000]).Split(',').Length;
}

And here is an example of the summary table on plugged-off laptop:

        Method |       PowerPlan |     Mean |     Error |    StdDev |
-------------- |---------------- |---------:|----------:|----------:|
 IterationTest | HighPerformance | 40.80 us | 0.4168 us | 0.6109 us |
     SplitJoin | HighPerformance | 13.24 us | 0.2514 us | 0.3763 us |
 IterationTest |   UserPowerPlan | 79.72 us | 2.5623 us | 3.8352 us |
     SplitJoin |   UserPowerPlan | 24.54 us | 2.1062 us | 3.1525 us |

As you can see, the power plan produces a noticeable effect on the benchmark results.

This feature is available on Windows only.

Milestone details

In the v0.11.5 scope, 16 issues were resolved and 16 pull requests were merged. This release includes 44 commits by 12 contributors.

Resolved issues (16)

  • #68 Power management
  • #826 MarkdownExporter.StackOverflow fails to indent jobs' runtime descriptions (assignee: @alinasmirnova)
  • #976 System.NotSupportedException: Line must start with GC (assignee: @adamsitnik)
  • #1020 Errors using undefined enum values as benchmark arguments (assignee: @adamsitnik)
  • #1068 The csproj setting CopyLocalLockFileAssemblies is ignored
  • #1070 System.InvalidOperationException: Sequence contains more than one matching element after 0.11.4 (assignee: @AndreyAkinshin)
  • #1071 Enum flags results into compiler errors (assignee: @adamsitnik)
  • #1073 The error message for users who want to Debug benchmarks is not clear (assignee: @adamsitnik)
  • #1074 Results should be exported to a file with unique name
  • #1079 Dont display the same Validation Error many times (assignee: @adamsitnik)
  • #1086 XmlExporter.Full fails with StackOverflowException
  • #1107 Unhandled Exception: System.InvalidOperationException: Benchmark method '' has incorrect signature. Method shouldn't have any arguments. (assignee: @AndreyAkinshin)
  • #1109 Issue with DefaultOrderer
  • #1110 DisassemblyDiagnoser assumes indentation uses spaces (assignee: @AndreyAkinshin)
  • #1116 Use Median instead of Mean whe deducing Overhead (assignee: @AndreyAkinshin)
  • #1119 MemoryDiagnoserAttribute on methods (assignee: @Rizzen)

Merged pull requests (16)

  • #952 Implemented power-management, add docs (#68) (by @MarekM25)
  • #1080 Improved Environment Variables API, resolves #1069 (by @AndreyAkinshin)
  • #1081 Shortify MemoryDiagnoser column titles (by @AndreyAkinshin)
  • #1082 Make it so that the code analysis settings are disabled. (by @glennawatson)
  • #1083 make it possible to Don't Overwrite Results, fixes #1074 (by @adamsitnik)
  • #1084 introduce BenchmarkDotNet.Annotations (by @adamsitnik)
  • #1088 Typo (by @Stromberg90)
  • #1090 XmlExporter.Full fails (by @daveMueller)
  • #1093 make InProcessEmitToolchain the default one (by @adamsitnik)
  • #1096 move more simple Attributes to BenchmarkDotNet.Annotations (by @adamsitnik)
  • #1104 fix #826 (by @alinasmirnova)
  • #1105 Just spell check (by @sungam3r)
  • #1108 Fix #1068 - Copy value of CopyLocalLockFileAssemblies (by @WojciechNagorski)
  • #1112 Fixed event wire-up before Session acquisition (by @jzabroski)
  • #1122 Restrict MemoryDiagnoserAttribute usage to class (by @Rizzen)
  • #1126 Styling in docfx fashion (by @robertmuehsig)

Commits (44)

  • 0a63e4 Postrelease update of v0.11.4 changelog (by @AndreyAkinshin)
  • f94616 Fix namespace for JobTests (by @AndreyAkinshin)
  • 995e05 Support modern CPUs in ProcessorBrandStringHelper (by @AndreyAkinshin)
  • f946ba Repair custom orderers, fixes #1070 (by @AndreyAkinshin)
  • 1d3783 allow passing Enum Flags and undefined enum values as benchmarks arguments/pa... (by @adamsitnik)
  • dfe9ca make sure that we can pass undefined enum values and the bug never comes back... (by @adamsitnik)
  • 76b467 improve the error message when users try to run the benchmarks in Debug, fixe... (by @adamsitnik)
  • 98d9f8 Dont display the same Validation Error many times, fixes #1079 (by @adamsitnik)
  • 03981c fix the unit test that I broke when I was fixing #1071 (by @adamsitnik)
  • 877aba Throw exception about private benchmark method (by @AndreyAkinshin)
  • e983cd Print some outlier values in OutliersAnalyser (by @AndreyAkinshin)
  • b5d324 Shortify MemoryDiagnoser column titles (#1081) (by @AndreyAkinshin)
  • c5c4c4 handle undefined negative enum values, #1020 (thanks @TylerBrinkley) (by @adamsitnik)
  • 2f273c Improved Environment Variables API, resolves #1069 (#1080) (by @AndreyAkinshin)
  • ff2847 Implement FrequencyTests.ParseTest (by @AndreyAkinshin)
  • 5e25da Make it so that the code analysis settings are disabled for compiled builds (... (by @glennawatson)
  • db701e make it possible to Don't Overwrite Results, fixes #1074 (#1083) (by @adamsitnik)
  • a12d8c introduce BenchmarkDotNet.Annotations (#1084) (by @adamsitnik)
  • dcc40a Typo (#1088) (by @Stromberg90)
  • 2c392a XmlExporter.Full fails #1090 (by @daveMueller)
  • 8968bb make InProcessEmitToolchain the default one (#1093) (by @adamsitnik)
  • 4c9136 move more simple Attributes to BenchmarkDotNet.Annotations (#1096) (by @adamsitnik)
  • 6f524f Added extra output lines for jobs in test (by @alinasmirnova)
  • 27ed8a Logger with prefix works correctly with multiline input (by @alinasmirnova)
  • a846b8 just spell check (#1105) (by @sungam3r)
  • a2da9a Reverted reporter change (by @alinasmirnova)
  • e1c9b9 Merge pull request #1104 from alinasmirnova/master (by @AndreyAkinshin)
  • e45adc Fix #1068 - Copy value of CopyLocalLockFileAssemblies (#1108) (by @WojciechNagorski)
  • 2f823c don't fail with exception if user has written something to output in GlobalCl... (by @adamsitnik)
  • 8bc2cc Fixed event wire-up before Session acquisition (#1112) (by @jzabroski)
  • db3a8f Better handling of benchmark with incorrect signature, fixes #1107 (by @AndreyAkinshin)
  • 8a8e01 Fix GlobalSetupAttributeMethodsMustHaveNoParameters (by @AndreyAkinshin)
  • 52eca7 Better disasm indentation, fixes #1110 (by @AndreyAkinshin)
  • d9901b Use Median instead of Mean for overhead calculations, fixes #1116 (by @AndreyAkinshin)
  • 1c1913 add dotnet/performance to the list of users (by @adamsitnik)
  • e1d4d2 Restrict MemoryDiagnoserAttribute usage to class (#1122), fixes #1119 (by @Rizzen)
  • 8aa6ad Implemented power-management, add docs (#68) (#952) (by @MarekM25)
  • 9c6cad Improve ApplyUserPowerPlan message (by @AndreyAkinshin)
  • d12414 Initial v0.11.5 changelog (by @AndreyAkinshin)
  • 1babcb styling in docfx fashion (by @robertmuehsig)
  • fb1286 Add netstandard2.0 target framework in BenchmarkDotNet.Annotations (by @AndreyAkinshin)
  • b7118e Add BenchmarkDotNet.Annotations in build-and-pack.cmd (by @AndreyAkinshin)
  • 8721a9 Prepare v0.11.5 changelog (by @AndreyAkinshin)
  • 31ea3d Set library version: 0.11.5 (by @AndreyAkinshin)

Contributors (12)

  • Adam Sitnik (@adamsitnik)
  • Alina Smirnova (@alinasmirnova)
  • Andrey Akinshin (@AndreyAkinshin)
  • David Müller (@daveMueller)
  • Glenn (@glennawatson)
  • Ivan Maximov (@sungam3r)
  • John Zabroski (@jzabroski)
  • Marek Moraczyński (@MarekM25)
  • Mark Tkachenko (@Rizzen)
  • Robert Muehsig (@robertmuehsig)
  • Strømberg (@Stromberg90)
  • Wojciech Nagórski (@WojciechNagorski)

Thank you very much!

Additional details

Date: April 2, 2019

Milestone: v0.11.5 (List of commits)

NuGet Packages:

  • https://www.nuget.org/packages/BenchmarkDotNet/0.11.5
  • https://www.nuget.org/packages/BenchmarkDotNet.Diagnostics.Windows/0.11.5
  • https://www.nuget.org/packages/BenchmarkDotNet.Tool/0.11.5
  • https://www.nuget.org/packages/BenchmarkDotNet.Annotations/0.11.5
  • Improve this Doc
In This Article
Back to top Copyright © 2013–2023 .NET Foundation and contributors