BenchmarkDotNet v0.11.5

Highlights

  • Features and noticeable improvements
    • Power plan management
      Now BenchmarkDotNet executes all benchmarks with enabled High-Performance power plan (configurable, Windows-only). You can find some details below. #68 #952
    • Better Environment Variables API
      Now we have some additional extension methods which allow defining environment variables in user jobs. In the previous version, users always had to set an array of environment variables like this: job.With(new[] { new EnvironmentVariable("a", "b") }). Now it's possible to define an environment variable like job.With(new EnvironmentVariable("a", "b")) or job.WithEnvironmentVariable("a", "b"). Also, it's possible to clear the list of environment variables via job.WithoutEnvironmentVariables(). #1069 #1080
    • Better outlier messages
      The previous version of BenchmarkDotNet printed information about detected or removed outliers like this: "3 outliers were detected". It was nice, but it didn't provide additional information about these outliers (users had to read the full log to find the outliers values). Now BenchmarkDotNet prints additional information about outlier values like this: "3 outliers were detected (2.50 us..2.70 us)". e983cd31
    • Support modern CPU architecture names
      In the environment information section, BenchmarkDotNet prints not only the processor brand string, but also its architecture (e.g., "Intel Core i7-4770K CPU 3.50GHz (Haswell)"). However, it failed to recognize some recent processors. Now it's able to detect the architecture for modern Intel processors correctly (Kaby Lake, Kaby Lake R, Kaby Lake G, Amber Lake Y, Coffee Lake, Cannon Lake, Whiskey Lake). 995e053d
    • Introduce BenchmarkDotNet.Annotations
      Currently, BenchmarkDotNet targets .NET Standard 2.0. It makes some users unhappy because they want to define benchmarks in projects with lower target framework. We decided to start working on the BenchmarkDotNet.Annotations NuGet package which targets .NET Standard 1.0 and contains classes that users need to define their benchmarks. However, it's not easy to refactor the full source code base and move all relevant public APIs to this package. In v0.11.5, we did the first step and moved some of these APIs to BenchmarkDotNet.Annotations. We want to continue moving classes to this package and get full-featured annotation package in the future. #1084 #1096
    • Use InProcessEmitToolchain by default in InProcess benchmarks
      In BenchmarkDotNet 0.11.4, we introduced InProcessEmitToolchain. It's a new, full-featured InProcess toolchain which allows executing benchmarks in the current process without spawning additional process per benchmark. It supports [Arguments], [ArgumentsSource], passing the arguments by out, ref and returning stack-only types like Span<T>. However, in v0.11.4, it can be activated only if InProcessEmitToolchain is declared explicitly. Now it's enabled by default when [InProcessAttribute] is used. #1093
    • Introduce an option which prevents overwriting results
      Currently, BenchmarkDotNet overwrites results each time when the benchmarks are executed. It allows avoiding tons of obsolete files in the BenchmarkDotNet.Artifacts folder. However, the behavior doesn't fit all use cases: sometimes users want to keep results for old benchmark runs. Now we have a special option for it. The option can be activated via --noOverwrite console line argument or DontOverwriteResults extension method for IConfig #1074 #1083
  • Other improvements and bug fixes
    • Diagnostics and validation
      • Better benchmark declaration error processing
        In the previous version, BenchmarkDotNet threw an exception when some benchmark methods had an invalid declaration (e.g., invalid signature or invalid access modifiers). Now it prints a nice error message without ugly stack traces. #1107
      • Better error message for users who want to debug benchmarks
        #1073
      • Don't show the same validation error multiple times
        Now each error will be printed only once. #1079
      • Restrict MemoryDiagnoserAttribute usage to class
        Now it's impossible to accidentally mark a method with this attribute. #1119 #1122
    • Export
      • Better indentation in disassembly listings
        Now DissassemblyDiagnoser correctly process source code which contains tab as the indentation symbol #1110
      • Fix incorrect indentation for StackOverflow exporter
        Previously, StackOverflow exporter doesn't have a proper indent for job runtimes in the environment information. Now it's fixed. #826 #1104
      • Fix StackOverflowException in XmlExporter.Full
        #1086 #1090
      • Shortify MemoryDiagnoser column titles
        Now we use the following column titles: "Allocated" instead of "Allocated Memory/Op", "Gen 0" instead of "Gen 0/1k Op". The full description of each column can be found in the legend section below the summary table. #1081
    • Benchmark generation and execution
      • Fixed broken Orderers
        The previous version has a nasty bug with custom orderers. Now it's fixed. #1070 #1109
      • Better overhead evaluation
        In the previous version, BenchmarkDotNet evaluated the benchmark overhead as a mean value of all overhead iteration. It was fine in most cases, but in some cases, the mean value can be spoiled by outliers. Now BenchmarkDotNet uses the median value. #1116
      • Respect CopyLocalLockFileAssemblies
        Now BenchmarkDotNet respect the CopyLocalLockFileAssemblies value and copies it to the generated benchmark project. #1068 #1108
      • Disable CodeAnalysisRuleSet for generated benchmarks
        Previously, generated benchmarks may fail if the CodeAnalysisRuleSet is defined in Directory.Build.Props. #1082
      • Supported undefined enum values
        #1020 #1071
    • Other minor improvements and bug fixes

Power plans

In #952, power plan management was implemented. It resolves a pretty old issue #68 which was created more than three years ago. Now BenchmarkDotNet forces OS to execute a benchmark on the High-Performance power plan. You can disable this feature by modifying PowerPlanMode property. Here is an example where we are playing with this value:

[Config(typeof(Config))]
public class IntroPowerPlan
{
    private class Config : ManualConfig
    {
        public Config()
        {
            Add(Job.MediumRun.WithPowerPlan(PowerPlan.HighPerformance));
            Add(Job.MediumRun.WithPowerPlan(PowerPlan.UserPowerPlan));
        }
    }

    [Benchmark]
    public int IterationTest()
    {
        int j = 0;
        for (int i = 0; i < short.MaxValue; ++i)
            j = i;
        return j;
    }

    [Benchmark]
    public int SplitJoin()
        => string.Join(",", new string[1000]).Split(',').Length;
}

And here is an example of the summary table on plugged-off laptop:

        Method |       PowerPlan |     Mean |     Error |    StdDev |
-------------- |---------------- |---------:|----------:|----------:|
 IterationTest | HighPerformance | 40.80 us | 0.4168 us | 0.6109 us |
     SplitJoin | HighPerformance | 13.24 us | 0.2514 us | 0.3763 us |
 IterationTest |   UserPowerPlan | 79.72 us | 2.5623 us | 3.8352 us |
     SplitJoin |   UserPowerPlan | 24.54 us | 2.1062 us | 3.1525 us |

As you can see, the power plan produces a noticeable effect on the benchmark results.

This feature is available on Windows only.

Milestone details

In the v0.11.5 scope, 16 issues were resolved and 16 pull requests were merged. This release includes 44 commits by 12 contributors.

Resolved issues (16)

  • #68 Power management
  • #826 MarkdownExporter.StackOverflow fails to indent jobs' runtime descriptions (assignee: @alinasmirnova)
  • #976 System.NotSupportedException: Line must start with GC (assignee: @adamsitnik)
  • #1020 Errors using undefined enum values as benchmark arguments (assignee: @adamsitnik)
  • #1068 The csproj setting CopyLocalLockFileAssemblies is ignored
  • #1070 System.InvalidOperationException: Sequence contains more than one matching element after 0.11.4 (assignee: @AndreyAkinshin)
  • #1071 Enum flags results into compiler errors (assignee: @adamsitnik)
  • #1073 The error message for users who want to Debug benchmarks is not clear (assignee: @adamsitnik)
  • #1074 Results should be exported to a file with unique name
  • #1079 Dont display the same Validation Error many times (assignee: @adamsitnik)
  • #1086 XmlExporter.Full fails with StackOverflowException
  • #1107 Unhandled Exception: System.InvalidOperationException: Benchmark method '' has incorrect signature. Method shouldn't have any arguments. (assignee: @AndreyAkinshin)
  • #1109 Issue with DefaultOrderer
  • #1110 DisassemblyDiagnoser assumes indentation uses spaces (assignee: @AndreyAkinshin)
  • #1116 Use Median instead of Mean whe deducing Overhead (assignee: @AndreyAkinshin)
  • #1119 MemoryDiagnoserAttribute on methods (assignee: @Rizzen)

Merged pull requests (16)

Commits (44)

Contributors (12)

Thank you very much!

Additional details

Date: April 2, 2019

Milestone: v0.11.5 (List of commits)

NuGet Packages: