Table of Contents

Choosing RunStrategy

If you run a benchmark, you always (explicitly or implicitly) use a job. Each Job has the RunStrategy parameter which allows switching between different benchmark modes. The default RunStrategy is Throughput, and it works fine for most cases. However, other strategies are also useful in some specific cases.

Throughput

Throughput is the default RunStrategy, works perfectly for microbenchmarking. It's automatically choosing the amount of operation in main iterations based on a set of pilot iterations. The amount of iterations will also be chosen automatically based on accuracy job settings. A benchmark method should have a steady state.

Of course, you can manually set all the characteristics. An example:

[SimpleJob(launchCount: 3, warmupCount: 10, iterationCount: 30)]
public class MyBenchmarkClass

Sample: IntroColdStart

If you want to measure cold start (without the pilot and warmup stage), the ColdStart strategy is your choice.

Usage

[SimpleJob(RunStrategy.ColdStart, launchCount:50)]
public class MyBenchmarkClass

Source code

using System;
using System.Threading;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Engines;

namespace BenchmarkDotNet.Samples
{
    [SimpleJob(RunStrategy.ColdStart, iterationCount: 5)]
    [MinColumn, MaxColumn, MeanColumn, MedianColumn]
    public class IntroColdStart
    {
        private bool firstCall;

        [Benchmark]
        public void Foo()
        {
            if (firstCall == false)
            {
                firstCall = true;
                Console.WriteLine("// First call");
                Thread.Sleep(1000);
            }
            else
                Thread.Sleep(10);
        }
    }
}

Output

Result       1: 1 op, 1002034900.00 ns, 1.0020 s/op
Result       2: 1 op, 10219700.00 ns, 10.2197 ms/op
Result       3: 1 op, 10406200.00 ns, 10.4062 ms/op
Result       4: 1 op, 10473900.00 ns, 10.4739 ms/op
Result       5: 1 op, 10449400.00 ns, 10.4494 ms/op

 Method |     Mean |      Error |   StdDev |      Min |        Max |   Median |
------- |---------:|-----------:|---------:|---------:|-----------:|---------:|
    Foo | 208.7 ms | 1,707.4 ms | 443.5 ms | 10.22 ms | 1,002.0 ms | 10.45 ms |

Sample: IntroMonitoring

If a benchmark method takes at least 100ms, you can also use the Monitoring strategy. In this case, the pilot stage will be omitted, by default you get 1 iteration = 1 operation (or you can manually set amount of operation in an iteration). Also you can use [IterationSetup] and [IterationCleanup] in this case: it shouldn't affect time measurements (but it can affect results of MemoryDiagnoser). It's a perfect mode for benchmarks which doesn't have a steady state and the performance distribution is tricky: Monitoring will help you to collect a set of measurements and get statistics.

Usage

[SimpleJob(RunStrategy.Monitoring, launchCount: 10, warmupCount: 0, iterationCount: 100)]
public class MyBenchmarkClass

Source code

using System;
using System.Threading;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Engines;

namespace BenchmarkDotNet.Samples
{
    [SimpleJob(RunStrategy.Monitoring, iterationCount: 10, id: "MonitoringJob")]
    [MinColumn, Q1Column, Q3Column, MaxColumn]
    public class IntroMonitoring
    {
        private Random random = new Random(42);

        [Benchmark]
        public void Foo()
        {
            Thread.Sleep(random.Next(10) * 10);
        }
    }
}

Output

Result       1: 1 op, 61552600.00 ns, 61.5526 ms/op
Result       2: 1 op, 10141700.00 ns, 10.1417 ms/op
Result       3: 1 op, 10482900.00 ns, 10.4829 ms/op
Result       4: 1 op, 50410900.00 ns, 50.4109 ms/op
Result       5: 1 op, 10421400.00 ns, 10.4214 ms/op
Result       6: 1 op, 20556100.00 ns, 20.5561 ms/op
Result       7: 1 op, 70473200.00 ns, 70.4732 ms/op
Result       8: 1 op, 50581700.00 ns, 50.5817 ms/op
Result       9: 1 op, 10559000.00 ns, 10.5590 ms/op
Result      10: 1 op, 70496300.00 ns, 70.4963 ms/op
Method Mean Error StdDev Min Q1 Q3 Max
Foo 36.57 ms 40.03 ms 26.47 ms 10.14 ms 10.48 ms 61.55 ms 70.50 ms