Table of Contents

Statistics

Sample: IntroStatisticsColumns

Source code

using System;
using System.Security.Cryptography;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    [MediumRunJob, SkewnessColumn, KurtosisColumn]
    public class IntroStatisticsColumns
    {
        private const int N = 10000;
        private readonly byte[] data;

        private readonly MD5 md5 = MD5.Create();
        private readonly SHA256 sha256 = SHA256.Create();

        public IntroStatisticsColumns()
        {
            data = new byte[N];
            new Random(42).NextBytes(data);
        }

        [Benchmark(Baseline = true)]
        public byte[] Md5A() => md5.ComputeHash(data);

        [Benchmark]
        public byte[] Md5B() => md5.ComputeHash(data);

        [Benchmark]
        public byte[] Sha256() => sha256.ComputeHash(data);
    }
}

Output

Method Mean Error StdDev Skewness Kurtosis Ratio RatioSD
Md5A 15.91 us 0.0807 us 0.1209 us 0.4067 1.646 1.00 0.00
Md5B 15.89 us 0.0709 us 0.1062 us 0.5893 2.141 1.00 0.01
Sha256 36.62 us 0.6390 us 0.9564 us 1.1363 4.014 2.30 0.06

Sample: IntroPercentiles

The percentile represents a higher boundary for specified percentage of the measurements. For example, 95th percentile = 500ms means that 95% of all samples are not slower than 500ms. This metric is not very useful in microbenchmarks, as the values from consequent runs have a very narrow distribution. However, real-world scenarios often have so-called long tail distribution (due to IO delays, locks, memory access latency and so on), so the average execution time cannot be trusted.

The percentiles allow to include the tail of distribution into the comparison. However, it requires some preparations steps. At first, you should have enough runs to count percentiles from. The IterationCount in the config should be set to 10-20 runs at least.

Second, the count of iterations for each run should not be very high, or the peak timings will be averaged. The IterationTime = 25 works fine for most cases; for long-running benchmarks the Mode = Mode.SingleRun will be the best choice. However, feel free to experiment with the config values.

Third, if you want to be sure that measurements are repeatable, set the LaunchCount to 3 or higher.

And last, don't forget to include the columns into the config. They are not included by default (as said above, these are not too useful for most of the benchmarks). There're predefined StatisticColumn.P0..StatisticColumn.P100 for absolute timing percentiles.

Example

Run the IntroPercentiles sample. It contains three benchmark methods.

  • First delays for 20 ms constantly.
  • The second has random delays for 10..30 ms.
  • And the third delays for 10ms 85 times of 100 and delays for 40ms 15 times of 100.

Here's the output from the benchmark (some columns removed for brevity):

Method Median StdDev Ratio P0 P50 P80 P85 P95 P100
ConstantDelays 20.3813 ms 0.2051 ms 1.00 20.0272 ms 20.3813 ms 20.4895 ms 20.4954 ms 20.5869 ms 21.1471 ms
RandomDelays 19.8055 ms 5.7556 ms 0.97 10.0793 ms 19.8055 ms 25.4173 ms 26.5187 ms 29.0313 ms 29.4550 ms
RareDelays 10.3385 ms 11.4828 ms 0.51 10.0157 ms 10.3385 ms 10.5211 ms 40.0560 ms 40.3992 ms 40.4674 ms

Also, it's very easy to screw the results with incorrect setup. For example, the same code being run with

new Job
{
	IterationCount = 5,
	IterationTime = 500
}

completely hides the peak values:

Method Median StdDev Ratio P0 P50 P80 P85 P95 P100
ConstantDelays 20.2692 ms 0.0308 ms 1.00 20.1986 ms 20.2692 ms 20.2843 ms 20.2968 ms 20.3097 ms 20.3122 ms
RandomDelays 18.9965 ms 0.8601 ms 0.94 18.1339 ms 18.9965 ms 19.8126 ms 19.8278 ms 20.4485 ms 20.9466 ms
RareDelays 14.0912 ms 2.8619 ms 0.70 10.2606 ms 14.0912 ms 15.7653 ms 17.3862 ms 18.6728 ms 18.6940 ms

Source code

using System;
using System.Threading;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Engines;

namespace BenchmarkDotNet.Samples
{
    // Using percentiles for adequate timings representation
    [Config(typeof(Config))]
    [SimpleJob(RunStrategy.ColdStart, launchCount: 4,
        warmupCount: 3, iterationCount: 20, id: "MyJob")]
    public class IntroPercentiles
    {
        // To share between runs.
        // DO NOT do this in production code. The System.Random IS NOT thread safe.
        private static readonly Random Rnd = new Random();

        private class Config : ManualConfig
        {
            public Config()
            {
                AddColumn(
                    StatisticColumn.P0,
                    StatisticColumn.P25,
                    StatisticColumn.P50,
                    StatisticColumn.P67,
                    StatisticColumn.P80,
                    StatisticColumn.P85,
                    StatisticColumn.P90,
                    StatisticColumn.P95,
                    StatisticColumn.P100);
            }
        }

        [Benchmark(Baseline = true)]
        public void ConstantDelays() => Thread.Sleep(20);

        [Benchmark]
        public void RandomDelays() => Thread.Sleep(10 + (int) (20 * Rnd.NextDouble()));

        [Benchmark]
        public void RareDelays()
        {
            int rndTime = 10;
            // Bigger delays for 15% of the runs
            if (Rnd.NextDouble() > 0.85)
            {
                rndTime += 30;
            }

            Thread.Sleep(rndTime);
        }
    }
}

Sample: IntroRankColumn

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Mathematics;
using BenchmarkDotNet.Order;

namespace BenchmarkDotNet.Samples
{
    [ShortRunJob]
    [Orderer(SummaryOrderPolicy.FastestToSlowest)]
    [RankColumn(NumeralSystem.Arabic)]
    [RankColumn(NumeralSystem.Roman)]
    [RankColumn(NumeralSystem.Stars)]
    public class IntroRankColumn
    {
        [Params(1, 2)]
        public int Factor;

        [Benchmark]
        public void Foo() => Thread.Sleep(Factor * 100);

        [Benchmark]
        public void Bar() => Thread.Sleep(Factor * 200);
    }
}

Output

 Method | Factor |     Mean |    Error |    StdDev | Rank | Rank | Rank |
------- |------- |---------:|---------:|----------:|-----:|-----:|-----:|
    Foo |      1 | 100.8 ms | 2.250 ms | 0.1272 ms |    1 |    I |    * |
    Foo |      2 | 200.8 ms | 4.674 ms | 0.2641 ms |    2 |   II |   ** |
    Bar |      1 | 200.9 ms | 2.012 ms | 0.1137 ms |    2 |   II |   ** |
    Bar |      2 | 400.7 ms | 4.509 ms | 0.2548 ms |    3 |  III |  *** |

Sample: IntroMultimodal

Source code

using System;
using System.Threading;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Engines;

namespace BenchmarkDotNet.Samples
{
    [MValueColumn]
    [SimpleJob(RunStrategy.Throughput, 1, 0, -1, 1, "MyJob")]
    public class IntroMultimodal
    {
        private readonly Random rnd = new Random(42);

        private void Multimodal(int n)
            => Thread.Sleep((rnd.Next(n) + 1) * 100);

        [Benchmark] public void Unimodal() => Multimodal(1);
        [Benchmark] public void Bimodal() => Multimodal(2);
        [Benchmark] public void Trimodal() => Multimodal(3);
        [Benchmark] public void Quadrimodal() => Multimodal(4);
    }
}

Output

      Method |     Mean |      Error |      StdDev |   Median | MValue |
------------ |---------:|-----------:|------------:|---------:|-------:|
    Unimodal | 100.5 ms |  0.0713 ms |   0.0667 ms | 100.5 ms |  2.000 |
     Bimodal | 144.5 ms | 16.9165 ms |  49.8787 ms | 100.6 ms |  3.571 |
    Trimodal | 182.5 ms | 27.4285 ms |  80.8734 ms | 200.5 ms |  4.651 |
 Quadrimodal | 226.6 ms | 37.2269 ms | 109.7644 ms | 200.7 ms |  5.882 |

// * Warnings *
MultimodalDistribution
  IntroMultimodal.Bimodal: MainJob     -> It seems that the distribution is bimodal (mValue = 3.57)
  IntroMultimodal.Trimodal: MainJob    -> It seems that the distribution is multimodal (mValue = 4.65)
  IntroMultimodal.Quadrimodal: MainJob -> It seems that the distribution is multimodal (mValue = 5.88)

Sample: IntroOutliers

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using Perfolizer.Mathematics.OutlierDetection;

namespace BenchmarkDotNet.Samples
{
    [Config(typeof(Config))]
    public class IntroOutliers
    {
        private class Config : ManualConfig
        {
            public Config()
            {
                var jobBase = Job.Default.WithWarmupCount(0).WithIterationCount(10).WithInvocationCount(1).WithUnrollFactor(1);
                AddJob(jobBase.WithOutlierMode(OutlierMode.DontRemove).WithId("DontRemoveOutliers"));
                AddJob(jobBase.WithOutlierMode(OutlierMode.RemoveUpper).WithId("RemoveUpperOutliers"));
            }
        }

        private int counter;

        [Benchmark]
        public void Foo()
        {
            counter++;
            int noise = counter % 10 == 0 ? 500 : 0;
            Thread.Sleep(100 + noise);
        }
    }
}

Output

 Method |                 Job | OutlierMode |     Mean |       Error |      StdDev |
------- |-------------------- |------------ |---------:|------------:|------------:|
    Foo |  DontRemoveOutliers |  DontRemove | 150.5 ms | 239.1911 ms | 158.2101 ms |
    Foo | RemoveUpperOutliers | RemoveUpper | 100.5 ms |   0.1931 ms |   0.1149 ms |

// * Hints *
Outliers
  IntroOutliers.Foo: DontRemoveOutliers  -> 1 outlier  was  detected
  IntroOutliers.Foo: RemoveUpperOutliers -> 1 outlier  was  removed