Table of Contents

Parameterization

Sample: IntroParams

You can mark one or several fields or properties in your class by the [Params] attribute. In this attribute, you can specify set of values. Every value must be a compile-time constant. As a result, you will get results for each combination of params values.

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroParams
    {
        [Params(100, 200)]
        public int A { get; set; }

        [Params(10, 20)]
        public int B { get; set; }

        [Benchmark]
        public void Benchmark() => Thread.Sleep(A + B + 5);
    }
}

Output

|    Method |   A |  B |     Mean |   Error |  StdDev |
|---------- |---- |--- |---------:|--------:|--------:|
| Benchmark | 100 | 10 | 115.3 ms | 0.13 ms | 0.12 ms |
| Benchmark | 100 | 20 | 125.4 ms | 0.14 ms | 0.12 ms |
| Benchmark | 200 | 10 | 215.5 ms | 0.19 ms | 0.18 ms |
| Benchmark | 200 | 20 | 225.4 ms | 0.17 ms | 0.16 ms |

Sample: IntroParamsSource

In case you want to use a lot of values, you should use [ParamsSource] You can mark one or several fields or properties in your class by the [Params] attribute. In this attribute, you have to specify the name of public method/property which is going to provide the values (something that implements IEnumerable). The source must be within benchmarked type!

Source code

using System.Collections.Generic;
using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroParamsSource
    {
        // property with public setter
        [ParamsSource(nameof(ValuesForA))]
        public int A { get; set; }

        // public field
        [ParamsSource(nameof(ValuesForB))]
        public int B;

        // public property
        public IEnumerable<int> ValuesForA => new[] { 100, 200 };

        // public static method
        public static IEnumerable<int> ValuesForB() => new[] { 10, 20 };

        [Benchmark]
        public void Benchmark() => Thread.Sleep(A + B + 5);
    }
}

Output

|    Method |  B |   A |     Mean |   Error |  StdDev |
|---------- |--- |---- |---------:|--------:|--------:|
| Benchmark | 10 | 100 | 115.5 ms | 0.17 ms | 0.16 ms |
| Benchmark | 10 | 200 | 215.6 ms | 0.15 ms | 0.14 ms |
| Benchmark | 20 | 100 | 125.5 ms | 0.19 ms | 0.18 ms |
| Benchmark | 20 | 200 | 225.5 ms | 0.23 ms | 0.22 ms |

Remarks

A remark about IParam.

You don't need to use IParam anymore since 0.11.0. Just use complex types as you wish and override ToString method to change the display names used in the results.

Sample: IntroParamsAllValues

If you want to use all possible values of an enum or another type with a small number of values, you can use the [ParamsAllValues] attribute, instead of listing all the values by hand. The types supported by the attribute are:

  • bool
  • any enum that is not marked with [Flags]
  • Nullable<T>, where T is an enum or boolean

Source code

using BenchmarkDotNet.Attributes;
using System.Threading;

namespace BenchmarkDotNet.Samples
{
    [DryJob]
    public class IntroParamsAllValues
    {
        public enum CustomEnum
        {
            One = 1,
            Two,
            Three
        }

        [ParamsAllValues]
        public CustomEnum E { get; set; }

        [ParamsAllValues]
        public bool? B { get; set; }

        [Benchmark]
        public void Benchmark()
        {
            Thread.Sleep(
                (int)E * 100 +
                (B == true ? 20 : B == false ? 10 : 0));
        }
    }
}

Output

    Method |     E |     B |     Mean | Error |
---------- |------ |------ |---------:|------:|
 Benchmark |   One |     ? | 101.4 ms |    NA |
 Benchmark |   One | False | 111.1 ms |    NA |
 Benchmark |   One |  True | 122.0 ms |    NA |
 Benchmark |   Two |     ? | 201.3 ms |    NA |
 Benchmark |   Two | False | 212.1 ms |    NA |
 Benchmark |   Two |  True | 221.3 ms |    NA |
 Benchmark | Three |     ? | 301.4 ms |    NA |
 Benchmark | Three | False | 311.5 ms |    NA |
 Benchmark | Three |  True | 320.8 ms |    NA |

Sample: IntroParamsPriority

In order to sort columns of parameters in the results table you can use the Property Priority inside the params attribute. The priority range is [Int32.MinValue;Int32.MaxValue], lower priorities will appear earlier in the column order. The default priority is set to 0.

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroParamsPriority
    {
        [Params(100)]
        public int A { get; set; }

        [Params(10, Priority = -100)]
        public int B { get; set; }

        [Benchmark]
        public void Benchmark() => Thread.Sleep(A + B + 5);
    }
}

Output

|    Method |  B |   A |     Mean |   Error |  StdDev |
|---------- |--- |---- |---------:|--------:|--------:|
| Benchmark | 10 | 100 | 115.4 ms | 0.12 ms | 0.11 ms |

Sample: IntroArguments

As an alternative to using [Params], you can specify arguments for your benchmarks. There are several ways to do it (described below).

The [Arguments] allows you to provide a set of values. Every value must be a compile-time constant (it's C# language limitation for attributes in general). You can also combine [Arguments] with [Params]. As a result, you will get results for each combination of params values.

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArguments
    {
        [Params(true, false)] // Arguments can be combined with Params
        public bool AddExtra5Milliseconds;

        [Benchmark]
        [Arguments(100, 10)]
        [Arguments(100, 20)]
        [Arguments(200, 10)]
        [Arguments(200, 20)]
        public void Benchmark(int a, int b)
        {
            if (AddExtra5Milliseconds)
                Thread.Sleep(a + b + 5);
            else
                Thread.Sleep(a + b);
        }
    }
}

Output

|    Method | AddExtra5Miliseconds |   a |  b |     Mean |     Error |    StdDev |
|---------- |--------------------- |---- |--- |---------:|----------:|----------:|
| Benchmark |                False | 100 | 10 | 110.1 ms | 0.0056 ms | 0.0044 ms |
| Benchmark |                False | 100 | 20 | 120.1 ms | 0.0155 ms | 0.0138 ms |
| Benchmark |                False | 200 | 10 | 210.2 ms | 0.0187 ms | 0.0175 ms |
| Benchmark |                False | 200 | 20 | 220.3 ms | 0.1055 ms | 0.0986 ms |
| Benchmark |                 True | 100 | 10 | 115.3 ms | 0.1375 ms | 0.1286 ms |
| Benchmark |                 True | 100 | 20 | 125.3 ms | 0.1212 ms | 0.1134 ms |
| Benchmark |                 True | 200 | 10 | 215.4 ms | 0.0779 ms | 0.0691 ms |
| Benchmark |                 True | 200 | 20 | 225.4 ms | 0.0775 ms | 0.0725 ms |

Sample: IntroArgumentsSource

In case you want to use a lot of values, you should use [ArgumentsSource].

You can mark one or several fields or properties in your class by the [ArgumentsSource] attribute. In this attribute, you have to specify the name of public method/property which is going to provide the values (something that implements IEnumerable). The source must be within benchmarked type!

Source code

using System;
using System.Collections.Generic;
using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArgumentsSource
    {
        [Benchmark]
        [ArgumentsSource(nameof(Numbers))]
        public double ManyArguments(double x, double y) => Math.Pow(x, y);

        public IEnumerable<object[]> Numbers() // for multiple arguments it's an IEnumerable of array of objects (object[])
        {
            yield return new object[] { 1.0, 1.0 };
            yield return new object[] { 2.0, 2.0 };
            yield return new object[] { 4.0, 4.0 };
            yield return new object[] { 10.0, 10.0 };
        }

        [Benchmark]
        [ArgumentsSource(nameof(TimeSpans))]
        public void SingleArgument(TimeSpan time) => Thread.Sleep(time);

        public IEnumerable<object> TimeSpans() // for single argument it's an IEnumerable of objects (object)
        {
            yield return TimeSpan.FromMilliseconds(10);
            yield return TimeSpan.FromMilliseconds(100);
        }
    }
}

Output

| Method |  x |  y |      Mean |     Error |    StdDev |
|------- |--- |--- |----------:|----------:|----------:|
|    Pow |  1 |  1 |  9.360 ns | 0.0190 ns | 0.0149 ns |
|    Pow |  2 |  2 | 40.624 ns | 0.3413 ns | 0.3192 ns |
|    Pow |  4 |  4 | 40.537 ns | 0.0560 ns | 0.0524 ns |
|    Pow | 10 | 10 | 40.395 ns | 0.3274 ns | 0.3063 ns |

Another example

If the values are complex types you need to override ToString method to change the display names used in the results.

[DryJob]
public class WithNonPrimitiveArgumentsSource
{
    [Benchmark]
    [ArgumentsSource(nameof(NonPrimitive))]
    public void Simple(SomeClass someClass, SomeStruct someStruct)
    {
        for (int i = 0; i < someStruct.RangeEnd; i++)
            Console.WriteLine($"// array.Values[{i}] = {someClass.Values[i]}");
    }

    public IEnumerable<object[]> NonPrimitive()
    {
        yield return new object[] { new SomeClass(Enumerable.Range(0, 10).ToArray()), new SomeStruct(10) };
        yield return new object[] { new SomeClass(Enumerable.Range(0, 15).ToArray()), new SomeStruct(15) };
    }

    public class SomeClass
    {
        public SomeClass(int[] initialValues) => Values = initialValues.Select(val => val * 2).ToArray();

        public int[] Values { get; }

        public override string ToString() => $"{Values.Length} items";
    }

    public struct SomeStruct
    {
        public SomeStruct(int rangeEnd) => RangeEnd = rangeEnd;

        public int RangeEnd { get; }

        public override string ToString() => $"{RangeEnd}";
    }
}

| Method | someClass | someStruct |     Mean | Error |
|------- |---------- |----------- |---------:|------:|
| Simple |  10 items |         10 | 887.2 us |    NA |
| Simple |  15 items |         15 | 963.1 us |    NA |

Sample: IntroArrayParam

Warning

The cost of creating the arguments is not included in the benchmark.

So if you want to pass an array as an argument, we are going to allocate it before running the benchmark, and the benchmark will not include this operation.

Source code

using System;
using System.Collections.Generic;
using System.Linq;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArrayParam
    {
        [Benchmark]
        [ArgumentsSource(nameof(Data))]
        public int ArrayIndexOf(int[] array, int value)
            => Array.IndexOf(array, value);

        [Benchmark]
        [ArgumentsSource(nameof(Data))]
        public int ManualIndexOf(int[] array, int value)
        {
            for (int i = 0; i < array.Length; i++)
                if (array[i] == value)
                    return i;

            return -1;
        }

        public IEnumerable<object[]> Data()
        {
            yield return new object[] { new int[] { 1, 2, 3 }, 4 };
            yield return new object[] { Enumerable.Range(0, 100).ToArray(), 4 };
            yield return new object[] { Enumerable.Range(0, 100).ToArray(), 101 };
        }
    }
}

Output

|        Method |      array | value |      Mean |     Error |    StdDev | Allocated |
|-------------- |----------- |------ |----------:|----------:|----------:|----------:|
|  ArrayIndexOf | Array[100] |     4 | 15.558 ns | 0.0638 ns | 0.0597 ns |       0 B |
| ManualIndexOf | Array[100] |     4 |  5.345 ns | 0.0668 ns | 0.0625 ns |       0 B |
|  ArrayIndexOf |   Array[3] |     4 | 14.334 ns | 0.1758 ns | 0.1558 ns |       0 B |
| ManualIndexOf |   Array[3] |     4 |  2.758 ns | 0.0905 ns | 0.1208 ns |       0 B |
|  ArrayIndexOf | Array[100] |   101 | 78.359 ns | 1.8853 ns | 2.0955 ns |       0 B |
| ManualIndexOf | Array[100] |   101 | 80.421 ns | 0.6391 ns | 0.5978 ns |       0 B |

Sample: IntroArgumentsPriority

Like Params also Argument columns can be sorted in the table result through their Priority. The priority should be defined only once for multiple Arguments and will keep their inner order as they are defined in the method.

Source code

using System.Collections.Generic;
using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArgumentsPriority
    {
        [Params(100, Priority = 0)] // Argument priority can be combined with Params priority
        public int A { get; set; }

        [Arguments(5, Priority = -10)] // Define priority just once for multiple argument attributes
        [Arguments(10)]
        [Arguments(20)]
        [Benchmark]
        public void Benchmark(int b) => Thread.Sleep(A + b);

        [Benchmark]
        [ArgumentsSource(nameof(Numbers), Priority = 10)]
        public void ManyArguments(int c, int d) => Thread.Sleep(A + c + d);

        public IEnumerable<object[]> Numbers()
        {
            yield return new object[] { 1, 2 };
        }
    }
}

Output

|        Method |  b |   A | c | d |     Mean |   Error |  StdDev |
|-------------- |--- |---- |-- |-- |---------:|--------:|--------:|
| ManyArguments |  ? | 100 | 1 | 2 | 103.4 ms | 0.09 ms | 0.08 ms |
|     Benchmark |  5 | 100 | ? | ? | 105.5 ms | 0.21 ms | 0.19 ms |
|     Benchmark | 10 | 100 | ? | ? | 110.5 ms | 0.14 ms | 0.14 ms |
|     Benchmark | 20 | 100 | ? | ? | 120.4 ms | 0.16 ms | 0.15 ms |