Benchmarking with BenchmarkDotNet

Analogy categoryAnalogies

Why You Can't Just Use a Stopwatch

Naive micro-benchmarking with Stopwatch.StartNew() around a loop is misleading because it doesn't account for JIT warm-up (the first calls to a method run through the interpreter or tier-0 JIT before tiered compilation optimizes it), GC pauses interrupting the timed region, CPU frequency scaling, and background OS noise, all of which can make a benchmark's result vary wildly between runs. BenchmarkDotNet solves this by running your code through a real, isolated process per benchmark, executing a pilot stage to estimate how many iterations are needed, then running enough iterations to get statistically stable measurements, and reporting mean, error, standard deviation, and allocated bytes per operation rather than a single noisy number.

🏏

Cricket analogy: It's like judging a bowler's true pace off one delivery in a casual net session versus using a proper speed gun across a full spell in a match at the WACA; BenchmarkDotNet is the speed gun, not the eyeballed guess.

Writing a Benchmark with [Benchmark] and [MemoryDiagnoser]

A BenchmarkDotNet benchmark class marks the methods you want measured with the [Benchmark] attribute, optionally designating one as [Baseline] so other results are reported as a ratio against it, which makes comparing alternative implementations (say, StringBuilder versus string concatenation) immediately readable. Adding [MemoryDiagnoser] to the class reports Gen0/Gen1/Gen2 collection counts and bytes allocated per operation alongside timing, which often matters more than raw speed, since fewer allocations mean less GC pressure across the whole application, not just in the benchmarked method. You run benchmarks by calling BenchmarkRunner.Run<T>() from a Release-mode console app; BenchmarkDotNet refuses to run meaningfully in Debug mode because the JIT doesn't apply the same optimizations, and it will warn you loudly if you try.

🏏

Cricket analogy: It's like comparing two batting techniques side by side across a full net session with a designated baseline technique to measure improvement against, rather than judging from a single shot; [Baseline] gives that same fair comparison point.

Avoiding Dead Code Elimination in Benchmarks

A subtle trap in hand-rolled benchmarks is that the JIT's optimizer can notice a computed value is never used and eliminate the entire call as dead code, making an operation look impossibly fast because it never actually ran. BenchmarkDotNet sidesteps this by requiring [Benchmark] methods to return their computed value, which the harness consumes, forcing the compiler to keep the work; if you write a void benchmark method that computes something and discards it, you risk measuring nothing at all rather than the operation you intended. This is why every benchmark example returns its result explicitly, even when that value would otherwise be thrown away in normal application code.

🏏

Cricket analogy: It's like a scorer who stops logging a bowler's figures the moment nobody's watching the board, so the spell technically happened but was never recorded; a void benchmark risks the same silent 'it ran but nothing counted' problem.

csharp

[MemoryDiagnoser]
[SimpleJob(RuntimeMoniker.Net80)]
public class StringConcatBenchmarks
{
    private const int Iterations = 1000;

    [Benchmark(Baseline = true)]
    public string StringConcat()
    {
        string result = "";
        for (int i = 0; i < Iterations; i++)
            result += i.ToString();
        return result;
    }

    [Benchmark]
    public string StringBuilderConcat()
    {
        var sb = new StringBuilder();
        for (int i = 0; i < Iterations; i++)
            sb.Append(i);
        return sb.ToString();
    }
}

public class Program
{
    // Must be run from a Release build: dotnet run -c Release
    public static void Main() =>
        BenchmarkRunner.Run<StringConcatBenchmarks>();
}

Never trust benchmark numbers gathered from a Debug build. Debug builds disable tiered JIT optimizations and insert extra instrumentation, so timings can be several times slower and completely unrepresentative of production performance. BenchmarkDotNet will detect and warn about this, but always double check your build configuration first.

Use dotnet run -c Release --filter *StringConcat* style filters when you only want to run a subset of benchmarks in a class with many [Benchmark] methods, which saves significant time during iterative tuning.

A raw Stopwatch loop is unreliable for micro-benchmarks due to JIT warm-up, GC pauses, and OS noise.
BenchmarkDotNet runs each benchmark in an isolated process with a pilot stage to determine a statistically stable iteration count.
[Benchmark] marks methods to measure; [Baseline] on one method makes others report as a ratio against it.
[MemoryDiagnoser] reports allocated bytes and Gen0/1/2 collection counts per operation, often as important as raw speed.
Benchmarks must be run in Release mode; Debug builds produce unrepresentative, misleading results.
BenchmarkRunner.Run<T>() from a console app entry point executes the configured benchmark class.
Filtering benchmarks by name speeds up iterative performance tuning of a large suite.