Java Microbenchmark Harness (JMH)

Kim Pedersen
Reading time: 10 minutes.
Page content

This article dives into the Java Microbenchmark Harness aka. JMH. To keep it simple and focused on JHM, I’ll benchmark two search algorithms: linear and binary search on sorted arrays of different length.

Setting Up a Project

The JMH is pretty simple to setup, just follow the command line usage described on the Github page. I used this:

1 mvn archetype:generate \
2   -DinteractiveMode=false \
3   -DarchetypeGroupId=org.openjdk.jmh \
4   -DarchetypeArtifactId=jmh-java-benchmark-archetype \
5   -DgroupId=dk.anno1980 \
6   -DartifactId=java-microbenchmark-harness \
7   -Dversion=1.0

Next import the project into you favorite IDE.

The Example Code To Benchmark

You can find the full example code on Github. That also contains some comments on the possible configuration annotation that can be used.

 1public class SearchBenchmark {
 2
 3  // the size of the arrays to sort, generated in setup
 4  // the @Benchmark methods will run once per element in this array
 5  @Param({"1000", "1000000"})
 6  int size;
 7
 8  // the generated array
 9  int[] array;
10  // a random int to find in the array
11  int toFind;
12
13  @Setup
14  public void setup() {
15    // generate data in setup, to avoid constant folding
16    // (as opposed to having it generated once and reused, a clever compiler might recognize that)
17    RandomDataGenerator randomDataGenerator = new RandomDataGenerator();
18    array = randomDataGenerator.nextPermutation(size * 10, size);
19    Arrays.sort(array);
20    toFind = randomDataGenerator.nextInt(0, size * 10);
21  }
22
23  @Benchmark
24  public int binarySearch() {
25    // make sure to return the result, to avoid dead code elimination
26    return BinarySearch.find(array, toFind);
27  }
28
29  @Benchmark
30  public int linearSearch() {
31    // make sure to return the result, to avoid dead code elimination
32    return LinearSearch.find(array, toFind);
33  }
34}

Running a JMH Project

Simply build and then run the benchmarks.jar

1mvn clean verify
2# You can also consider using the -t parameter, to specify the number of threads to use (default is 1)
3# This will not make the benchmark run faster, but it will run the benchmark thread number times more.
4java -jar target/benchmarks.jar

It takes approximately 20 minutes to run on my machine.

The Output

Human readable output from running the benchmarks.jar with -t 3 looks like this (machine readable output is also an option):

1Benchmark                      (size)   Mode  Cnt      Score      Error   Units
2SearchBenchmark.binarySearch     1000  thrpt   15  66545,321 ± 4187,760  ops/ms
3SearchBenchmark.binarySearch  1000000  thrpt   15  34420,445 ± 2167,309  ops/ms
4SearchBenchmark.linearSearch     1000  thrpt   15   8539,415 ±  830,537  ops/ms
5SearchBenchmark.linearSearch  1000000  thrpt   15     13,226 ±   19,281  ops/ms

It should be no surprise that binary search is faster than linear search. On an array with 1.000 elements binary search is about 8 times faster. For an array with 1.000.000 elements, binary search is 2602 times faster on my machine.

Note that microbenchmarks are difficult to get right