Reproduce the literature head-to-head

A path under benchmarks/ on this page names a record of the d-SEAMS 2.0 benchmark campaign. The records and the Snakemake rules that write them are in the private paper repository and the Hugging Face dataset d-SEAMS/dseams2-benchmark-records (docs/records/README.md names the revision); the scripts under scripts/ that time a run stay here.

scripts/h2h_rc3_sitting.py writes benchmarks/rc3-sitting.csv. One warm-up, then six timed calls. The median and the type-7 interquartile range are the columns on the sitting page. This page is how each guest code is given a thread width, and what that width did to the median.

The driver starts every row under taskset. One thread is taskset -c 0. Eight threads are taskset -c 0-7. It also sets OMP_NUM_THREADS to that width and OMP_DYNAMIC=FALSE. Each code then receives the same width through its own control.

code

what is timed

thread control

freud

local q4 and q6

freud.parallel.set_num_threads(n)

OVITO

CHILL+, primitive rings, polyhedral template matching

OVITO_THREAD_COUNT=n before import ovito

mdapy

CHILL+, q4, q6, rings

mdapy.set_num_threads(n)

vesin

full cell-list neighbour pairs

NeighborList(..., n_threads=n, algorithm="cell_list")

PLUMED

symfunc q4 and q6

PLUMED_NUM_THREADS was unset, so both columns are one thread

rings-statistics-matter

primitive rings, Rust backend

ring_statistics(..., num_threads=n)

Polyhedral template matching is the OVITO modifier. Its row is PTM. rings-statistics-matter is the rsm package.

python3 scripts/h2h_rc3_sitting.py

What the medians do

On ideal ice Ic at 110592 and 884736 molecules, freud, OVITO, mdapy, vesin, and rings-statistics-matter are faster at eight threads than at one. OVITO’s template matching and its ring finder fall with the same control as its CHILL+. meson test docs_h2h_threads checks those inequalities against the CSV.

The PLUMED rows in this sitting were measured on one thread because PLUMED_NUM_THREADS was unset. The driver set OMP_NUM_THREADS and left PLUMED_NUM_THREADS alone. The installed header plumed/tools/OpenMP.h exposes PLMD::OpenMP::getNumThreads, which returns one thread when that variable is unset. That is why the threads=1 column and the threads=8 column of these rows are the same one-thread measurement. benchmarks/paper-cells/sitting-large.csv sets PLUMED_NUM_THREADS to the width of the row, and that q6 time falls from one thread to eight. These rc3 rows are not that measurement.

#include "plumed/tools/OpenMP.h"
#include <cstdio>

int main() {
  unsigned threads = PLMD::OpenMP::getNumThreads();
  std::printf("plumed threads %u\n", threads);
  return threads == 0 ? 1 : 0;
}

Rendered from benchmarks/rc3-sitting.csv. t1 and t8 are the medians in the threads=1 and threads=8 columns. ratio is t1 / t8. A PLUMED row was measured on one thread because PLUMED_NUM_THREADS was unset. Both of its columns are that measurement.

One thread against eight

code

operation

molecules

t1_ms

t8_ms

ratio

note

freud

q4

110592

276.580

51.302

5.39

freud

q4

884736

2358.091

431.376

5.47

freud

q6

110592

362.326

68.760

5.27

freud

q6

884736

2973.372

550.161

5.40

OVITO

CHILL+

110592

622.424

88.685

7.02

OVITO

CHILL+

884736

4890.789

666.288

7.34

OVITO

PTM

110592

2790.624

370.341

7.54

OVITO

PTM

884736

21304.364

2849.382

7.48

OVITO

rings

110592

583.334

354.605

1.65

OVITO

rings

884736

4883.899

2582.748

1.89

mdapy

CHILL+

110592

61.938

12.379

5.00

mdapy

CHILL+

884736

744.487

125.578

5.93

mdapy

q4

110592

64.687

12.400

5.22

mdapy

q4

884736

807.101

96.238

8.39

mdapy

q6

110592

79.232

14.572

5.44

mdapy

q6

884736

921.574

107.347

8.59

mdapy

rings

110592

2276.747

375.967

6.06

mdapy

rings

884736

19221.718

3361.913

5.72

vesin

neighbours

110592

51.194

22.269

2.30

vesin

neighbours

884736

1112.702

196.866

5.65

rings-statistics-matter

rings

110592

1151.862

264.953

4.35

rings-statistics-matter

rings

884736

11504.451

2665.750

4.32

PLUMED

q4

110592

1310.025

1316.423

1.00

one thread; PLUMED_NUM_THREADS unset

PLUMED

q4

884736

10053.680

10122.158

0.99

one thread; PLUMED_NUM_THREADS unset

PLUMED

q6

110592

1685.454

1680.281

1.00

one thread; PLUMED_NUM_THREADS unset

PLUMED

q6

884736

12947.767

12818.837

1.01

one thread; PLUMED_NUM_THREADS unset