Signature-search thread controls

A path under benchmarks/ on this page names a record of the d-SEAMS 2.0 benchmark campaign. The records and the Snakemake rules that write them are in the private paper repository and the Hugging Face dataset d-SEAMS/dseams2-benchmark-records (docs/records/README.md names the revision); the scripts under scripts/ that time a run stay here.

Commit 9896a478 on perf/trace-threads retimes the hydrate signature search at 1, 2, 4 and 8 threads, after the pool and the row placement changed. The census at every width is the one-thread census. The rows are benchmarks/trace-threads.csv in the benchmark records dataset; the tables below are rendered from it, with each median and type-7 interquartile range recomputed from samples_ms.

The process

scripts/paper_stats.py builds the environment for one width \(n\):

variable

value

OMP_NUM_THREADS

\(n\)

OMP_DYNAMIC

FALSE

OMP_PROC_BIND

close

OMP_PLACES

cores

RAYON_NUM_THREADS

\(n\)

OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, NUMEXPR_NUM_THREADS

1

The process is pinned with taskset -c 0 when \(n = 1\), and with taskset -c 0-(n-1) otherwise. One warm-up is dropped. Eight repeats remain. The median of an even count is the mean of the two central samples. The interquartile range is the type-7 upper quartile minus the lower one. cores is the size of the affinity set. affinity is Cpus_allowed_list. The commit column is the source commit the sitting recorded.

Every kernel

Commit 848426f9 sizes the team on every kernel. caeb71f8 puts the one-thread carried update back on a direct copy and sort, binds a narrowed team to that many cores, and skips the periodic shift inside an interior cutoff cell. 1ce3be3b retimes the same eight kernels after that change. The rows are benchmarks/pool-audit.csv. The commit column is caeb71f8. Eight kernels run on ideal ice Ic: primitive rings, local and cutoff \(q_4\) and \(q_6\), CHILL+, the shell label, and one carried hop-ball update. The sizes are 4096, 32768, 110592 and 884736 molecules. Each size is 32 rows, the eight kernels at 1, 2, 4 and 8 threads. One warm-up, then eight timed calls. The fingerprint of a kernel is the same at every width. meson test docs_pool_audit recomputes each median and type-7 interquartile range from samples_ms.

The rings kernel is primitive::ringNetwork on a neighbour graph built before the timer starts. It is not the full cutoff call. The rings row of benchmarks/paper-cells/sitting-large.csv is seams_frame_rings, which builds the cutoff list and then searches the rings. Those two rows are not the same measurement.

The full cutoff ring call

Several threads inside neighListLinkcell build an orthorhombic or triclinic cutoff list with the cell-list CSR. One thread still uses linkcell’s buffered walk. The rows are the same sorted cutoff set, so the ring search matches across thread counts. cad887b2 times that full call, the neighbour list plus the index conversion plus the depth-6 search, on ice Ic at 884736 molecules. It is interleaved with the 1ce3be3b builder. Each sample is the last of three repeats in a fresh process. The table is benchmarks/cutoff-rings.csv. link is the neighbour-list build. This sitting is not the ringNetwork kernel above, and it is not the rings row of the 884736-molecule sitting.

Rendered from benchmarks/cutoff-rings.csv. Ice Ic at 884736 molecules. linkcell is the 1ce3be3b builder. cell-list is the orthorhombic list inside the ring call. total medians are recomputed from samples_ms.

Full cutoff ring call

code

operation

threads

n

median_ms

iqr_ms

cell-list

total

1

8

654.896

8.296

cell-list

total

2

8

418.741

3.510

cell-list

total

4

8

252.877

1.991

cell-list

total

8

8

173.215

2.554

linkcell

total

1

8

653.639

9.726

linkcell

total

2

8

521.980

4.294

linkcell

total

4

8

323.101

1.161

linkcell

total

8

8

228.632

5.324

cell-list

link

1

8

132.279

cell-list

link

2

8

86.208

cell-list

link

4

8

52.179

cell-list

link

8

8

35.622

linkcell

link

1

8

131.875

linkcell

link

2

8

206.762

linkcell

link

4

8

137.551

linkcell

link

8

8

107.620

cell-list

convert

1

8

28.719

cell-list

convert

2

8

15.368

cell-list

convert

4

8

7.783

cell-list

convert

8

8

4.628

linkcell

convert

1

8

28.935

linkcell

convert

2

8

15.060

linkcell

convert

4

8

7.941

linkcell

convert

8

8

4.544

cell-list

search

1

8

492.962

cell-list

search

2

8

316.537

cell-list

search

4

8

192.863

cell-list

search

8

8

132.576

linkcell

search

1

8

493.166

linkcell

search

2

8

298.425

linkcell

search

4

8

177.345

linkcell

search

8

8

115.945

seams::teamFor is the width of a per-atom walk. Below 2048 atoms it is one thread. Below 8192 atoms, when the caller asked for more than four, it is four. At or above 8192 atoms it is the OpenMP width. seams::TeamWidth sets that width for the rest of the call and restores the caller’s width afterwards. When the width is narrower than the process, it binds the workers once to the first cores of the allowed set, one core for each worker, and sets OMP_PROC_BIND to false so OpenMP does not put them back on the whole mask. The bind is once for that width and is not undone when the call returns. Four workers asked for on an eight-core mask therefore stay on four cores. Bond order, published CHILL+, the k-nearest list, the shell label, and local \(q_l\) use TeamWidth. Those calls do not start a Rayon pool.

seams::adoptThreads no longer pauses the OpenMP team and no longer sets OMP_WAIT_POLICY. It calls prepareCompute, then lc_set_threads. The linkcell pool is still one per process, and its workers still sleep between searches. The OpenMP team is left running, so the next region does not pay the wake-up that used to move by tens of milliseconds.

Enumerating the changed ring sources uses sourceTeam, not teamFor. That search stays on the calling thread until the sources fill the workers, about 80 sources per thread. Fewer than that is a serial enumeration, because the fork costs more than the search. teamFor is a different rule: it would also put a crystal of a few hundred molecules on one thread. On one thread the carried update copies the new rings and sorts them directly, then merges the rings that stay with a direct walk. The parallel index sort and the parallel merge are the several-thread path. They were slower than the direct copy when only one thread ran. Several threads still sort the indices, scan for rings that meet \(T\), and merge the rings that are kept, using teamFor. Dropped ring buffers are kept and reused. They are not freed on the update.

The orthorhombic cutoff list is one shared CSR. Each pair is counted once, from the atom with the lower slot, on the z layer that owns that atom. A layer writes its own atoms and the atoms in the next layer, so layers of one parity do not share an atom. Even layers run together. When the number of layers is odd, the last layer wraps onto layer 0 and runs only after the even layers have stopped. Odd layers run after that. An interior cell has no periodic image. Its thirteen forward neighbours are plain index steps and the shift is zero, so that cell skips the periodic-shift arithmetic. A cell on the boundary still applies the shift. One thread uses the same half stencil without a team. Published CHILL+ and cutoff \(q_6\) both read this list. The one-thread and several-thread pair lists match.

The linkcell cap below 4096 molecules, and the row placement below \(2^{20}\) pairs, are the signature-search rules above. They are not replaced by teamFor.

C names seams_set_num_threads and seams_get_num_threads.

These tables are rendered from benchmarks/trace-threads.csv.

signature cages, sI

cell

molecules

threads

cores

n

median_ms

iqr_ms

sI 2x2x2

368

1

8

8

1.671145

0.042545

sI 2x2x2

368

2

8

8

1.194630

0.144788

sI 2x2x2

368

4

8

8

0.845288

0.093749

sI 2x2x2

368

8

8

8

0.889254

0.067923

sI 4x4x4

2944

1

8

8

11.155500

0.052250

sI 4x4x4

2944

2

8

8

7.572065

0.876455

sI 4x4x4

2944

4

8

8

5.198810

0.470322

sI 4x4x4

2944

8

8

8

5.295710

0.664862

sI 6x6x6

9936

1

8

8

34.513400

0.145950

sI 6x6x6

9936

2

8

8

29.893900

0.793650

sI 6x6x6

9936

4

8

8

15.499400

0.382550

sI 6x6x6

9936

8

8

8

11.406800

1.692000

sI 8x8x8

23552

1

8

8

79.230800

0.313300

sI 8x8x8

23552

2

8

8

50.058800

0.456100

sI 8x8x8

23552

4

8

8

33.261250

4.790825

sI 8x8x8

23552

8

8

8

26.431450

0.435175

signature cages, sII

cell

molecules

threads

cores

n

median_ms

iqr_ms

sII 2x2x2

1088

1

8

8

4.815100

0.054395

sII 2x2x2

1088

2

8

8

3.748070

0.253115

sII 2x2x2

1088

4

8

8

2.464180

0.202713

sII 2x2x2

1088

8

8

8

2.543545

0.405943

sII 4x4x4

8704

1

8

8

35.907600

0.137650

sII 4x4x4

8704

2

8

8

22.485750

0.262925

sII 4x4x4

8704

4

8

8

14.209000

0.490550

sII 4x4x4

8704

8

8

8

11.173800

3.512300

sII 6x6x6

29376

1

8

8

121.799000

0.416000

sII 6x6x6

29376

2

8

8

74.774150

1.357250

sII 6x6x6

29376

4

8

8

45.948400

0.754300

sII 6x6x6

29376

8

8

8

31.094250

0.210175

sII 8x8x8

69632

1

8

8

292.810500

2.154000

sII 8x8x8

69632

2

8

8

180.097000

1.709250

sII 8x8x8

69632

4

8

8

111.554000

3.843000

sII 8x8x8

69632

8

8

8

73.108700

0.648400

Rendered from benchmarks/pool-audit.csv. Each size is one 32-row table: eight kernels at 1, 2, 4 and 8 threads. median_ms and iqr_ms are recomputed from samples_ms.

ice Ic, 4096 molecules

kernel

threads

cores

n

median_ms

iqr_ms

rings

1

8

8

2.466905

0.075503

rings

2

8

8

1.495700

0.050533

rings

4

8

8

0.940597

0.151879

rings

8

8

8

0.633803

0.038622

q4-local

1

8

8

0.667118

0.018324

q4-local

2

8

8

0.389290

0.007018

q4-local

4

8

8

0.267276

0.001623

q4-local

8

8

8

0.262831

0.002778

q6-local

1

8

8

0.771595

0.003968

q6-local

2

8

8

0.451925

0.007984

q6-local

4

8

8

0.380479

0.008200

q6-local

8

8

8

0.391622

0.020511

q4-cutoff

1

8

8

0.612442

0.006985

q4-cutoff

2

8

8

0.378497

0.008923

q4-cutoff

4

8

8

0.242802

0.039543

q4-cutoff

8

8

8

0.243986

0.037453

q6-cutoff

1

8

8

0.637176

0.018393

q6-cutoff

2

8

8

0.412152

0.013879

q6-cutoff

4

8

8

0.230091

0.008668

q6-cutoff

8

8

8

0.242103

0.006373

chill

1

8

8

0.907933

0.045465

chill

2

8

8

0.552909

0.012054

chill

4

8

8

0.348419

0.008053

chill

8

8

8

0.349171

0.004840

symmetry

1

8

8

0.913122

0.011245

symmetry

2

8

8

0.691176

0.007351

symmetry

4

8

8

0.414814

0.012065

symmetry

8

8

8

0.414618

0.021285

hop

1

8

8

0.978623

0.022944

hop

2

8

8

0.597030

0.034203

hop

4

8

8

0.410767

0.017664

hop

8

8

8

0.381499

0.009265

ice Ic, 32768 molecules

kernel

threads

cores

n

median_ms

iqr_ms

rings

1

8

8

19.360550

0.379825

rings

2

8

8

10.805550

0.215250

rings

4

8

8

6.449795

0.082868

rings

8

8

8

4.263060

0.236065

q4-local

1

8

8

5.905020

0.029900

q4-local

2

8

8

3.253615

0.053870

q4-local

4

8

8

1.909015

0.111800

q4-local

8

8

8

1.299955

0.023633

q6-local

1

8

8

6.897740

0.058900

q6-local

2

8

8

3.728225

0.037745

q6-local

4

8

8

3.084135

0.023317

q6-local

8

8

8

1.956415

0.062893

q4-cutoff

1

8

8

5.263250

0.058237

q4-cutoff

2

8

8

2.753380

0.027987

q4-cutoff

4

8

8

1.770700

0.079300

q4-cutoff

8

8

8

1.213175

0.046030

q6-cutoff

1

8

8

5.426560

0.037195

q6-cutoff

2

8

8

2.854520

0.034102

q6-cutoff

4

8

8

1.822180

0.086745

q6-cutoff

8

8

8

1.316010

0.059367

chill

1

8

8

7.429445

0.132305

chill

2

8

8

3.889890

0.125662

chill

4

8

8

2.437750

0.107562

chill

8

8

8

1.680220

0.097135

symmetry

1

8

8

7.109115

0.078567

symmetry

2

8

8

4.891060

0.148090

symmetry

4

8

8

2.897955

0.084083

symmetry

8

8

8

1.865990

0.062373

hop

1

8

8

7.778165

0.065072

hop

2

8

8

4.798770

0.679093

hop

4

8

8

2.998450

0.069390

hop

8

8

8

2.032610

0.068672

ice Ic, 110592 molecules

kernel

threads

cores

n

median_ms

iqr_ms

rings

1

8

8

62.012050

0.158025

rings

2

8

8

35.865750

0.459375

rings

4

8

8

21.042200

0.582800

rings

8

8

8

13.537600

0.173700

q4-local

1

8

8

20.345300

0.172975

q4-local

2

8

8

11.096000

0.052775

q4-local

4

8

8

6.376390

0.068050

q4-local

8

8

8

4.051045

0.039383

q6-local

1

8

8

23.794950

0.063875

q6-local

2

8

8

12.896700

0.082325

q6-local

4

8

8

7.310575

0.056100

q6-local

8

8

8

4.613615

0.274352

q4-cutoff

1

8

8

16.288450

0.182575

q4-cutoff

2

8

8

8.978580

0.105928

q4-cutoff

4

8

8

5.604980

0.102873

q4-cutoff

8

8

8

3.591890

0.234900

q6-cutoff

1

8

8

16.811850

0.136250

q6-cutoff

2

8

8

9.404800

0.225812

q6-cutoff

4

8

8

5.770415

0.166370

q6-cutoff

8

8

8

3.852335

0.112190

chill

1

8

8

24.159000

0.204625

chill

2

8

8

13.322850

0.096675

chill

4

8

8

8.032445

0.149675

chill

8

8

8

5.275310

0.284212

symmetry

1

8

8

24.393350

0.072975

symmetry

2

8

8

17.057900

0.214850

symmetry

4

8

8

13.666650

0.453375

symmetry

8

8

8

6.160520

0.194820

hop

1

8

8

26.381800

0.228900

hop

2

8

8

16.760500

2.426175

hop

4

8

8

9.924360

0.561755

hop

8

8

8

7.194500

0.529503

ice Ic, 884736 molecules

kernel

threads

cores

n

median_ms

iqr_ms

rings

1

8

8

495.117000

2.365750

rings

2

8

8

299.999500

2.423750

rings

4

8

8

184.109500

3.346250

rings

8

8

8

112.990000

2.134500

q4-local

1

8

8

201.293000

6.906000

q4-local

2

8

8

108.775500

2.981250

q4-local

4

8

8

61.171350

2.056050

q4-local

8

8

8

40.111950

2.153775

q6-local

1

8

8

239.785500

4.282750

q6-local

2

8

8

129.450500

4.643500

q6-local

4

8

8

80.323450

6.257450

q6-local

8

8

8

51.277600

0.623550

q4-cutoff

1

8

8

154.053000

7.017500

q4-cutoff

2

8

8

80.075400

1.599300

q4-cutoff

4

8

8

43.635150

1.151825

q4-cutoff

8

8

8

25.552200

0.728500

q6-cutoff

1

8

8

158.820000

2.469750

q6-cutoff

2

8

8

85.462600

0.926100

q6-cutoff

4

8

8

45.467300

0.915550

q6-cutoff

8

8

8

26.620450

0.664350

chill

1

8

8

232.527500

2.841500

chill

2

8

8

122.900000

0.568750

chill

4

8

8

67.026200

0.327775

chill

8

8

8

39.195350

0.346225

symmetry

1

8

8

214.114000

11.427750

symmetry

2

8

8

138.153500

7.535750

symmetry

4

8

8

79.281800

2.953025

symmetry

8

8

8

48.350600

1.473875

hop

1

8

8

256.551500

6.978250

hop

2

8

8

171.991500

8.024750

hop

4

8

8

103.765500

4.727250

hop

8

8

8

62.761750

2.417975