Skip to content

Performance dashboard

Pull requests to Mallard are tested on an NVIDIA A100 GPU before they merge: CUDA builds, the test suites, MPI tests and the performance set in benchmarks/perf/ of the repository. Each benchmark runs three times after a warm-up and keeps the best, and is compared with its A100 baseline in benchmarks/perf/baselines.csv. A benchmark slower than its baseline by more than both 5% and the spread of its runs is a regression and fails the commit's gpu/perf status. Baselines change only on purpose, after a performance change, so a step in the baseline marks one.

Each point is a merge into main whose tested commit already contained everything on main before it, so the measured code is the code of main after the merge. Hover over (or tap) a point for its pull request; select it to open the pull request on GitHub. The Performance page describes the benchmarks and the measurements behind them.

Summary

11 merges into main from Oct 4 to Oct 15, on one NVIDIA A100-SXM4-80GB.

No regression recorded: in none of the 11 merges was a benchmark slower than its baseline by more than both 5% and the spread of its runs.

BenchmarkLatest, cells/sBaselinevs baselinesince first mergeMerges
teno5_2d53.6M54.2M-1.1%-1.1%11
muscl_2d291M265M+9.8%+9.8%11
teno5_3d5.87M5.85M+0.3%+122.3%11
air_3d54.3M52.9M+2.6%+0.7%4
flame_3d6.69M6.77M-1.2%+0.6%4
h2o2@dt=1e-0811.2M9.71M+15.8%+72.4%11
h2o2@dt=1e-065.43M5.44M-0.4%+105.2%11
gri30@dt=1e-08526k524k+0.4%+0.4%11
gri30@dt=1e-06515k529k-2.7%-2.7%11

Benchmark history

Throughput in cells per second (cells advanced by one time step or, for chemistry alone, one splitting step), higher is better. The flow benchmarks run the full solver; h2o2 and gri30 integrate the chemistry alone, at two time-step sizes.

  • best of 3 runs, cells per second
  • baseline ±5%
  • release
  • ▼ regression
teno5_2d2D Riemann problem, TENO-E 5, 1M quadrilateralsLatest: 53.6M cells/s, -1.1% vs baseline
52M54M56Mv0.6.0v0.7.0v0.8.0#169 (Oct 4): 54.2M cells/s. In-situ statistics and probes#177 (Oct 4): 54.1M cells/s; -0.2% vs baseline 54.2M. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 53.2M cells/s; -1.8% vs baseline 54.2M. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 53.6M cells/s; -1.1% vs baseline 54.2M. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 53.7M cells/s; -0.9% vs baseline 54.2M. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 53.8M cells/s; -0.7% vs baseline 54.2M. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 53.7M cells/s; -0.9% vs baseline 54.2M. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 53.6M cells/s; -1.1% vs baseline 54.2M. Conversion warnings (4/n): solver#263 (Oct 15): 53.1M cells/s; -2.0% vs baseline 54.2M. Curved high-order boundaries (#162)#265 (Oct 15): 53.5M cells/s; -1.3% vs baseline 54.2M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 53.6M cells/s; -1.1% vs baseline 54.2M. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main
muscl_2d2D Riemann problem, MUSCL, 1M quadrilateralsLatest: 291M cells/s, +9.8% vs baseline
260M280Mv0.6.0v0.7.0v0.8.0#169 (Oct 4): 265M cells/s. In-situ statistics and probes#177 (Oct 4): 266M cells/s; +0.4% vs baseline 265M. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 253M cells/s; -4.5% vs baseline 265M. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 254M cells/s; -4.2% vs baseline 265M. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 291M cells/s; +9.8% vs baseline 265M. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 291M cells/s; +9.8% vs baseline 265M. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 290M cells/s; +9.4% vs baseline 265M. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 291M cells/s; +9.8% vs baseline 265M. Conversion warnings (4/n): solver#263 (Oct 15): 292M cells/s; +10.2% vs baseline 265M. Curved high-order boundaries (#162)#265 (Oct 15): 293M cells/s; +10.6% vs baseline 265M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 291M cells/s; +9.8% vs baseline 265M. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main
teno5_3d3D Taylor–Green vortex, viscous, TENO-E 5Latest: 5.87M cells/s, +0.3% vs baseline
4M6Mv0.6.0v0.7.0v0.8.0#169 (Oct 4): 2.64M cells/s. In-situ statistics and probes#177 (Oct 4): 2.65M cells/s; +0.4% vs baseline 2.64M. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 5.73M cells/s; -2.1% vs baseline 5.85M. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 5.74M cells/s; -1.9% vs baseline 5.85M. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 5.76M cells/s; -1.5% vs baseline 5.85M. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 5.75M cells/s; -1.7% vs baseline 5.85M. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 5.75M cells/s; -1.7% vs baseline 5.85M. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 5.74M cells/s; -1.9% vs baseline 5.85M. Conversion warnings (4/n): solver#263 (Oct 15): 5.77M cells/s; -1.4% vs baseline 5.85M. Curved high-order boundaries (#162)#265 (Oct 15): 5.88M cells/s; +0.5% vs baseline 5.85M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 5.87M cells/s; +0.3% vs baseline 5.85M. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main
air_3d3D flow of air as a multicomponent mixtureLatest: 54.3M cells/s, +2.6% vs baseline
50M52M54Mv0.6.0v0.7.0v0.8.0#271 (Oct 10): 53.9M cells/s; +1.9% vs baseline 52.9M. Conversion warnings (4/n): solver#263 (Oct 15): 54.3M cells/s; +2.6% vs baseline 52.9M. Curved high-order boundaries (#162)#265 (Oct 15): 54.1M cells/s; +2.3% vs baseline 52.9M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 54.3M cells/s; +2.6% vs baseline 52.9M. Enable -Wconversion -Wsign-conversionOct 10Oct 15merges into main
flame_3d3D propane–air flame, two-step chemistry, TFLESLatest: 6.69M cells/s, -1.2% vs baseline
6.4M6.6M6.8M7Mv0.6.0v0.7.0v0.8.0#271 (Oct 10): 6.65M cells/s; -1.8% vs baseline 6.77M. Conversion warnings (4/n): solver#263 (Oct 15): 6.69M cells/s; -1.2% vs baseline 6.77M. Curved high-order boundaries (#162)#265 (Oct 15): 6.69M cells/s; -1.2% vs baseline 6.77M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 6.69M cells/s; -1.2% vs baseline 6.77M. Enable -Wconversion -Wsign-conversionOct 10Oct 15merges into main
h2o2@dt=1e-08H2/O2 chemistry alone, dt = 1e-8 sLatest: 11.2M cells/s, +15.8% vs baseline
6M8M10Mv0.6.0v0.7.0v0.8.0#169 (Oct 4): 6.52M cells/s. In-situ statistics and probes#177 (Oct 4): 6.49M cells/s; -0.5% vs baseline 6.52M. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 6.46M cells/s; -1.0% vs baseline 6.52M. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 6.5M cells/s; -0.4% vs baseline 6.52M. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 6.54M cells/s; +0.3% vs baseline 6.52M. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 6.56M cells/s; +0.6% vs baseline 6.52M. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 6.55M cells/s; +0.4% vs baseline 6.52M. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 11.2M cells/s; +15.4% vs baseline 9.71M. Conversion warnings (4/n): solver#263 (Oct 15): 11.2M cells/s; +15.8% vs baseline 9.71M. Curved high-order boundaries (#162)#265 (Oct 15): 11.3M cells/s; +15.8% vs baseline 9.71M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 11.2M cells/s; +15.8% vs baseline 9.71M. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main
h2o2@dt=1e-06H2/O2 chemistry alone, dt = 1e-6 sLatest: 5.43M cells/s, -0.4% vs baseline
3M4M5Mv0.6.0v0.7.0v0.8.0#169 (Oct 4): 2.64M cells/s. In-situ statistics and probes#177 (Oct 4): 2.61M cells/s; -1.2% vs baseline 2.64M. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 2.6M cells/s; -1.5% vs baseline 2.64M. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 2.61M cells/s; -1.5% vs baseline 2.64M. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 3.06M cells/s; +15.6% vs baseline 2.64M. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 3.05M cells/s; +15.5% vs baseline 2.64M. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 3.05M cells/s; +15.4% vs baseline 2.64M. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 5.42M cells/s; -0.5% vs baseline 5.44M. Conversion warnings (4/n): solver#263 (Oct 15): 5.42M cells/s; -0.4% vs baseline 5.44M. Curved high-order boundaries (#162)#265 (Oct 15): 5.43M cells/s; -0.3% vs baseline 5.44M. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 5.43M cells/s; -0.4% vs baseline 5.44M. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main
gri30@dt=1e-08GRI-Mech 3.0 chemistry alone, dt = 1e-8 sLatest: 526k cells/s, +0.4% vs baseline
500k520k540kv0.6.0v0.7.0v0.8.0#169 (Oct 4): 524k cells/s. In-situ statistics and probes#177 (Oct 4): 509k cells/s; -2.9% vs baseline 524k. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 506k cells/s; -3.4% vs baseline 524k. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 508k cells/s; -3.1% vs baseline 524k. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 533k cells/s; +1.7% vs baseline 524k. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 530k cells/s; +1.3% vs baseline 524k. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 533k cells/s; +1.7% vs baseline 524k. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 527k cells/s; +0.5% vs baseline 524k. Conversion warnings (4/n): solver#263 (Oct 15): 527k cells/s; +0.6% vs baseline 524k. Curved high-order boundaries (#162)#265 (Oct 15): 528k cells/s; +0.7% vs baseline 524k. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 526k cells/s; +0.4% vs baseline 524k. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main
gri30@dt=1e-06GRI-Mech 3.0 chemistry alone, dt = 1e-6 sLatest: 515k cells/s, -2.7% vs baseline
500k520k540k560kv0.6.0v0.7.0v0.8.0#169 (Oct 4): 529k cells/s. In-situ statistics and probes#177 (Oct 4): 513k cells/s; -3.1% vs baseline 529k. Regularize fractional reaction orders below 1 at vanishing concentration#245 (Oct 9): 509k cells/s; -3.8% vs baseline 529k. Carbuncle test: a drifting shock, not one at rest on cell faces#249 (Oct 9): 511k cells/s; -3.4% vs baseline 529k. Release 0.7.0: version in CMake and CITATION.cff#262 (Oct 9): 531k cells/s; +0.3% vs baseline 529k. Chemistry load balancing with SIMPLER coupling#267 (Oct 9): 520k cells/s; -1.7% vs baseline 529k. Conversion warnings (1/n): common, physics, chemistry, io#269 (Oct 9): 525k cells/s; -0.9% vs baseline 529k. Conversion warnings (2/n): mesh, boundary#271 (Oct 10): 516k cells/s; -2.6% vs baseline 529k. Conversion warnings (4/n): solver#263 (Oct 15): 516k cells/s; -2.5% vs baseline 529k. Curved high-order boundaries (#162)#265 (Oct 15): 516k cells/s; -2.5% vs baseline 529k. TENO 3D: less device memory per cell, setup peak no higher than steady state (#58)#276 (Oct 15): 515k cells/s; -2.7% vs baseline 529k. Enable -Wconversion -Wsign-conversionOct 4Oct 15merges into main

Multi-GPU scaling

Parallel efficiency of the multi-GPU runs behind multi-GPU scaling, with each partitioner Mallard offers.

NVIDIA A100-80GB, 4 per node, NVLink within a node, HDR InfiniBand with GPUDirect RDMA between nodes, GPU-aware MPI, one rank per GPU. Steady time per step, setup excluded; the best of repeated runs.

3D strong scalingTaylor–Green vortex (viscous), 96³ hexahedra, TENO-E 5. Efficiency T₁ / (n Tₙ). Measured in #261.
  • multi-jagged (default)
  • Hilbert
  • graph (dKaMinPar)
60%70%80%90%100%1481216GPUsmulti-jagged (default), 1 GPU: 154 ms per step, 100% efficiencymulti-jagged (default), 4 GPUs: 41.2 ms per step, 93% efficiencymulti-jagged (default), 8 GPUs: 23.1 ms per step, 83% efficiencymulti-jagged (default), 12 GPUs: 16.6 ms per step, 77% efficiencymulti-jagged (default), 16 GPUs: 13.1 ms per step, 73% efficiencyHilbert, 1 GPU: 154 ms per step, 100% efficiencyHilbert, 4 GPUs: 41.3 ms per step, 93% efficiencyHilbert, 8 GPUs: 23.1 ms per step, 83% efficiencyHilbert, 12 GPUs: 17.8 ms per step, 72% efficiencyHilbert, 16 GPUs: 13.2 ms per step, 73% efficiencygraph (dKaMinPar), 1 GPU: 154 ms per step, 100% efficiencygraph (dKaMinPar), 8 GPUs: 23.5 ms per step, 82% efficiencygraph (dKaMinPar), 12 GPUs: 17.2 ms per step, 75% efficiencygraph (dKaMinPar), 16 GPUs: 13.3 ms per step, 72% efficiency73% multi-jagged (default)73% Hilbert72% graph (dKaMinPar)60%70%80%90%100%1481216GPUsmulti-jagged (default), 1 GPU: 154 ms per step, 100% efficiencymulti-jagged (default), 4 GPUs: 41.2 ms per step, 93% efficiencymulti-jagged (default), 8 GPUs: 23.1 ms per step, 83% efficiencymulti-jagged (default), 12 GPUs: 16.6 ms per step, 77% efficiencymulti-jagged (default), 16 GPUs: 13.1 ms per step, 73% efficiencyHilbert, 1 GPU: 154 ms per step, 100% efficiencyHilbert, 4 GPUs: 41.3 ms per step, 93% efficiencyHilbert, 8 GPUs: 23.1 ms per step, 83% efficiencyHilbert, 12 GPUs: 17.8 ms per step, 72% efficiencyHilbert, 16 GPUs: 13.2 ms per step, 73% efficiencygraph (dKaMinPar), 1 GPU: 154 ms per step, 100% efficiencygraph (dKaMinPar), 8 GPUs: 23.5 ms per step, 82% efficiencygraph (dKaMinPar), 12 GPUs: 17.2 ms per step, 75% efficiencygraph (dKaMinPar), 16 GPUs: 13.3 ms per step, 72% efficiency73% multi-jagged (default)73% Hilbert72% graph (dKaMinPar)
Times per step
GPUs1481216
multi-jagged (default)154 ms41.2 ms23.1 ms16.6 ms13.1 ms
Hilbert154 ms41.3 ms23.1 ms17.8 ms13.2 ms
graph (dKaMinPar)154 ms23.5 ms17.2 ms13.3 ms
3D weak scalingTaylor–Green vortex (viscous), 64³ hexahedra per GPU, TENO-E 5. Efficiency T₁ / Tₙ. Measured in #261.
  • multi-jagged (default)
  • Hilbert
  • graph (dKaMinPar)
60%70%80%90%100%1481216GPUsmulti-jagged (default), 1 GPU: 48.1 ms per step, 100% efficiencymulti-jagged (default), 4 GPUs: 49.8 ms per step, 97% efficiencymulti-jagged (default), 8 GPUs: 50.5 ms per step, 95% efficiencymulti-jagged (default), 12 GPUs: 51.7 ms per step, 93% efficiencymulti-jagged (default), 16 GPUs: 52.7 ms per step, 91% efficiencyHilbert, 1 GPU: 48.1 ms per step, 100% efficiencyHilbert, 4 GPUs: 49.8 ms per step, 97% efficiencyHilbert, 8 GPUs: 50.8 ms per step, 95% efficiencyHilbert, 12 GPUs: 54.7 ms per step, 88% efficiencyHilbert, 16 GPUs: 55.8 ms per step, 86% efficiencygraph (dKaMinPar), 1 GPU: 48.1 ms per step, 100% efficiencygraph (dKaMinPar), 8 GPUs: 51.4 ms per step, 94% efficiencygraph (dKaMinPar), 12 GPUs: 52.7 ms per step, 91% efficiencygraph (dKaMinPar), 16 GPUs: 53.9 ms per step, 89% efficiency91% multi-jagged (default)89% graph (dKaMinPar)86% Hilbert60%70%80%90%100%1481216GPUsmulti-jagged (default), 1 GPU: 48.1 ms per step, 100% efficiencymulti-jagged (default), 4 GPUs: 49.8 ms per step, 97% efficiencymulti-jagged (default), 8 GPUs: 50.5 ms per step, 95% efficiencymulti-jagged (default), 12 GPUs: 51.7 ms per step, 93% efficiencymulti-jagged (default), 16 GPUs: 52.7 ms per step, 91% efficiencyHilbert, 1 GPU: 48.1 ms per step, 100% efficiencyHilbert, 4 GPUs: 49.8 ms per step, 97% efficiencyHilbert, 8 GPUs: 50.8 ms per step, 95% efficiencyHilbert, 12 GPUs: 54.7 ms per step, 88% efficiencyHilbert, 16 GPUs: 55.8 ms per step, 86% efficiencygraph (dKaMinPar), 1 GPU: 48.1 ms per step, 100% efficiencygraph (dKaMinPar), 8 GPUs: 51.4 ms per step, 94% efficiencygraph (dKaMinPar), 12 GPUs: 52.7 ms per step, 91% efficiencygraph (dKaMinPar), 16 GPUs: 53.9 ms per step, 89% efficiency91% multi-jagged (default)89% graph (dKaMinPar)86% Hilbert
Times per step
GPUs1481216
multi-jagged (default)48.1 ms49.8 ms50.5 ms51.7 ms52.7 ms
Hilbert48.1 ms49.8 ms50.8 ms54.7 ms55.8 ms
graph (dKaMinPar)48.1 ms51.4 ms52.7 ms53.9 ms
2D strong scaling across nodes2D Riemann problem, Hilbert partitions. Efficiency relative to one node of 4 GPUs. Measured in #228, #261.
  • TENO-E 5, 4M cells
  • TENO-E 5, 16M cells
  • MUSCL, 16M cells
60%70%80%90%100%481216GPUsTENO-E 5, 4M cells, 4 GPUs: 20.3 ms per step, 100% efficiencyTENO-E 5, 4M cells, 8 GPUs: 10.9 ms per step, 93% efficiencyTENO-E 5, 4M cells, 12 GPUs: 7.79 ms per step, 87% efficiencyTENO-E 5, 4M cells, 16 GPUs: 6.14 ms per step, 83% efficiencyTENO-E 5, 16M cells, 4 GPUs: 76.6 ms per step, 100% efficiencyTENO-E 5, 16M cells, 8 GPUs: 39.5 ms per step, 97% efficiencyTENO-E 5, 16M cells, 12 GPUs: 27.2 ms per step, 94% efficiencyTENO-E 5, 16M cells, 16 GPUs: 20.7 ms per step, 93% efficiencyMUSCL, 16M cells, 4 GPUs: 14.6 ms per step, 100% efficiencyMUSCL, 16M cells, 8 GPUs: 7.67 ms per step, 95% efficiencyMUSCL, 16M cells, 12 GPUs: 5.49 ms per step, 89% efficiencyMUSCL, 16M cells, 16 GPUs: 4.12 ms per step, 89% efficiency93% TENO-E 5, 16M cells89% MUSCL, 16M cells83% TENO-E 5, 4M cells60%70%80%90%100%481216GPUsTENO-E 5, 4M cells, 4 GPUs: 20.3 ms per step, 100% efficiencyTENO-E 5, 4M cells, 8 GPUs: 10.9 ms per step, 93% efficiencyTENO-E 5, 4M cells, 12 GPUs: 7.79 ms per step, 87% efficiencyTENO-E 5, 4M cells, 16 GPUs: 6.14 ms per step, 83% efficiencyTENO-E 5, 16M cells, 4 GPUs: 76.6 ms per step, 100% efficiencyTENO-E 5, 16M cells, 8 GPUs: 39.5 ms per step, 97% efficiencyTENO-E 5, 16M cells, 12 GPUs: 27.2 ms per step, 94% efficiencyTENO-E 5, 16M cells, 16 GPUs: 20.7 ms per step, 93% efficiencyMUSCL, 16M cells, 4 GPUs: 14.6 ms per step, 100% efficiencyMUSCL, 16M cells, 8 GPUs: 7.67 ms per step, 95% efficiencyMUSCL, 16M cells, 12 GPUs: 5.49 ms per step, 89% efficiencyMUSCL, 16M cells, 16 GPUs: 4.12 ms per step, 89% efficiency93% TENO-E 5, 16M cells89% MUSCL, 16M cells83% TENO-E 5, 4M cells
Times per step
GPUs481216
TENO-E 5, 4M cells20.3 ms10.9 ms7.79 ms6.14 ms
TENO-E 5, 16M cells76.6 ms39.5 ms27.2 ms20.7 ms
MUSCL, 16M cells14.6 ms7.67 ms5.49 ms4.12 ms