Running NSuite

The second stage of the NSuite workflow is running benchmark and validation tests on simulation engines that were installed in the first stage when Installing NSuite.

Benchmarks and validation tests are launched with the respective scripts and

In the example workflow below, NEURON and CoreNEURON are first installed in a path called nrn using a user-specified environment configuration, then benchmark and validation tests are run on the installed engines.

# download and install NEURON and CoreNEURON in a directory called nrn
./ neuron coreneuron --prefix=nrn

# run default benchmarks for NEURON and CoreNEURON
./ neuron coreneuron --prefix=nrn

# run validation tests for NEURON
./ neuron --prefix=nrn

The benchmark and validation runners take as arguments the simulators to test, and the prefix where the simulation engines were installed.


The environment does not have to be specified by the user using the --env flag, because the environment used to configure and build each simulation engine is saved during the installation with, and automatically loaded for each simulation engine by the runners.

Flags and options for benchmark and validation runners are described in detail below.


The full set of command line arguments for the benchmark runner are:

Flag Default value Explanation
--help   Display help message.
simulator none Which simulation engines to benchmark. Any number of the following: {arbor, neuron, coreneuron}.
--prefix current path Path where simulation engines to benchmark were installed by All benchmark inputs and outputs will be saved here. Can be either a relative or absolute path.
--model ring A list of benchmark models to run. At least one of {ring, kway}.
--config small A list of configurations to run for each benchmark model. At least one of {small, medium, large}.
--output '%m/%p/%s' Override default path to benchmark outputs. The provided path name will be appended to prefix. Use --help for all format string options.

The --model and --config flags specify which benchmarks to run and how they should be configured. Currently there are two benchmark models, ring and kway; detailed descriptions are in Benchmarks.

# run default benchmarks with Arbor
./ arbor

# run ring and kway benchmarks with Arbor
./ arbor --model='ring kway'

# run kway benchmark in medium and large configuration with Arbor
./ arbor --model=kway --config='medium large'

Each benchmark model has three configurations to choose from: small, medium and large. The configurations can be used to test simulation engine performance at different scales. For example, the small configuration has fewer cells with with simpler morphologies than the medium and large configurations. The small configuration requires little time to run, and is useful for modelling performance characteristics of simpler models. Likewise, models in large configuration take much longer to run, with considerably more parallel work for benchmarking performance of large models on powerful HPC nodes.


NEURON is used to generate input models for CoreNEURON. Before running a benchmark in CoreNEURON, the benchmark must first be run in NEURON.

Benchmark output

Two forms of output are generated when a benchmark case is run. The first is a summary table printed to standard output, and the second is a CSV file that can be saved for use by tools later analysis of benchmark output. In the example below the kway model is run in the small configuration for Arbor and NEURON.

./ arbor neuron --model=kway --config=small --prefix=install
==  platform:          linux
==  cores per socket:  4
==  threads per core:  1
==  threads:           4
==  sockets:           1
==  mpi:               ON

==  benchmark: arbor kway-small
  cells compartments    wall(s)  throughput  mem-tot(MB) mem-percell(MB)
      2          90       0.041        48.8       0.318       0.159
      4         184       0.038       105.3       0.529       0.132
      8         368       0.039       205.1       0.822       0.103
     16         736       0.058       275.9       1.449       0.091
     32        1462       0.106       301.9       2.642       0.083
     64        2882       0.206       310.7       5.010       0.078
    128        5778       0.406       315.3       9.517       0.074
    256       11516       0.802       319.2      18.705       0.073

==  benchmark: neuron kway-small
  cells compartments    wall(s)  throughput  mem-tot(MB) mem-percell(MB)
      2          84       0.174        11.5           -           -
      4         172       0.179        22.4           -           -
      8         348       0.342        23.4           -           -
     16         688       0.711        22.5           -           -
     32        1384       1.380        23.2           -           -
     64        2792       3.600        17.8           -           -
    128        5596      14.049         9.1           -           -
    256       11188      33.246         7.7           -           -

Benchmark output for each {simulator, model, config} tuple is stored in the output path prefix/output/benchmarks/${output}. By default ${output} is, model/config/simulator, which can be overriden by the --output flag. For the example above, two output files are generated, one for each simulator:


cells,    walltime,      memory,  ranks,threads,    gpu
    2,       0.041,       0.318,      1,      4,     no
    4,       0.038,       0.529,      1,      4,     no
    8,       0.039,       0.822,      1,      4,     no
   16,       0.058,       1.449,      1,      4,     no
   32,       0.106,       2.642,      1,      4,     no
   64,       0.206,       5.010,      1,      4,     no
  128,       0.406,       9.517,      1,      4,     no
  256,       0.802,      18.705,      1,      4,     no


cells,    walltime,      memory,  ranks,threads,    gpu
    2,       0.174,            ,      1,      4,     no
    4,       0.179,            ,      1,      4,     no
    8,       0.342,            ,      1,      4,     no
   16,       0.711,            ,      1,      4,     no
   32,       1.380,            ,      1,      4,     no
   64,       3.600,            ,      1,      4,     no
  128,      14.049,            ,      1,      4,     no
  256,      33.246,            ,      1,      4,     no

Descriptions and units for each column are tabulated below.

Column Units Explanation
Total number of cells in the model.
walltime seconds Time taken to run the simulation. Does not include model building or teardown times.
memory megabytes Total memory allocated during model building and simulation. Measured as the difference in total memory allocated between just after MPI is initialized and the simulation finishing.
The number of MPI ranks.
Number of threads per MPI rank.
If a GPU was used. One of yes/no.

Validation Tests

Validation tests are composed of a model, corresponding to a physical system to be simulated, and a parameter set, which specifies parameters within that system.

The script runs all or a subset of the models for one or more installed simulators, saving test artefacts in a configurable output directory and a presenting pass/fail status for each test on standard output.


The existing validation scripts use functionality from the scipy and xarray Python modules. These modules need to be available in the Python module search path.


SIMULATOR can be any of the simulators installed with By default, will use the current directory as the installation and output base directory. If no models are explicitly selected with the --model option (see below), all models and parameter sets will be run against each specified simulator.

SIMULATOR can optionally have a sequence of _tags_ appended, which are keywords specific to simulator implementations of validation models that change the global behaviour of that simulator. For any given simulator, the set of supported tags may differ from model to model. See the file in each validation model directory for information regarding supported tags.

Options are as follows:

Option Explanation
-h, --help Display help message and exit.
-l, --list-models List all available model/parameter sets.
--prefix=PREFIX Base directory for local installation and output directories. Validation tests may also create reference datasets in PREFIX/cache.
-m, --model=MODEL[/PARAM] A model or model/parameter set to run. MODEL alone is equivalent to MODEL/default.
-r, --refresh Regenerate any required cached reference data sets.
-o, --output=FORMAT Substitute fields in FORMAT and use the resulting absolute or relative path for the validation test output directory. Relative paths are with respect to PREFIX/output/validation.

By default, the outputs for a validation test run are stored in PREFIX/output/validation/SIMULATOR/MODEL/PARAM, corresponding to an output format of %s/%m/%p. Fields in the FORMAT string are substituted as follows:

%T Timestamp of invocation of (ISO 8601/RFC 3339)
%H NSuite git commit hash (with + suffix if modified)
%h NSuite git commit short hash (with + suffix if modified)
%S System name (if defined in system environment script) or host name
%s Simulator name (with tags, if any)
%m Model name
%p Parameter set name
%% Literal ‘%’

Output will print pass/fail information to stdout, but will also record information in the per-test output directories:

File Content
run.out Captured standard output from test script
run.err Captured standard error from test script
status Pass/fail status (see below)

The status is one of:

  1. pass — validation test succeeded.
  2. fail — validation test failed.
  3. missing — no implementation for the validation test found for requested simulator.
  4. error — an error occurred during validation test execution.

The output directory may contain other test artefacts. By convention only, these may include:

File Content Numerical results from simulator run in NetCDF4 format. Computed differences from reference data.