A validation test runs a particular model, representing some physical system to simulate, against one or more sets of parameters and compares the output to a reference solution. If the output deviates from the reference by more than a given threshold, the respective test is marked as failed for that simulator.
Simulator output for each model and parameter set is by convention stored in NetCDF format, where it can be analysed with generic tools.
Validation models are set up in the NSuite source tree according to a specific layout.
Data and scripts required to run a particular validation model MODEL will all
be found under in the
validation/MODEL directory. At minimum, there must be
an executable run script called
run (see below) and a default parameter
default.param. Any additional parameter sets must have a
The interpretation of a parameter set file is particular to any given model,
but by convention, and for compatibility with the existing run scripts, they
should comprise a sequence of
key=value assignments, one per line, with
key being a string without any whitespace and
value a (possibly fractional)
Model run scripts¶
A run script is invoked with the following arguments:
- The output directory.
- The simulator name (with tags).
- The parameter set name.
The script should run the implementation of the model for the simulator, if it exists, with the parameters described in the corresponding parameter set file.
The simulator name may have one or more tag suffixes, of the form
these correspond to global flags applied to a simulator to modify its
behaviour. It is hoped that any supported tags for a simulator have the
same meaning across different models;
neuron:firstorder, for example,
should be interpreted uniformly as asking NEURON to run with its first
order solver. This behaviour, however, is not enforced (see the
implementation notes below).
The exit code determines the status of the test:
Apart from cached reference data, any files created by the run script should be
restricted to the output directory. As the files
status in the output directory are written by the
script, these files should not be written to by the run script itself.
Reference data generated by the run script can be stored in the output
directory, or optionally in the NSuite cache directory. The cache
directory is defined in the environment variable
data for a particular model MODEL should be stored in a subdirectory
of the cache directory also named MODEL.
If a validation run script does use cached data, that data should
be regenerated if the environment variable
has a non-empty value.
In order to generate reference data or to construct a simulator implementation of particular model, there may be a requirement to build some extra software at install time.
install-local.sh is run, the directory
validation/src is scanned
for subdirectories containing a
CMakeLists.txt file. These are then
built with CMake unless there is a file named
BUILDFOR in the subdirectory.
BUILDFOR file, if present, contains a whitespace-separated list
of relevant simulators; the project will only be built if the corresponding
simulator has been installed in the invocation of
There is no requirement that validation tests use NetCDF as a format for
simulator results and reference data, but there are two tools provided
thresholdx, that may simplify
the creation of tests that do use NetCDF representations.
comparex program compares variables across two different NetCDF
files, producing deltas, absolute errors, and relative errors. It can
optionally compare a variable against an interpolated reference variable
and estimate a lower bound on the absolute and relative errors via a computed
estimate of the interpolation error.
thresholdx program applies a sequence of simple predicates of
the form variable
op value to the data in a NetCDF file, where
op is one of
>=. It prints the
predicate and a pass or fail message, and exits with a non-zero value
if any of the predicates failed.
If NetCDF is used as the output representation for simulation results, it is strongly recommended that the follow convention be followed:
- All key = value settings in a model paramset should be recorded
as scalar global attributes of type
- In addition, there should be global string attributes
simulatorshould be set to the name of the simulator followed by any tags, separated by ‘:’ and in alphabetical order, e.g.
simulator_buildshould contain version information for the simulator used to produce the output, ideally with sufficient detail to be able to recreate the simulator binary.
validation_modelshould contain the name of the model, but not include the name of the parameter set, e.g.
- Variables should have a
unitsattribute describing the units for the data in SI using standard abbreviations compatible with UDUNITS and as much as feasible, other unit parsing libraries.
- If other metadata is provided, it should broadly follow common NetCDF conventions such as those described by the CF conventions.
The existing run scripts use a helper script
to assist in marshalling parameters and invoking particular model
implementations; please refer to the comments in this script for
details. For a simulator
SIM, the run scripts then look for an
implementation-specific script called
run-SIM, which expects
command line arguments of the form:
run-SIM -o OUTPUT [--tag TAG]... [KEY=VALUE ...]
The simulator-specific run script is responsible for returning the unsupported tag error code if it does not support a requested tag.
A python helper module
nsuite.stdarg is intended to make parsing
these options more straightforward. Similarly, the C++ header
validation/src/include/common_args.h is used for the Arbor
As much as is feasible, it is recommended that model implementations for a given simulator support the same set of tags. Tags used in current implementations include:
binevents: bin event delivery times to simulation dt. Default behaviour is to use precise event times, without any binning.
- NEURON and CoreNEURON:
firstorder: use the first order, fixed time step integrator. Default behaviour is to use the second order fixed time step integrator.