Validation

A validation test runs a particular model, representing some physical system to simulate, against one or more sets of parameters and compares the output to a reference solution. If the output deviates from the reference by more than a given threshold, the respective test is marked as failed for that simulator.

Simulator output for each model and parameter set is by convention stored in NetCDF format, where it can be analysed with generic tools.

Architecture

Validation models are set up in the NSuite source tree according to a specific layout.

Data and scripts required to run a particular validation model MODEL will all be found under in the validation/MODEL directory. At minimum, there must be an executable run script called run (see below) and a default parameter set default.param. Any additional parameter sets must have a .param suffix.

The interpretation of a parameter set file is particular to any given model, but by convention, and for compatibility with the existing run scripts, they should comprise a sequence of key=value assignments, one per line, with key being a string without any whitespace and value a (possibly fractional) decimal number.

Model run scripts

A run script is invoked with the following arguments:

1. The output directory.
2. The simulator name (with tags).
3. The parameter set name.

The script should run the implementation of the model for the simulator, if it exists, with the parameters described in the corresponding parameter set file.

The simulator name may have one or more tag suffixes, of the form :tag — these correspond to global flags applied to a simulator to modify its behaviour. It is hoped that any supported tags for a simulator have the same meaning across different models; neuron:firstorder, for example, should be interpreted uniformly as asking NEURON to run with its first order solver. This behaviour, however, is not enforced (see the implementation notes below).

The exit code determines the status of the test:

Exit code	Interpretation
0	Success
96	Test failure
97	Missing implementation
98	Unsupported tag
other	Execution error

Apart from cached reference data, any files created by the run script should be restricted to the output directory. As the files run.out, run.err, and status in the output directory are written by the run-validation.sh script, these files should not be written to by the run script itself.

Reference data generated by the run script can be stored in the output directory, or optionally in the NSuite cache directory. The cache directory is defined in the environment variable ns_cache_path; data for a particular model MODEL should be stored in a subdirectory of the cache directory also named MODEL.

If a validation run script does use cached data, that data should be regenerated if the environment variable ns_cache_refresh has a non-empty value.

Building tests

In order to generate reference data or to construct a simulator implementation of particular model, there may be a requirement to build some extra software at install time.

When install-local.sh is run, the directory validation/src is scanned for subdirectories containing a CMakeLists.txt file. These are then built with CMake unless there is a file named BUILDFOR in the subdirectory.

The BUILDFOR file, if present, contains a whitespace-separated list of relevant simulators; the project will only be built if the corresponding simulator has been installed in the invocation of install-local.sh.

Common tools

There is no requirement that validation tests use NetCDF as a format for simulator results and reference data, but there are two tools provided in common/bin, viz. comparex and thresholdx, that may simplify the creation of tests that do use NetCDF representations.

The comparex program compares variables across two different NetCDF files, producing deltas, absolute errors, and relative errors. It can optionally compare a variable against an interpolated reference variable and estimate a lower bound on the absolute and relative errors via a computed estimate of the interpolation error.

The thresholdx program applies a sequence of simple predicates of the form variable op value to the data in a NetCDF file, where op is one of =, <, >, <=, >=. It prints the predicate and a pass or fail message, and exits with a non-zero value if any of the predicates failed.

NetCDF conventions

If NetCDF is used as the output representation for simulation results, it is strongly recommended that the follow convention be followed:

• All key = value settings in a model paramset should be recorded as scalar global attributes of type NC_DOUBLE.
• In addition, there should be global string attributes simulator, simulator_build, and validation_model:
  • simulator should be set to the name of the simulator followed by any tags, separated by ‘:’ and in alphabetical order, e.g. neuron:firstorder.
  • simulator_build should contain version information for the simulator used to produce the output, ideally with sufficient detail to be able to recreate the simulator binary.
  • validation_model should contain the name of the model, but not include the name of the parameter set, e.g. rallpack1, not rallpack1/default.
• Variables should have a units attribute describing the units for the data in SI using standard abbreviations compatible with UDUNITS and as much as feasible, other unit parsing libraries.
• If other metadata is provided, it should broadly follow common NetCDF conventions such as those described by the CF conventions.

Implementation notes

The existing run scripts use a helper script scripts/model_common.sh to assist in marshalling parameters and invoking particular model implementations; please refer to the comments in this script for details. For a simulator SIM, the run scripts then look for an implementation-specific script called run-SIM, which expects command line arguments of the form:

run-SIM -o OUTPUT [--tag TAG]... [KEY=VALUE ...]

The simulator-specific run script is responsible for returning the unsupported tag error code if it does not support a requested tag.

A python helper module nsuite.stdarg is intended to make parsing these options more straightforward. Similarly, the C++ header validation/src/include/common_args.h is used for the Arbor model implementations.

As much as is feasible, it is recommended that model implementations for a given simulator support the same set of tags. Tags used in current implementations include:

• Arbor:
  • binevents: bin event delivery times to simulation dt. Default behaviour is to use precise event times, without any binning.
• NEURON and CoreNEURON:
  • firstorder: use the first order, fixed time step integrator. Default behaviour is to use the second order fixed time step integrator.