Evaluation of test outputs

The evaluation of semantic and stochastic test results is done by comparing the numerical output produced by a simulation system to expected outputs.

Format of results data

The expected (or reference) results stored in the file NNNNN-results.csv for each test case NNNNN are simply organized as a table of values. The values will be slightly different depending on whether a TimeCourse or FluxBalanceSteadyState test was requested.

Results data format for TimeCourse tests

The expected (or reference) results stored in the file NNNNN-results.csv are organized as a table of values. The first column is simulation time, and the remaining columns are variables in the model (often species, but not necessarily—they could be compartments, parameters, or reaction rates too) in the same order as they are listed in the variables: line in the NNNNN-settings.txt file. An optional header line is permitted at the top of the file. Here is a short example:

time,S1,S2
0,1.5e-15,0
0.1,1.357256127053693e-15,1.427438729463058e-16
0.2,1.228096128602129e-15,2.719038713978695e-16
0.3,1.111227330205311e-15,3.887726697946884e-16
0.4,1.005480064513687e-15,4.945199354863119e-16
0.5,1.005480064513687e-15,4.945199354863119e-16

The first line of the file lists the columns, and the rest are numerical data. The total number of lines of data in the file is X + 1, where X is the value of the steps: line in the NNNNN-settings.txt file.

It is possible for some values to be Not a Number (indicating the result is not mathematically defined, such as attempting to divide by zero or perform an operation involving infinity). It is also possible for values to be +∞ or -∞. There does not appear to be an agreed-upon standard way of expressing these values in comma-separated files, so the SBML Test Suite uses the following convention:

Value	Symbol used
NaN	NaN
+∞	INF
-∞	-INF

These symbols are treated in a case-insensitive manner by the SBML Test Suite.

Results data format for FluxBalanceSteadyState tests

As for the ‘TimeCourse’ results, FluxBalanceSteadyState test results are stored in the file NNNNN-results.csv, and organized as a table of values. The first, header line indicates the expected variables, and the second line indicates the value of those variables at steady state:

R01,R26,R10,R07,OBJF
1.0,1.0,0.5,0.5,1.0

As for the TimeCourse results, it is also possible for FluxBalanceSteadyState results to contain Not a Number, +∞ and -∞, reported as described in the table above.

Interpreting tolerances and errors

The NNNNN-settings.txt file defines the tolerances allowed on the results of each test case. There are two values: an absolute tolerance, and a relative tolerance. The absolute and relative tolerances are used in the following way.

Let

then a data point U_ij is considered to be within tolerances if and only if the following holds: