SBML.org — the global portal for all things SBML

Case Descriptions

Contents


This page explains in more detail the various files included with each SBML Test Suite test case, and how those files are intended to be interpreted. The same files are used in the standalone SBML Test Suite.

Directory contents

Each test case consists of a collection of files put into a directory named after the case. These names are five digit numbers (e.g., "00001", "00002", "00003", etc.). Each directory contains inside it the following files. In the table below, the "Computed" column indicates files that are generated automatically from the other files.

File name Explanation Computed?
NNNNN-model.m A human-readable plain-text description of the cases, along with feature tags, and formatting using a wiki-like syntax, all embedded as a comment in the file. For some cases, this file also contains the definition of the model in a compact format (originally based on the syntax of MathSBML). Image:icon-unchecked.gif
NNNNN-model.html An HTML file generated from the the -model.m file by a filter that converts the plain-text description into HTML. This is the HTML used by the Test Suite software when displaying the description of a case to users. Image:icon-checked.gif
NNNNN-settings.txt A summary of the case run parameters. Most of the information is pulled out automatically from the CSV file after a test case simulation result is finalized. See below for an explanation. Image:icon-unchecked.gif
NNNNN-sbml-l1v2.xml The model in SBML Level 1 Version 2 format. Image:icon-unchecked.gif
NNNNN-sbml-l2v1.xml The model in SBML Level 2 Version 1 format. Image:icon-unchecked.gif
NNNNN-sbml-l2v2.xml The model in SBML Level 2 Version 2 format. Image:icon-unchecked.gif
NNNNN-sbml-l2v3.xml The model in SBML Level 2 Version 3 format. Image:icon-unchecked.gif
NNNNN-sbml-l2v4.xml The model in SBML Level 2 Version 4 format. Image:icon-unchecked.gif
NNNNN-sbml-l3v1.xml The model in SBML Level 3 Version 1 format. Image:icon-unchecked.gif
NNNNN-sbml-l1v2-sedml.xml A SED-ML file for running the Level 1 Version 2 case. Image:icon-checked.gif
NNNNN-sbml-l2v1-sedml.xml A SED-ML file for running the Level 2 Version 1 case. Image:icon-checked.gif
NNNNN-sbml-l2v2-sedml.xml A SED-ML file for running the Level 2 Version 2 case. Image:icon-checked.gif
NNNNN-sbml-l2v3-sedml.xml A SED-ML file for running the Level 2 Version 3 case. Image:icon-checked.gif
NNNNN-sbml-l2v4-sedml.xml A SED-ML file for running the Level 2 Version 4 case. Image:icon-checked.gif
NNNNN-sbml-l3v1-sedml.xml A SED-ML file for running the Level 3 Version 1 case. Image:icon-checked.gif
NNNNN-results.csv Reference results for the model. Image:icon-unchecked.gif
NNNNN-results.xlsx (Some models only.) The results of using an analytical function in Microsoft Excel to produce the results for the model. Used to produce the NNNNN-results.csv file for those models. Image:icon-unchecked.gif
NNNNN-antimony.txt (Some models only.) A description of the test model in Antimony format used to generate the SBML file. Image:icon-unchecked.gif
NNNNN-plot.jpg A plot of the simulation results in JPG format. Used for displaying to humans as part of the case explanations. Image:icon-checked.gif

The model description file

To give an idea of what is contained in a model description, here is an example file. The description portion is at the beginning of the file, enclosed in Mathematica comment delimiters (the (* and *) character sequences). The rest of this particular case file contains commands for a program written in Mathematica to generate the results; not all Test Suite cases have those commands because other cases were created using other software tools.

(*

category:        Test
synopsis:        Basic single forward reaction with two species in one compartment
componentTags:   Compartment, Species, Reaction, Parameter 
testTags:        InitialAmount
testType:        TimeCourse
levels:          1.2, 2.1, 2.2, 2.3, 2.4, 2.5, 3.1
generatedBy:     Analytic
packagesPresent:

The model contains one compartment called "compartment". There are two species called S1 and S2
and one parameter called k1. The model contains one reaction:

[{width: 30em, margin: 1em auto}|  *Reaction*  |  *Rate*  |
| S1 -> S2 | $k1 * S1 * compartment$ |]

The model does not contain any rules.

The initial conditions are as follows:

[{width: 30em, margin: 1em auto}|    |*Value*          |*Units*        |
|Initial amount of S1                |$1.5 \x 10^-15$  |mole           |
|Initial amount of S2                |$0$              |mole           |
|Value of parameter k1               |$1$              |second^-1^     |
|Volume of compartment "compartment" |$1$              |litre          |]

The species' initial quantities are given in terms of substance units to make it easier to use
the model in a discrete stochastic simulator, but (as per usual SBML principles) their 
symbols represent their values in concentration units where they appear in expressions.

Note: The test data for this model was generated from an analytical solution of the 
system of equations.

*)

newcase[ "0001" ];

addCompartment[ compartment, size -> 1 ];
addSpecies[ S1, initialAmount -> 1.5 10^-15 ];
addSpecies[ S2, initialAmount -> 0 ];
addParameter[ k1, value -> 1 ];
addReaction[ S1 -> S2, reversible -> False,
	     kineticLaw -> k1*S1*compartment ];

makemodel[]

The lines at the beginning, enclosed in comment character sequences (* and *), are used by the Test Suite software to summarize and categorize the test cases. The text that follows is text in Wiky format. It describes the model and can be used to automatically generate the HTML description files.

Here's an explanation of the lines that head each test case:

Field Explanation
category: Many of the models used in the test suite are not biologically meaningful. This field determines whether this case refers to a test model or a more realistic model. In the current (mid-2011) version of the test suite, the only value present for this tag is Test.
synopsis: Brief textual description of the model, in English. The value of this field may span over more than one line.
componentTags: These tags describe the SBML components that are present in this model. The tags are described on separate page.
testTags: These tags describe the aspects of SBML interpretation that are being tested in this model. The tags are described on separate page.
testType: Since it is possible to simulate data from models in different ways, this tag indicates the type of test to perform on the given model. Currently (2013), there are two tags: TimeCourse, for the majority of tests in the suite, and FluxBalanceSteadyState, for tests that involve the 'Flux Balance Analysis' package.
levels: Not all SBML components and attributes exist in every level and version of SBML. This tag indicates the relevant SBML Levels+Version combinations permissible for this particular case. The format of the tag is two integers separated by a dot; e.g., 2.4 signifies Level 2 Version 4.
generatedBy: This tag indicates whether the results data for this case has been generated analytically (tag value Analytic) or numerically (tag value Numeric).
packagesPresent: This tag indicates whether any SBML Level 3 packages are present in the model. Possible tag values are (at present) comp, if elements from the Hierarchical Model Composition package are present, and fbc, if elements from the Flux Balance Constraints package are present.

The "settings" file

The information about run parameters is stored in a file named "NNNNN-settings.txt". This file is generated automatically after a test case is created by an author. The generation is accomplished using a script that reads the CSV file and summarizes the actual test run; this approach reduces the chances of human errors introducing a mismatch between the claimed simulation settings and the actual settings used to generate the reference data.

The format of the file is very simple. Here's an example:

start: 0
duration: 5.0
steps: 50
variables: S1, S2
absolute: 1e-12
relative: 0.0001
amount: S1, S2
concentration:

Depending on the type of test requested (TimeCourse or FluxBalanceSteadyState) this data will have slightly different meanings

The "settings" file for TimeCourse tests

The information about run parameters is stored in a file named "NNNNN-settings.txt". This file is generated automatically after a test case is created by an author. The generation is accomplished using a script that reads the CSV file and summarizes the actual test run; this approach reduces the chances of human errors introducing a mismatch between the claimed simulation settings and the actual settings used to generate the reference data.

The format of the file is very simple. Here's an example:

start: 0
duration: 5.0
steps: 50
variables: S1, S2
absolute: 1e-12
relative: 0.0001
amount: S1, S2
concentration:

Here's an explanation of each line in turn.

Field Explanation
start: The start time of the simulation time-series data in the output (CSV) file. Often this is 0, but not necessarily.
duration: The duration of the simulation run, in the model's units of time.
steps: The number of steps at which the output is sampled. The samples are evenly spaced. When a simulation system calculates the data points to record, it will typically divide the duration by the number of time steps. Thus, for X steps, the data file will have X+1 data rows.
variables: The variables (in addition to time) whose values are tabulated in the CSV file. These are SBML model id's separated by commas. Order is significant, as results files without headers will be assumed to be output in the order present on this line. Important: if a symbol in this list refers to a species in the model, then that symbol will also be listed in either the amount or concentration lists below.

NOTE:If a listed variable has two underscores in it ('__'), that variable is actually present only in a submodel of the main model, from the Hierarchical Model Composition package, in the format submodelID__variableID. If the model is flattened, the variable will appear automatically.
absolute: A float-point number representing the absolute difference permitted for this test case when evaluating a software tool. The formula used to calculate data point differences is discussed below.
relative: A float-point number representing the relative difference permitted for this test case when evaluating a software tool. The value of 0.0001 was the tolerance agreed upon by the SBML community during discussions at SBML Hackathons in 2008. The formula used to calculate data point differences is discussed below.
amount: A list of the variable whose output in the results file is in amount (not concentration) units. This list of variables must be a subset of the names listed in variables.
concentration: A list of the variable whose output in the results file is in concentration (not amount) units. This list of variables must be a subset of the names listed in variables.

Tolerances and errors for TimeCourse tests

The absolute and relative tolerances are used in the following way. Let

Ta stand for the absolute tolerance for this test case,
Tr stand for the relative tolerance for this test case,
Cij stand for the expected correct value for row i, column j, of the result data set, and
Uij stand for the the user's uploaded result value.

Then a data point Uij is considered to be within tolerances if and only if

|Cij − Uij|( Ta + Tr × |Cij| ).

The "settings" file for FluxBalanceSteadyState tests

For tests with the 'testType' tag FluxBalanceSteadyState, the settings file is the same format as for TimeCourse tests, but more lines are left blank, as they have no meaning. Here's an example:

start:
duration:
steps:
variables: J0, J1, OBJF
absolute: 0.001
relative: 0.001
amount:
concentration:

The variables line still indicates which variable values are to be output after steady state is reached, and the relative and absolute lines indicate the relative and absolute error allowed for each data point.

The other lines are irrelevant to a flux balance steady state simulation: because the system is being analyzed at steady state, there is no start, duration, or steps to be taken. Because only fluxes and objective functions are being analyzed, no species may be requested as output (and therefore, there is no ambiguity with regards to concentration vs. amount).

Results data format

The expected (or reference) results stored in the file NNNNN-results.csv are simply organized as a table of values. The values will be slightly different depending on whether a TimeCourse or FluxBalanceSteadyState test was requested.

Results data format for TimeCourse tests

The expected (or reference) results stored in the file NNNNN-results.csv are simply organized as a table of values. The first column is simulation time, and the remaining columns are variables in the model (often species, but not necessarily—they could be compartments, parameters, or reaction rates too) in the same order as they are listed in the variables: line in the NNNNN-settings.txt file. An optional header line is permitted at the top of the file. Here is a short example:

time,S1,S2
0,1.5e-15,0
0.1,1.357256127053693e-15,1.427438729463058e-16
0.2,1.228096128602129e-15,2.719038713978695e-16
0.3,1.111227330205311e-15,3.887726697946884e-16
0.4,1.005480064513687e-15,4.945199354863119e-16
0.5,1.005480064513687e-15,4.945199354863119e-16

The first line of the file lists the columns, and the rest are numerical data. The total number of lines of data in the file is X + 1, where X is the value of the steps: line in the NNNNN-settings.txt file.

It is possible for some values to be Not a Number (indicating the result is not mathematically defined, such as attempting to divide by zero or perform an operation involving infinity). It is also possible for values to be +∞ or -∞. There does not appear to be an agreed-upon standard way of expressing these values in comma-separated files, so the SBML Test Suite uses the following convention:

Value Symbol used
NaN NaN
+∞ INF
-∞ -INF


These symbols are treated in a case-insensitive manner by the SBML Test Suite.

The results data format for FluxBalanceSteadyState tests

As for the TimeCourse results, FluxBalanceSteadyState test results are stored in the file NNNNN-results.csv, and organized as a table of values. The first, header line indicates the expected variables, and the second line indicates the value of those variables at steady state:

R01,R26,R10,R07,OBJF
1.0,1.0,0.5,0.5,1.0

As for the TimeCourse results, it is also possible for FluxBalanceSteadyState results to contain Not a Number, +∞ and -∞, reported as above.

Retrieved from "http://sbml.org/Software/SBML_Test_Suite/Case_Descriptions"

This page was last modified 22:17, 7 January 2017.



Please use our issue tracking system for any questions or suggestions about this website. This page was last modified 22:17, 7 January 2017.