Step 2: Running the tests

Testing a software package using the SBML Test Suite requires simulating specially-designed SBML models using the package, and then comparing the simulation output to expected results.

The definition of a test case

Each test model in the SBML Test Suite is organized in a folder named with a five digit number (i.e., "NNNNN"). In the case of the archive produced by the Online SBML Test Suite for you to download, the folder contains the following files:

In addition to the above, the standalone SBML Test Suite and the full archive of test cases available from SourceForge.net contains the following extra files:

The standalone SBML Test Suite and the full archive also contain every Level/Version combination possible (that is, the NNNNN-sbml-lXvY.xml files mentioned above) for every test case.

Running a test case

Running a single test in an SBML-compatible software package requires performing the following steps:

1. Examine NNNNN-settings.txt to determine the simulation start time, duration, tolerances, number of output sampling steps, and variables to be sampled/plotted. (The format of this file is described on a separate page.)
2. Instruct the software you are testing to read the SBML model definition file for this case (NNNNN-sbml-lXvY.xml, chosing Level X and Version Y as appropriate for the software).
3. Instruct the software to run a simulation of the model (using the settings determined in step #1) and save the output to a file whose name contains the case number (e.g., myresultsNNNNN.csv).
4. (If necessary) Convert the output to comma-separated value format.
5. Use the SBML Test Suite to compare the output to the expected results and analyze the implications.

This process must be repeated for every test case in the suite, i.e., for every folder NNNNN provided in the SBML Test Suite collection of test cases. How this is accomplished depends very much on the particular software package being tested and which version of the SBML Test Suite you are using. The standalone SBML Test Suite provides an automation framework which takes care of looping over all the NNNNN folders. It only needs to be given a shell script with which to control the software being tested. Conversely, the online SBML Test Suite does not provide a loop/control facility; it leaves the task of running the tests to you, and provides the ability to upload the results to have them evaluated using the online system.

Gathering the results of many tests for uploading to the Online SBML Test Suite

The format of the results accepted by the online system is very simple. As mentioned above, the result of running each test model must be stored in a comma-separated value file whose name contains the case number (e.g., myresultsNNNNN.csv or even just NNNNN.csv). Simply create a zip archive containing all of the output files:

myresults00001.csv

myresults00002.csv

myresults00003.csv

myresults00004.csv

…

myresultsNNNNN.csv

Gaps are allowed in the files put in the archive (for example, if you chose not to run some tests, or the software cannot simulate some test models). The final number (NNNNN) depends on the number of tests provided in the current release of the SBML Test Suite, as well as the tests you selected to run, and the capabilities of the software you are testing.