Step 2: Running the tests
Testing a software package using the SBML Test Suite requires simulating specially-designed SBML models using the package, and then comparing the simulation output to expected results.
The definition of a test case
Each test model in the SBML Test Suite is organized in a folder named with a five digit number (i.e., "NNNNN"). In the case of the archive produced by the Online SBML Test Suite for you to download, the folder contains the following files:
|
| (Only one file will be present.) These are the SBML files defining the model that constitutes a particular test. All the files define the same model, but in different SBML Level/Version formats. The downloadable archive produced by the Online Test Suite will contain just one of these files, for the chosen Level/Version combination, with the additional consideration that some tests apply to features that do not exist in all SBML Levels or Versions and therefore not all test cases come in all formats. |
NNNNN-settings.txt
| This is a text file listing the general simulation control values that should be used when running the given model. The format of this file is described on a separate page. |
NNNNN-model.html
| A brief, nicely-formatted, human-readable description of the purpose of this specific test. |
NNNNN-plot.jpg
| A plot of the expected time-course simulation results in JPEG format. |
|
| (Only one file will be present.) A file in SED-ML format for running the test case in software systems that can automate their execution using SED-ML. |
In addition to the above, the standalone SBML Test Suite and the full archive of test cases available from SourceForge.net contains the following extra files:
NNNNN-results.csv
| The results expected from simulating the model. The file format is simple comma-separated values. Approximately 1/5 of SBML Test Suite cases have results generated from an analytical solution to the model; the rest come from numerical solutions produced agreed-upon by at least two different SBML-compatible tools. |
NNNNN-model.m
| A description of the test model in a format used to generate the HTML file. (In some models, it contains commands used to generate the SBML file as well, but this is not available for all test case files.) |
NNNNN-plot.svg
| A plot of the expected time-course simulation results in SVG format. |
NNNNN-antimony.txt
| (Some models only.) A description of the test model in Antimony format used to generate the SBML file. |
NNNNN-results.xlsx
| (Some models only.) The results of using an analytical function in Microsoft Excel to produce the results for the model. This is in turn used to produce the NNNNN-results.csv file for those models.
|
It is worth noting that the standalone SBML Test Suite, and the full test case archive available as a separate download, both contain every Level/Version combination possible (that is, the NNNNN-sbml-lXvY.xml files mentioned above) for every test case, rather than just a specific Level/Version combination. (The Online SBML Test Suite provides a selection system that lets users create and download a subset of the test cases with desired features.)
Running a test case
Running a single test in an SBML-compatible software package requires performing the following steps:
- Examine
NNNNN-settings.txtto determine the simulation start time, duration, tolerances, number of output sampling steps, and variables to be sampled/plotted. (The format of this file is described on a separate page.) - Instruct the software you are testing to read the SBML model definition file for this case (
NNNNN-sbml-lXvY.xml, chosing LevelXand VersionYas appropriate for the software). - Instruct the software to run a simulation of the model (using the settings determined in step #1) and save the output to a file whose name contains the case number (e.g.,
myresultsNNNNN.csvor even justNNNNN.csv). - (If necessary) Convert the output to comma-separated value format.
- Use the SBML Test Suite to compare the output to the expected results and analyze the implications.
This process must be repeated for every test case in the suite, i.e., for every folder NNNNN provided in the SBML Test Suite collection of test cases. How this is accomplished depends very much on the particular software package being tested and which version of the SBML Test Suite you are using. The standalone SBML Test Suite provides an automation framework which takes care of looping over all the NNNNN folders. It only needs to be given a shell script with which to control the software being tested. Conversely, the online SBML Test Suite does not provide a loop/control facility; it leaves the task of running the tests to you, and provides the ability to upload the results to have them evaluated using the online system.
Gathering the results of many tests for uploading to the Online SBML Test Suite
The format of the results accepted by the online system is very simple. As mentioned above, the result of running each test model must be stored in a comma-separated value file, such that the file name contains the case number (e.g., myresultsNNNNN.csv). Simply create a zip archive containing all of the output files:
-
myresults00001.csv -
myresults00002.csv -
myresults00003.csv -
myresults00004.csv - …
-
myresultsNNNNN.csv
Gaps are allowed in the files put in the archive (for example, if you chose not to run some tests, or the software cannot simulate some test models). The final number (NNNNN) depends on the number of tests provided in the current release of the SBML Test Suite, as well as the tests you selected to run, and the capabilities of the software you are testing.
After creating the archive, go to Step 3 (uploading the results).