SBML.org — the global portal for all things SBML

Step 2: Running the tests

Testing a software package using the SBML Test Suite requires simulating specially-designed SBML models using the package, and then comparing the simulation output to expected results.

The definition of a test case

Each test model in the SBML Test Suite is organized in a folder named with a five digit number (i.e., "NNNNN"). The folder contains the following files:

NNNNN-sbml-l3v1.xmlSBML Level 3 Version 1
NNNNN-sbml-l2v5.xmlSBML Level 2 Version 5
NNNNN-sbml-l2v4.xmlSBML Level 2 Version 4
NNNNN-sbml-l2v3.xmlSBML Level 2 Version 3
NNNNN-sbml-l2v2.xmlSBML Level 2 Version 2
NNNNN-sbml-l2v1.xmlSBML Level 2 Version 1
NNNNN-sbml-l1v2.xmlSBML Level 1 Version 2

These are the SBML files defining the model that constitutes a particular test. All the files define the same model, but in different SBML Level/Version formats. Note that some tests apply to features that do not exist in all SBML Levels or Versions and therefore not all test cases come in all formats.
NNNNN-settings.txt This is a text file listing the general simulation control values that should be used when running the given model. The format of this file is described on a separate page.
NNNNN-model.html A brief, nicely-formatted, human-readable description of the purpose of this specific test.
NNNNN-model.m A description of the test model in a format used to generate the HTML file. (In some models, it contains commands used to generate the SBML file as well, but this is not available for all test case files.) The format of this file is described on a separate page.
NNNNN-plot.jpg A plot of the expected time-course simulation results in JPEG format.
NNNNN-results.csv The results expected from simulating the model. The file format is simple comma-separated values. Approximately 1/5 of SBML Test Suite cases have results generated from an analytical solution to the model; the rest come from numerical solutions produced agreed-upon by at least two different SBML-compatible tools.

NNNNN-sbml-l3v1-sedml.xml
NNNNN-sbml-l2v5-sedml.xml
NNNNN-sbml-l2v4-sedml.xml
NNNNN-sbml-l2v3-sedml.xml
NNNNN-sbml-l2v2-sedml.xml
NNNNN-sbml-l2v1-sedml.xml
NNNNN-sbml-l1v2-sedml.xml

(Only one file will be present.) A file in SED-ML format for running the test case in software systems that can automate their execution using SED-ML.
NNNNN-antimony.txt (Some models only.) A description of the test model in Antimony format used to generate the SBML file.
NNNNN-results.xlsx (Some models only.) The results of using an analytical function in Microsoft Excel to produce the results for the model. This is in turn used to produce the NNNNN-results.csv file for those models.

It is worth noting that the standalone SBML Test Suite, and the full test case archive available as a separate download, both contain every Level/Version combination possible (that is, the NNNNN-sbml-lXvY.xml files mentioned above) for every test case, rather than just a specific Level/Version combination.

Running a single test case

Running a single test in an SBML-compatible software package requires performing the following steps:

  1. Examine NNNNN-model.m to determine what kind of test is being requested. The kind will be expressed by the value of the field named testType: in the file. (The format of this file is described on a separate page.)
  2. Depending on what kind of test is being requested, do as follows:
    1. If the type of test is TimeCourse:
      1. Examine the file NNNNN-settings.txt to determine the simulation start time, duration, tolerances, number of output sampling steps, and variables to be sampled/plotted. (The format of this file is described on a separate page.)
      2. Instruct the software you are testing to read the SBML model definition file for this case (i.e., the file NNNNN-sbml-lXvY.xml, chosing Level X and Version Y as appropriate for the software). Note: if constructs from the SBML Level 3 Hierarchical Model Composition ('comp') package are present, the model may need to be "flattened" first, depending on how the simulation software handles the 'comp' package.
      3. Instruct the software to run a simulation of the model (using the settings determined by reading the file NNNNN-settings.txt as described above) and save the output to a file whose name contains the case number (e.g., myresultsNNNNN.csv or even just NNNNN.csv).
      4. (If necessary) Convert the output to comma-separated value format.
      5. Use the SBML Test Suite to compare the output to the expected results and analyze the implications.
    2. If the type of test is FluxBalanceSteadyState:
      1. Examine NNNNN-settings.txt to determine the variables to be sampled/plotted. (The format of this file is described on a separate page.)
      2. Instruct the software you are testing to read the SBML model definition file for this case (i.e., specifically, the file NNNNN-sbml-l3v1.xml — all flux balance tests are SBML Level 3, because the test requires the use of the SBML Level 3 Flux Balance Constraints package).
      3. Instruct the software to run a simulation of the model to steady state, following the restrictions given by the Flux Balance Constraints package components in the model, and save the final values of the requested variables to a file whose name contains the case number (e.g., myresultsNNNNN.csv or even just NNNNN.csv).
      4. (If necessary) Convert the output to comma-separated value format.
      5. Use the SBML Test Suite to compare the output to the expected results and analyze the implications.

Running multiple test cases

To run more than one test case, the process above must be repeated for every test case in the suite that your software can handle. How this is accomplished depends very much on the particular software package being tested and which version of the SBML Test Suite you are using. The standalone SBML Test Suite provides an automation framework which takes care of looping over all the NNNNN folders, and provides methods to include or exclude tests by tag or by package. It only needs to be given a shell script with which to control the software being tested.

Many software packages may not be designed to pass all the tests in the suite: some particular test tag or component may not be supported, or the software might not even read some package information at all. In those cases, the unsupported tests can be excluded by tag or by package, and marked as 'unsupported' instead of 'failed'.


Retrieved from "http://sbml.org/Software/SBML_Test_Suite/Step_2:_Running_the_tests"

This page was last modified 22:28, 7 January 2017.



Please use our issue tracking system for any questions or suggestions about this website. This page was last modified 22:28, 7 January 2017.