Early design of version 2.0 of the SBML Test Suite
Note to readers: this was the top-level planning/design/discussion page for the initial SBML Test Suite project. It superceded the original SBML Semantic Test Suite. The design and implementation of the SBML Test Suite has evolved since this page was written, but we leave it here for historical interest.
The purpose of the SBML Test Suite is to provide a conformance testing suite for SBML. The basis of the suite will be a large number (certainly 100's, maybe 1000's) of test models and corresponding expected simulation results. Someone would take the models, run them in a simulation software package, compare the results to the expected results, and use the tool's behavior across the suite of test cases to assess whether the software handles SBML properly. Running and comparing so many test cases by hand would tedious at best, and that's where the Test Suite supporting software comes in.
Design and features
Software/SBML Test Suite/Design Sketches is a page of early design ideas for the interface and general structure. This is currently (2008-02-13) out of date with the most recent discussions and ideas.
Software/SBML Test Suite/Case Descriptions provides an explanation of the parts expected for each test case.
Software/SBML Test Suite/Evaluation describes how individual test simulation results will be compared to the reference values.
Additional background material
Video of discussions held in Okinawa, Japan, during the 2008 Super-Hackathon. Mike describes his view of the new Test Suite design. The recording was unfortunately started late, so the beginning of the discussion is missing, but what's there is still useful for more context.
The original grant proposal's technical body is worth reading for an even more detailed description of many of the features desired in this thing. The proposal didn't have mock-ups and didn't articulate a number of the new ideas, such as the configuration feature that permits users to define the interface to different software packages right from the test runner (as opposed to needing a wrapper shell script).
Andrew Finney's original test suite and an unreleased improved version made by Sarah Keating some time ago provides useful background on the first generation. There are major differences in this old design versus the new design sketched out here, and all the tests have to be redone anyway, but the old test suite provides a tremendous jumping point for this second-generation SBML Test Suite.