SBML Editors' process for handling issue reports

Procedure version of 2011-3-13

Anyone (whether they are a member of the SBML Forum or not) may raise an issue or record a request using the reporting system at http://sbml.org/issue-tracker. The tracker is hosted on SourceForge.net as part of the SBML project. As much as possible, the SBML Editors also use the tracker to log issues and discussions about those issues.

Sequence

The following is the intended sequence of events in the life of an issue.

1. Preliminary actions:
   1. The issue is entered into the tracking system by someone. The SourceForge tracking system defaults new issues to a status of Open. The SBML Editors mailing list will be automatically notified of the new issue.
   2. The first available person from the SBML Editors to examine the report will do a preliminary evaluation of whether to proceed further. If the issue is genuine, the Editor will change the tracker "Group" to Reported/Proposed. If it is not (perhaps because it is spam), the Editor will change the issue's status to Deleted and (if warranted) notify the original submitter to explain their rationale for removing the issue.
   3. Someone from the SBML Editors (possibly the same person as in the previous step) will choose to adopt the issue and shepherd it through the process. They will change the issue's Assigned To field to their name, and begin to lead the other SBML Editors in analyzing and discussing the issue.
2. Evaluation and discussion:
   1. The Editors will begin by indicating whether they accept the issue as valid or not by using one of the canned responses for this purpose in the tracking system. The canned responses are shown in the screenshot below. The relevant choices for this step are Accept as valid issue and Do not accept as valid issue. This step often involves discussion and exploration of various issues. By this process, the Editors first determine whether the issue is considered a new and genuine issue, or whether it is not (perhaps because it turns out to be effectively identical to another issue).
   2. The Assigned-to Editor monitors the issue until a 4 out of 5 majority of Editors either agree it is valid, or agree it is not valid. In the case where the issue is considered genuine and in need of action, the Editor will change the status of the issue to Pending (still leaving the "Group" as Reported/Proposed). If the Editors decide not to accept the issue, the Assigned-to Editor will change the status to Deleted, and notify the original submitter to explain their rationale for removing the issue.
      • Exception: simple matters such as typographical or formatting errors do not require a majority; they can be acted upon by a single Editor and the issue closed by that Editor after they made the relevant change(s). If any other Editors disagree with the action, they can reopen the issue and call for the issue to follow the process defined for Changes with conformance implications or Changes without conformance implications. (Updated 2011-03-13: previously minor issues required a 3/5 majority; this was changed after acceptance of tracker issue #2949518.)
   3. The Assigned-to Editor will continue to lead the SBML Editors in discussing the issue and how to resolve it. Generally this takes place by attaching comments in the tracking system, but sometimes (especially when there is a lot of uncertainty or there needs to be exploration of wider implications), the discussions may take place on the SBML Editors' internal mailing list. In the latter case, it is the Assigned-to Editor's responsibility to try to summarize or otherwise enter back into the tracking system as much relevant information as possible, so that the issue history reflects the reasoning and rationales for whatever decisions are ultimately made.
   4. Editors can indicate at any time their acceptance of a proposed action or rejection using the canned responses facility in the tracker. The two relevant options are Agree with the proposed change and Agree that the action should not be taken.
3. Resolution:
   1. In time, a consensus will emerge with 4 out of 5 editors agreeing one way or the other. If 4 out of 5 Editors agree that a change in the SBML specification is needed, the Assigned-to Editor will change the issue's "Group" to one of the Accepted groups. There are three versions:
      • Accepted: Formatting/non-content editing: These are formatting or other issues that do no fundamentally change any textual content of the specification document.
      • Accepted: Changes without conformance implications: These are issues that change the text, but do not affect SBML software in a way that may alter their conformance with the SBML specification. For example, a change to some terminology in SBML would not fundamentally change the mathematics of a model, and therefore would not be considered a change having conformance implications.
      • Accepted: Changes with conformance implications: These are changes that do affect the conformance of software reading, writing and/or processing data in SBML format.
   2. On the other hand, if 4 out of 5 Editors agree the issue should not be handled by a change to the SBML specification after all, then Assigned-to Editor will change the issue's "Group" to Rejected/No action and change its "Status" to Closed.
   3. Eventually, all Pending issues that have been Accepted in one form or another are handled by producing a new release of the affected SBML Level+Version specification. When an issue has been handled in this way, its "Status" is changed from Pending to Closed.
4. Post-resolution:
   1. Every accepted issue should be listed on the errata page for the specification involved. (An example of such a page is the SBML Level 2 Version 3 errata page.) All categories of issues (formatting/non-content editing, changes without conformance implications, and changes with conformance implications) should be listed in the errata, and each entry should provide a link to the tracker entry for that issue.
   2. Errata in a specification are flaws that should be corrected. The SBML Editors must weigh the benefits of producing a new release of an SBML specification (or a new version, or even a new level, if warranted) against the costs of doing so. The costs that must be considered include not only the effort required to produce a new specification document, but also the impact on the wider SBML community. The evaluation and decision is left up to the SBML Editors.

Additional notes

There is a natural tendency for issues to arise in discussions over the SBML Editors' internal mailing list. The Editors need to be proactive and recapitulate any discussions that lead to issues in the tracking system. It is vital to document as much as possible of the background of an issue with the issue's record so that some years in the future we can remember what lead to important decisions in SBML's development.