Software quality assurance is a hard problem. Most commercial software developers handle it as follows: Engineers are asked to keep the code clean and bug-free. They are then told to get it out on time. The code is then passed to overworked and under-appreciated teams of "testers". These folks create, maintain, and run suites of regression tests.
If a tester finds a bug well in advance of the shipping deadline, the bug will probably be fixed. As the deadline approaches, the bug's severity will be questioned. In the last few weeks, only "show stopping" bugs are fixed. All other bugs are relegated to consideration in following releases.
Once the product is shipped, the quality assurance focus shifts to the users and the support staff. As users find problems, they contact the support staff, asking for help. Often, the problems can be traced to user error. Remaining problems are logged as bugs and sent to the appropriate group (engineering, publications, etc.) within the organization.
In the freeware arena, the situation is similar, if less formal. Email is used for questions and bug reporting. The freeware author generally assumes all of the roles described above, and may or may not be highly motivated to handle particular requests. In any event, bug tracking and regression testing are likely to be done on an ad hoc basis.
Although there are some commercial packages for bug tracking and regression testing, most companies write their own. The reasons for this may have to do with local conditions (assumed or real), deficiencies in the commercial offerings, or cost factors. Regardless, the result is that a number of programmers are currently busy writing and/or maintaining private versions of these software systems. What's worse, these systems aren't really solving the problems at hand.
The only formal regression testing is done by the testers. The users don't have the regression test suite, don't have any standardized way to pass back results, and haven't been brought into the process. Users' bug reports tend to be difficult for engineers and testers to use. As a result, they are all too frequently ignored.
The bug tracking mechanism is also part of the problem. If the bug has to be transcribed from a telephone call or facsimile, it may get lost entirely, or end up in a form that cannot be verified. A good bug tracking mechanism should accept emailed bug reports, respond automatically with acknowledgements, handle follow-up inquiries and reports, etc. Most systems I've seen have nowhere near this much capability.
Cygnus Support found the same situation when they looked. Fortunately, they had the resources and motivation to solve it. Over the past few years, they have developed two complementary systems. Having developed them, they are now releasing them as freeware. The systems are admittedly imperfect, but they are not toys, and they will improve rapidly as their users complain about and/or improve upon them.
Deja GNU is a regression testing framework. It provides a standardized way of setting up and running regression tests. It can be used by a company's internal testers, but that's not all. It can also be distributed with the software itself:
Hi, here's our new software release. We've tested it carefully, but your site may have an unusual configuration. We therefore suggest that you run this test suite before placing this software into production use. If you find any problems, please let us know at once, so we can start working on them.
PRMS (Problem Report Management System) is an automated bug report routing and tracking system. It receives, tracks, files, and journals problem reports using email. (It also supports interactive inquiries and updates.) Using PRMS, the users and support staff can work together on bugs. Users can include the results of relevant regression tests in their reports, or even submit new tests for the regression suite.
PRMS and Deja GNU could become the basis for a whole new way of handling software quality assurance. Commercial vendors could bring their customers into the QA loop as full participants. The companies would benefit from better bug-reporting and happier customers. The customers would benefit from more reliable software and more responsive support.
As more companies and freeware projects use these tools, they will improve. The effort spent in maintaining them will be spread across all the users, and the benefits will be available for anyone who wants them. Sounds like a good idea to me!
Both of these packages can be found on any reasonable GNU server. Look in ftp://prep.ai.mit.edu/pub/gnu/ if you have no server that is closer to you. PRMS (known as GNATS in the GNU world) is stored as gnats-*.tar.gz, and takes up about 472 KB. Deja GNU is stored as dejagnu-*.tar.gz, and takes up about 529 KB.
Please note that all GNU software is currently distributed in GNU Zip format. GNU Zip yields much smaller files than compress, decompresses twice as fast, and does not appear to be covered by any annoying patents. Before you can unpack GNU Zip files, however, you will need to retrieve and build gzip-*.tar.
Oh yes; support. Both PRMS and Deja GNU are supported by Cygnus Support (+1 415 903 1400; email@example.com). Cygnus will fix (or at least consider) any reported bugs, and is happy to receive comments and suggestions. If, however, you want hand-holding, specific improvements and/or customizations, or other custom support, you can get these, too, for a fee. Cygnus makes its money from supporting free software, not from selling it. (Sounds like another good idea to me; pass it on...)