The IEEE Standards on Software Engineering include IEEE Std 1012-1998, which addresses this issue and could be helpful here. The solution to the problems cited by Professor King is simply an available, Open Source, body of test programs which can be run in an automated fashion with each revision of the total system. It would be foolhardy in the extreme to put any untested system into production; with automated testing, it is hardly a burden to rerun existing automated tests--Open Source Testing means that the test code can be supplemented by diverse contributions in the same way Open Source code is developed. Private testing organizations can add hidden (albeit real) value but they can also be required to run Open Source Tests as part of their service.
Most election admistrators would be well advised to test their software with extensive automated tests, open to all voters, open to technical poll watchers, and supplemental to the certification of private test organizations. Re-running these tests the week before an election would be good if any changes have been made to any part of the system. Re-running them the day before the election and then locking down the system would also be a good idea. Re-certification after minor operating system change is reasonably defined as the same test result from an extensive and comprehensive body of automated tests repeated on a newer version of the operating system. Monkey testing is neither efficient nor necessary, I believe, for re-certification, and almost any set of automated test programs can be run on voting software overnight.
Let's solve the problem and not get caught up in the impossible "error-free software" wild goose chase. The real questions are, how does it perform under test and has the test provided sufficient risk of failure to assure confidence that the behavior in use will reflect the behavior under test.
This is my personal and professional opinion, and it does not necessarily reflect the opinions of my employer.
Richard C. Johnson, Ph.D.
Merle King wrote:
Stan, Vince, et al - The question that I intended to ask during the teleconference of Nov. 30... Under this proposed scope, how will warranty work on the equipment be addressed? If each time a defective driver or other COTS component is identified and a warranty recall is issued (example: recent recall by Dell for a defective driver for their RAID configuration used on central tabulation machines), how will this impact requalification of a system? The owners of the systems (jurisdictions) will want to take advantage of warranty work while still covered by the warranty. If the warranty work requires the entire system to be requalified...that could be impractical, given the time and expense involved, as well as the potential quantity of warranty claims over the life of the unit. - Merle Merle S. King http://science.kennesaw.edu/csis Chair, CSIS Department Kennesaw State University 1000 Chastain Road, MB #1101 Kennesaw, GA 30144-5591 voice: 770-423-6354; fax: 770-423-6731"Stanley A. Klein" <sklein@CPCUG.ORG> 11/22/2004 10:37:40 AM >>>When P1583 was last balloted, there were numerous comments to the effect that COTS should not be exempt from inspection. An STG was set up to resolve those and other comments regarding COTS. The resolution generally was that COTS should undergo the exact same testing as any other parts of a voting system, unless it had been previously tested and certified as satisfying the requirements of the specification. It seems to me that the proper procedure is to incorporate the resolutions into the P1583 draft and move forward with the document. We are working to replace an FEC standard that is widely agreed to be inadequate. The recent election resulted in numerous reports of problems with voting systems that can easily be attributed to the inadequacy of both the FEC standards and the ITA certification process. The COTS exemption is a major, glaring loophole in the FEC standard. Given the technical alternatives for vendors and the procedures used in other mission critical systems, there is absolutely no reason for allowing the COTS exemption to remain. Of course it will be initially inconvenient for vendors, but any meaningful upgrade of the FEC standard will involve some initial inconvenience for vendors. >From my perspective, I see the leadership of P1583 slipping back into the kinds of process games that just a month or two ago they agreed to stop playing. Their credibility is now on the line, and by implication the credibility of IEEE Standards as it relates to an issue with major national attention. Stan Klein On Mon, 2004-11-22 at 08:31, R. Mercuri wrote:Herb -- I would like to add my voice to Vince's with regard to the attempted replacement of our sub-group's work by material submitted by Sandy, who isn't even a member of P1583. We were offended at the meeting when it was suggested that Sandy replace us as the co-chairs of the working group, and that materials be fed to him, rather than us. We had asked NUMEROUS TIMES for input, so if that is going to be provided, it should be to Vince and myself, not Sandy. Sandy is welcome to make suggestions, of course, but not as a co-chair of the COTS-STG. We had also provided material from our sub-committee for the draft document that was not included, despite REPEATED requests to do so, and this situation continues to be remiss. Now it seems that the entire standard will be compromised through COTS exemptions. Sandy's comment "code review of COTS software cannot ensure that the COTS firmware/software is failsafe" could be equally applied to the entire voting product -- why bother to have code review at all??? BECAUSE white box AND black box testing are NECESSARY to reveal different sorts of problems, such as VULNERABILITIES. Additionally, the code MUST be available in order to provide assurances and evidence if later problems arise. There are certainly ways in which vendors can contract with suppliers to provide their code -- the choices they make can/should include products that already are open to review (such as Linux or BSD-based). The bottom line is that there's no point in having a voting system standard if parts of the ballot casting and tabulation system are going to be exempt. It is bad enough that the standard does not require any auditability feature to provide independent recounts -- but to further allow non-inspected components, just makes the whole thing a joke. I seriously hope that this issue can be resolved appropriately, Rebecca Mercuri.-- Stanley A. Klein <email@example.com>