[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: COTS scope

On Fri, 2004-12-03 at 09:53, Dick Johnson wrote:
> I quite agree, Stan, with your compromise distinction between stringent 
> specification (no AVVPAT reqd to meet standards) and relaxed 
> specification.  That seems fair. 
> (1) But you don't really look carefully enough at the differences 
> between hardware and software concepts of MTBF; software traditional 
> requires far more specific testing in order to reliably estimate MTBF in 
> advance of extensive use.  Thus, reliability in software depends on 
> context (configuration, integration, installation, and myriad other 
> issues) and test automation (with repeatable context and repeatable 
> tests) becomes crucial in assessing system reliability.

I agree that software reliability is different from hardware

Software reliability has historically been addressed, IIRC, as rate of
discovery of bugs.  BTW, a colleague of mine used to define "software
release" as that point beyond which all further bugs will be found in
system operation.  :-)

It is sometimes hard to distinguish a hardware failure from a software
failure.  An example is touchscreen misalignment that results in
apparent switching from the voter's choice to another choice.

I would tend to interpret the MTBF as system reliability, which includes
both hardware and software.  You would need to run the same kind of
test, because you would need to ensure that all functions are operable
and that performance is maintained.

> (2) To inspect COTS software, as you suggest, would require vast 
> resources.  Consider the existing DREs running on Microsoft's operating 
> system and using Microsoft's Access database.  These two Microsoft 
> programs are not likely to be inspected line by line in the way Open 
> Source code might be.  In addition, the diverse means by which states 
> tabulate and transfer counts to central state voting authorities involve 
> highly complex and diverse systems, few of which have ever been or ever 
> will be tested as total systems with either proprietary or open voting 
> systems.  To be real about this, we have to treat specialized voting 
> software (and hardware) separately from COTS items.  Still, it is Open 
> Test using automated testware that would allow us to propose total 
> system testing where total system inspection is simply not feasible.  
> Then, a total system can be characterized meaningfully as to its 
> estimated MTBF.

I agree that there are inspection methods other than manual reading of
code.  There are automated code inspection tools, although I don't know
much about them.  I once heard a presentation by someone from the
Defense Information Systems Agency who talked in general about their
automated tools.  Apparently, they run them on all source code used
within the Defense Department.  She mentioned that Microsoft does
overseas outsourcing of about 40% of their development, so the Defense
Department examines all their code for problems before buying it.  She
was speaking at an Open Source conference, and pointed out that they run
the same tools on open source code.  However, she didn't give any
details and I doubt they can be publicly disclosed.

Your other comments have been discussed in other emails, and I will only
summarize that you seem to be confusing the P1583 scope with other
issues.  However there appear to be scope/boundary issues that need to
be addressed, particularly involving central count optical scan
tabulation (that may essentially involve distribution of ballot checking
and tabulating functions between polling place and central site).

> (3) Security should not so easily be dismissed.  All vendors of systems 
> meeting the standard should provide the security equivalent of 
> Environmental Impact Statements:  what are the risks of (1) failure and 
> (2) corruption in a given system and how does the system meet them?  No 
> system lacking a testable, behavioral risk analysis should be considered 
> to meet the standard.  Certification, especially if proprietary and 
> hidden as to its substance, is a very thin bulwark against error and 
> corruption.  Again, Open Test seems a reasonable approach to this problem.

The analysis of how the system protects against threats is already
required.  I wasn't "easily dismissing" the security requirements.  I
was using the Common Criteria Evaluation Assurance Levels as a means of
explaining the approach.  If you want to get into the details, I suggest
you take a look at the Common Criteria.  ("Wade through" might have been
a better term to use.  :-)

> Of course, if we need to inspect Microsoft's XP and Access, well, let me 
> know and I am sure I can find some time to volunteer.  I assume that the 
> exceptions found though inspection would be fixed before the stuff is 
> used in an election, right?

If they can't fix the exceptions, the systems would become ineligible
for use in elections.  I think vendors would want to require the systems
to be certified for election use before even considering them for
inclusion in voting machines.  That is the only "COTS exception" the STG
effectively allowed.

> This is my personal and professional opinion, and it does not 
> necessarily reflect the opinions of my employer.
> -- Dick
> Richard C. Johnson, Ph.D.
> Applications Architect
> Oracle Corporation
> 631-689-3736
> Stanley A. Klein wrote:
> >As I stated on the conference call, it seems to me that there are a
> >relatively small set of issues in which it may make sense to have two
> >requirements, one stringent and one relaxed.  If all the stringent
> >requirements are met, the auditability and recoverability provided by
> >AVVPAT would remain optional.  If any of the relaxed requirements are
> >selected, the auditability and recoverability provided by AVVPAT would
> >become mandatory.
> >
> >The issues I have identified thus far are:
> >
> >1.  Reliability - I have been highly critical of the 163 hour MTBF.  I
> >have concluded that the "stringent" spec should be 15000 hours (allowing
> >a 0.001 failure probability on election day).  The relaxed spec can be
> >something between 163 and 15000.  (BTW, my comment was to increase to
> >1500, but I now believe that is inadequate.)
> >
> >2.  COTS - The stringent spec would be to inspect all COTS exactly the
> >same as non-COTS (IIRC, this would fully apply all resolutions of the
> >COTS STG).  The relaxed spec would be to allow some COTS to be accepted
> >without inspection.
> >
> >3.  Security - Although the state of the Common Criteria Protection
> >Profile is uncertain at this point, I will state the specs in terms of
> >Common Criteria Evaluation Assurance Levels.  The stringent spec would
> >be something roughly equivalent to EAL-5 or 6 (although I advocate
> >stepping away from the prescribed EAL's that are mandatory only if
> >automatic international recognition of certification is important).  The
> >relaxed spec could be EAL-2, which represents basic commercial practice.
> >
> >There may be a few additional areas to which this approach would apply,
> >but I leave it to others to suggest them.
> >
> >
> >Stan Klein
> >
> >
> >
> >On Tue, 2004-11-30 at 15:34, Deutsch, Herb wrote:
> >  
> >
> >>If this is on central count equipment or any central site software or
> >>equipment, it is not covered in our scope.  It would have to be part
> >>of the polling place unit to be considered.  In that case, since I
> >>believe you take an image of the entire system including the OS, how
> >>would you treat that today?
> >>
> >>
> >>-----Original Message-----
> >>From: Merle King [mailto:mking@kennesaw.edu]
> >>Sent: Tuesday, November 30, 2004 12:45 PM
> >>To: sklein@cpcug.org; hdeutsch@ESSvote.com; stds-1583-disc@IEEE.ORG;
> >>stephen.berger@IEEE.ORG
> >>Cc: ren@EFF.ORG; s.tatiner@IEEE.ORG; cots@lipsio.com;
> >>vince@lipsio.com;
> >>notable@mindspring.com; rivest@mit.edu; s.morganstein@Populex.com
> >>Subject: Re: COTS scope
> >>
> >>
> >>Stan, Vince, et al -
> >>
> >>The question that I intended to ask during the teleconference of Nov.
> >>30... Under this proposed scope, how will warranty work on the
> >>equipment
> >>be addressed?  If each time a defective driver or other COTS component
> >>is identified and a warranty recall is issued (example: recent recall
> >>by
> >>Dell for a defective driver for their RAID configuration used on
> >>central
> >>tabulation machines), how will this impact requalification of a
> >>system? 
> >>The owners of the systems (jurisdictions) will want to take advantage
> >>of
> >>warranty work while still covered by the warranty.   If the warranty
> >>work requires the entire system to be requalified...that could be
> >>impractical, given the time and expense involved, as well as the
> >>potential quantity of warranty claims over the life of the unit.
> >>
> >>- Merle
> >>
> >>Merle S. King
> >>http://science.kennesaw.edu/csis
> >>Chair, CSIS Department
> >>Kennesaw State University
> >>1000 Chastain Road, MB #1101
> >>Kennesaw, GA  30144-5591
> >>voice: 770-423-6354; fax: 770-423-6731
> >>
> >>
> >>    
> >>
> >>>>>"Stanley A. Klein" <sklein@CPCUG.ORG> 11/22/2004 10:37:40 AM >>>
> >>>>>          
> >>>>>
> >>When P1583 was last balloted, there were numerous comments to the
> >>effect
> >>that COTS should not be exempt from inspection.  An STG was set up to
> >>resolve those and other comments regarding COTS.
> >>
> >>The resolution generally was that COTS should undergo the exact same
> >>testing as any other parts of a voting system, unless it had been
> >>previously tested and certified as satisfying the requirements of the
> >>specification.
> >>
> >>It seems to me that the proper procedure is to incorporate the
> >>resolutions into the P1583 draft and move forward with the document.
> >>
> >>We are working to replace an FEC standard that is widely agreed to be
> >>inadequate.  The recent election resulted in numerous reports of
> >>problems with voting systems that can easily be attributed to the
> >>inadequacy of both the FEC standards and the ITA certification
> >>process.
> >>
> >>The COTS exemption is a major, glaring loophole in the FEC standard.
> >>Given the technical alternatives for vendors and the procedures used
> >>in
> >>other mission critical systems, there is absolutely no reason for
> >>allowing the COTS exemption to remain.  Of course it will be initially
> >>inconvenient for vendors, but any meaningful upgrade of the FEC
> >>standard
> >>will involve some initial inconvenience for vendors.
> >>
> >>>From my perspective, I see the leadership of P1583 slipping back into
> >>the kinds of process games that just a month or two ago they agreed to
> >>stop playing.  Their credibility is now on the line, and by
> >>implication
> >>the credibility of IEEE Standards as it relates to an issue with major
> >>national attention.
> >>
> >>
> >>Stan Klein
> >>
> >>
> >>
> >>On Mon, 2004-11-22 at 08:31, R. Mercuri wrote:
> >>    
> >>
> >>>Herb --
> >>>
> >>>I would like to add my voice to Vince's with regard to the
> >>>attempted replacement of our sub-group's work by material submitted
> >>>by Sandy, who isn't even a member of P1583. We were offended
> >>>at the meeting when it was suggested that Sandy replace us
> >>>as the co-chairs of the working group, and that materials be
> >>>fed to him, rather than us.  We had asked NUMEROUS TIMES for
> >>>input, so if that is going to be provided, it should be to
> >>>Vince and myself, not Sandy. Sandy is welcome to make suggestions,
> >>>of course, but not as a co-chair of the COTS-STG. We had also
> >>>provided material from our sub-committee for the draft document
> >>>that was not included, despite REPEATED requests to do so, and
> >>>this situation continues to be remiss.
> >>>
> >>>Now it seems that the entire standard will be compromised through
> >>>COTS exemptions.  Sandy's comment "code review of COTS software
> >>>cannot ensure that the COTS firmware/software is failsafe"
> >>>could be equally applied to the entire voting product --
> >>>why bother to have code review at all???  BECAUSE white box
> >>>AND black box testing are NECESSARY to reveal different
> >>>sorts of problems, such as VULNERABILITIES.  Additionally,
> >>>the code MUST be available in order to provide assurances
> >>>and evidence if later problems arise. There are certainly
> >>>ways in which vendors can contract with suppliers to provide
> >>>their code -- the choices they make can/should include products
> >>>that already are open to review (such as Linux or BSD-based).
> >>>
> >>>The bottom line is that there's no point in having a voting
> >>>system standard if parts of the ballot casting and tabulation
> >>>system are going to be exempt. It is bad enough that the standard
> >>>does not require any auditability feature to provide independent
> >>>recounts -- but to further allow non-inspected components, just
> >>>makes the whole thing a joke.
> >>>
> >>>I seriously hope that this issue can be resolved appropriately,
> >>>Rebecca Mercuri.
> >>>      
> >>>
> >>--
> >>Stanley A. Klein <sklein@cpcug.org>
> >>    
> >>

Stanley A. Klein, D.Sc.
Principal Consultant
Stan Klein Associates, LLC