[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

My COTS comments



Greetings!
   At long last, here are my comments.  I did not comment on my own items, and I also disregarded the 8 items about the definition of COTS.

   Feedback welcome.

   Soon, I'll produce a spreadsheet with everyone's comments and try to get a consensus.  If anyone is interested in a conference call at this late date, please propose a time and we can see if others wish to and can participate.

Type to y'all later,
Vince


----------------------------------

Alice - 001	5.6.1.1	"source code generated by COTS code development package and embedded in software modules for compilation or interpretation shall be provided in human readable form"    Some newer programming tools do not necessary  generate  traditional source code as reference within this clause. 

Proposed Change: Delete this clause

My comment: No.  Rather, require to be provided in human readable form the code written by a human, which in this context would be the input to a COTS code development package that produces "source code", and not necessarily the code produced by the package.  If the input to a COTS code development package is not "traditional source code", it still must be provided in a human readable form, even is that means a screen shot or whatever.

----------------------------------

Corry-022	5.1.1	p.20, 3rd para., 3rd sentence	 COTS may be properly installed and configured but still not meet requirements unless latest security patches are installed.

Proposed Change: Notwithstanding the fact that system certifiers can rely upon the prior validations of the individual components of the system [ ] provided they are properly installed and configured [with the latest security patches], there must still be an evaluation of the integrated system to make certain that security holes have not been left or created during the integration process.

My comment: Eliminate "with the latest security patches" (cf my comment for RGH 117); otherwise, agree.

----------------------------------

Corry-023	5.1.1	p. 20, last para.   Last three sentences should be separate paragraph. 

Proposed Change: Start new paragraph: [As] COTS products require updates due to a detected security breach or vulnerability [the]  voting system vendor must provide a method to assess the impact of COTS updates on the voting system, as well as a method for providing notice and distribution of updates to purchasers[, testing facilities, and election officials and boards].  Where COTS products are known to be inherently risky ([e.g.,] memory leaks in the C++ language), vendors must adequately describe the control methods they have employed to ensure these risks have been mitigated.

My comment: Agree

----------------------------------

Corry-139	6.4.4.1	2nd para., 1st sentence   Systems that are simply cobbled together (kluge might be a better description) from COTS components must not be exempted from environmental testing. I've had too many problems with little doohickies hung on some piece of otherwise great equipment that caused problems when fielded.

Proposed Change: Delete first sentence of second paragraph.

My comment:  Rewrite.  Would accept wording proposed by Doug Landol. ('Suggested rewrite: "Systems that employ the use of COTS hardware whose configuration has not been modified in any manner and does not violate the evaluated configuration requirements of its previous testing are not subjected to this segment of hardware testing."')

----------------------------------

Dill-35	6.6.2   If COTS hardware or software is in the trusted subset, it must be treated exactly like software or hardware designed by the vendor. " 

Proposed Change: Specify that the COTS exclusion only applies to system components outside the trusted subset.

My comment: Disagree.  COTS testing (performed by the COTS supplier) shall be to the same criteria and standards as it would be tested to were it designed by a voting equipment vendor.

----------------------------------

Mercuri - 013 (formerly mercuri-034)	4.6	Add bullet at end   It needs to be specified how updates to software are going to be supplied and performed. " 

Proposed Change: Documentation describing how an update is to be certified and performed, should there be a declared or discovered defect in the voting system, software, hardware, or firmware, or any COTS products used in or in the development of the system that could compromise its operation as an election device.

My comment: Agree; but should add verbiage to the effect that should such a defect be found, the device shall be decertified and the V&V procedures shall be amended to flag the found defect, and full regression testing shall be performed before the corrected device be again certified.

----------------------------------

Mercuri - 022 (formerly mercuri-048)	6.6.2	Paragraphs 2-4   The decision by the FEC to exempt COTS products from inspection has created a serious security flaw.  It should not be imperative that the IEEE standard continue to reflect this inappropriate practice.  All exemptions for COTS product review should be removed from this standard 

Proposed Change: Remove all exemptions for COTS product review from this standard on the grounds that such pose a serious security flaw.  COTS products shall be presented in their entirety for open review in the same way that vendor software is examined.

My comment:  Agree.  Cross-reference my comment for RGH 072.

----------------------------------

Mercuri - 064 (formerly mercuri-143)	5.6.1.1	Section	2 Concerns addressing use of COTS products need to be added. 

Proposed Change: COTS products, especially software libraries, are a vulnerable attack point and must be subject to risks assessment prior to use in voting products.

My comment: Agree

Proposed Change (continued): Configuration management should include vendor updates and alerts when flaws are detected that could compromise election operations or cast ballot data integrity.

My comment: Agree with the following changes:
   1)	Change "should" to "shall".
   2)	Add requirement for regression testing and V&V.

Proposed Change (continued): Object code modules should be provided such that compiled versions of programs can be compared.

My comment: Agree, but should add "unless the source code is provided and neither the build process nor the source code is modified."

----------------------------------

Mercuri - 078 (new)	7.13   Provision is made in the standard for update for COTS products releases, but there is no such provision for updating or decertifying non-COTS voting system components if such have been revealed to be insecure. 

Proposed Change: System changes that have resulted from identification of insecure voting system components must be propagated to all systems currently deployed.  (This might be more appropriate in the configuration management section, or a different section under maintenance.)

My comment:  Language needs to be stronger; otherwise, accept.  Any demonstrated lack of security or integrity in any component, COTS or otherwise, of the voting device shall result in decertification, the remedying whereof shall require that the V&V procedures shall be amended to flag the found defect, and full regression testing shall be performed before the corrected device be again certified.

----------------------------------

PPLX-035  	5.6.2.3	5.6.2.3 Software Modularity and Programming   "This section of the draft has this language: "However, COTS software is not required to be inspected for compliance with this requirement but must be the most recent version of the COTS product incorporating all security patches," ...

Proposed Change (partial): This section has several problems.  The module usage should be changed to subroutine or function. The relationship of these terms is not yet standardized.

My comment:  Not correct; ANSI/IEEE Std 610.12-1990, "IEEE Standard Glossary of Software Engineering Terminology" defines module thusly: 

   module. (1) A program unit that is discrete and identifiable with respect to compiling, combining with other units, and loading; for example, the input to, or output from, an assembler, compiler, linkage editor, or executive routine.
           (2) A logically separable part of a program. Note: The terms "module," "component," and "unit" are often used interchangeably or defined to be sub-elements of one another in different ways depending upon the context.

Proposed Change (continued): Remove the strict requirement of only one exit per subroutine or function.

My comment:  Agree that it should be a suggestion, not a requirement.  However, use of only a single exit point so simplifies formal testing that I doubt anyone would want to not follow this rule.

Proposed Change (continued): Change so the most recent version of COTS is not required.

My comment:  Agree; explained in detail in my comments for RGH 117.

----------------------------------

RGH 006	5.1.1	last paragraph	   There is a change of gears just past the middle of the paragraph. 

Proposed Change: Paragraph break with the sentence beginning "COTS products require updates?"

My comment:  Agree.  

----------------------------------

RGH 007	5.1.1	last paragraph	   Memory leaks in C++ is not an example of an inherent risk in COTS products.

Proposed Change: More appropriate would be "security vulnerabilities in Microsoft products".

My comment:  Use my language found in Lipsio-16.  Methinks it inappropriate to cite a specific COTS vendor and not necessarily correct to reckon a memory leak as a "security" vulnerability.

----------------------------------

RGH 072	5.4	Second paragraph above clause 5.4.1. COTS equipment will be entrusted with counting votes but is exempted from this standard with a "proven record of performance"? OEMs of voting equipment also have "proven" track records but must still test to this standard? This seems unreasonable.

Proposed Change: Either require COTS equipment to comply to the same standards as all other voting equipment or remove the paragraph altogether. 

My comment: Agree, but should add a note that what distinguishes COTS from other components is that the testing of COTS components is performed by the suppliers thereof and proof of that testing is supplied with the COTS components, while for components designed by a voting machine vendor, the latter must perform testing.

----------------------------------

RGH 117	5.6.2.3	 "...COTS software ... must be the most recent version of the COTS product ..."     The most recent version is not always stable enough to deploy and may not be compatible with the other aspects of the application.   The vendor must have the latitude to employ the COTS versions and upgrades at the appropriate time.

Proposed Change: Remove this clause.

My comment: Agree.  Consider FAA and FDA criteria which allow whatever version of COTS is certified for the purpose.  The price of repeating V&V every time a COTS software component is revised is prohibitive.

----------------------------------

schneidewind - 005	5.6.2.3	Pg. 70	  "Why specify that COTS software must be designed in a modular or object oriented fashion and not inspect it for compliance? "

Proposed Change: Either eliminate the requirement or inspect for compliance.

My comment:  Agree.  Note that the mandated design simplifies testing, but I don't see that it should be a hard requirement, especially for where it is found necessary to use assembly language.  But if so mandated, then of course, it needs to be tested.   

----------------------------------

schneidewind - 006	6.4.4.1	Pg. 100   "Why exempt COTS hardware from environmental testing? "

Proposed Change: Require environmental testing of COTS hardware.

My comment:  Agree unless, of course, the COTS vendor supplies test data showing compliance and it can not be demonstrated that the manner wherein the hardware is used would invalidate that testing in context.

----------------------------------

schneidewind - 007	6.6.2	Pg. 107   "COTS software must work in conjunction with the voting application software. Therefore, it should be subjected to the same rigor of testing as the application software.  " 

Proposed Change: Eliminate the exemption of COTS software from the testing requirement.

My comment: Agree.

----------------------------------

Sklein-007	5.6	Para 5.6.1.1   Unmodified COTS must be evaluated at the source code level to protect against the threats identified in 5.3.2.1 (A). 

Proposed Change: Delete "Unmodified third-party software is not subject to code examination; however,"   and replace it with "All third party software shall be subject to source code an d other examination to preclude the presence of trap doors, hard-coded passwords, vulnerabilities and other non-deliberate errors, deliberate errors allowing the introduction of malicious code, and malicious code of any kind, especially malicious code intended to trigger upon use of the software in voting systems."

My comment:  Agree, but need to insert ", among other things," between "presence" and "of", lest it be implied that the list is exhaustive.

----------------------------------

Sklein-044	5.6.2.3	First paragraph   COTS must meet the requirements of 5.1.3.1 

Proposed Change: In the second sentence, after "security requirements defined in" insert "Section 5.1.3.1 and".

My comment: Agree.

----------------------------------

Sklein-045	5.6.2.3	First paragraph   COTS virus detection programs are not available for all operating systems. 

Proposed Change: In the second sentence, replace the comma after "security patches" with "and".  Replace "and must be tested" by ".  In complying with the requirement of 5.1.3.1, the vendor must document how the COTS has been defended against the threats identified in 5.1.2.3 (A-1), (A-3), (B-1) and (B-2), such as by testing".

My comment: Agree.

----------------------------------

Sklein-056	5.1.3	All   Voter verified paper needs to be mandatory under certain circumstances 

Proposed Change: Add to the section created under comment SK-4 above:  A voter verified paper audit trail is mandatory for any system in which any of the following conditions is found:  1.  Either the system software or any COTS used as either a system component or development tool, including compilers, libraries, and other tools, is too complex to clearly and thoroughly evaluate at the source code level to ensure absence of backdoors and other malicious code or means of introducing malicious code.  2.  All other security, accuracy, integrity, and availability requirements are not satisfied clearly, easily, and without any question or requirement for interpretation.  3. -  There are any reports or significant suspicions that similar technology may have failed to record all ballots exactly as cast.  4. - There is any question whatever about the ability of all using jurisdictions to easily and completely satisfy all assumptions regarding supervision of machines and relevant personnel at all times machines are in use, regarding fully secure storage of machines between elections, and regarding other procedures intended to prevent tampering.  This latter condition is triggered if there is any significant complexity, budgetary impact, requirement for personnel qualification or training, or other impediment to full and complete satisfaction of all assumptions and recommended procedures.

My comment:  Out of scope to mandate VVPT here.  However, please note that the stringency of V&V would not, in my opinion, be nearly so great in a system with an independent audit trail.  As things stand where no such audit trail exists, I am inclined to test the systems to the standards of, at least, RTCA/DO-178B, ("Software Considerations in Airborne Systems and Equipment Certification"), while with the presence of an independent audit trail, robustness is much less critical, and the main concern would be that the device is reliable, not crashing or malfunctioning so that it is an annoyance to use or be in charge of, which is to say less stringent than FDA class 2.

----------------------------------

Sklein-057	5.6.2.3	5.6.1.1	2   COTS evaluated should include compilers, libraries, and any other software tools used in system development and capable of introducing backdoors or other malicious code. 

Proposed Change: COTS to be evaluated shall include compilers, libraries, and any other software tools used in system development and capable of introducing backdoors or other malicious code.

My comment: Agree.

----------------------------------

Simons - 002	5.1.1	the sentence that reads, "The security countermeasures implemented by an IT system typically use functions of the underlying products and depend upon the correct operation of those products and their security functions."   This is a far too vague and does nothing to address the security issues. 

Proposed Change: Replace sentence with the following: "Underlying products, such as operating systems, database systems, firewalls, network devices, web browsers, smart cards, biometric devices, general purpose application components, libraries, and hardware platforms, that are crucial to the correct and secure operation of the entire system must be thoroughly tested.  This includes COTS systems.  In addition, there must be a line by line code review of ALL software that interacts with the voting system in any fashion.  This is required because of the potential risk of malicious code."

My comment:  Assuming that the scope is explicitly limited to what is internal t the voting device, I agree; for external devices connected to the voting device, proof that these devices can not affect the functioning of the voting device should be sufficient.

----------------------------------

Simons - 017	5.1.3.4.2	the entire section   There is no way to adequately test against all possible bugs and malicious code in COTS. 

Proposed Change: Add the requirement that all COTS used in any voting system must be open source.

My comment:  Don't know: Agree in principle that open source should be required but if we require all tools to be certified, to have undergone formal V&V, I don't see know that we can mandate this.  Disagree with the implication that open source will provide adequately testing; rather, it will improve the adequacy of testing. 

----------------------------------

VCW-02	5.1.1	2nd to last para   The COTS products may also be subject to a security evaluation themselves; such evaluations can support the  voting system evaluation process.

Proposed Change: delete second space before "voting system".

My comment:  Sure.  Turn over to the editing committee with the suggestion that they excise all redundant spaces within sentences.