23	Alice - 001	5.6.1.1		T	"""source code generated by COTS code development package and embedded in software modules for compilation or interpretation shall be provided in human readable form""    Some newer programming tools do not necessary  generate  traditional source code as reference within this clause. "	Delete this clause	

86	Corry-022	5.1.1	"p.20, 3rd para., 3rd sentence"	T	COTS may be properly installed and configured but still not meet requirements unless latest security patches are installed.	"Notwithstanding the fact that system certifiers can rely upon the prior validations of the individual components of the system [ ] provided they are properly installed and configured [with the latest security patches], there must still be an evaluation of the integrated system to make certain that security holes have not been left or created during the integration process."	

87	Corry-023	5.1.1	"p. 20, last para."	E	Last three sentences should be separate paragraph.	"Start new paragraph: [As] COTS products require updates due to a detected security breach or vulnerability [the]  voting system vendor must provide a method to assess the impact of COTS updates on the voting system, as well as a method for providing notice and distribution of updates to purchasers[, testing facilities, and election officials and boards].  Where COTS products are known to be inherently risky ([e.g.,] memory leaks in the C++ language), vendors must adequately describe the control methods they have employed to ensure these risks have been mitigated."	

203	Corry-139	6.4.4.1	"2nd para., 1st sentence"	T	Systems that are simply cobbled together (kluge might be a better description) from COTS components must not be exempted from environmental testing. I've had too many problems with little doohickies hung on some piece of otherwise great equipment that caused problems when fielded.	Delete first sentence of second paragraph.	

285	df1	3	No. 26	E	"COTS - ""These devices and software are exempted from certain portions of the qualification testing process so long as such products are not modified in any manner for use in the voting system."""	Delete sentence.  I do not believe that is appropriate in a reference section defining COTS.	

368	Dill-35	6.6.2		T	"If COTS hardware or software is in the trusted subset, it must be treated exactly like software or hardware designed by the vendor."	Specify that the COTS exclusion only applies to system components outside the trusted subset.	

382	Dill-7	3	Def #26	T	"Explanation about exemption is unnecessary, and may become inconsistent if we add  change requirements on COTS"	Delete last sentence of definition.	

505	Lipsio-12	5.1.1	Para. 5	T	"The treatment of COTS products contradicts section 5.1.2.2, "Elements of Security Outside of Vendor Control"."	"Change "COTS product may" to "COTS products shall".  Mandate compliance with section 4.3.11 ("Previously developed or purchased software") of IEEE Std 1228-1994, "IEEE Standard for Software Safety Plans"."	

507	Lipsio-14	5.1.1	Para. 7	T	There is implied a lack of testing in "COTS products require updates due to a detected security breach or vulnerability"; nothing that requires an update should pass testing.	"Mandate that testing preclude any security breach or vulnerability; mandate compliance with section 4.3.11 ("Previously developed or purchased software") of IEEE Std 1228-1994, "IEEE Standard for Software Safety Plans".  Mandate COTS be subject to the specifications of IEEE Std 1008ª-1987 (R1993), "IEEE Standard for Software Unit Testing".  Add reference to IEEE Std 982.1ª-1988, "IEEE Standard Dictionary of Measures to Produce Reliable Software".  "	

508	Lipsio-15	5.1.1	Para. 7	T	""The voting system vendor must provide a method to assess the impact of COTS updates on the voting system, as well as a method for providing notice and distribution of updates to purchasers" is inconsistent with IEEE Std 1012-1998."	"Bring into conformance with Annex D ("V&V of reusable software") of IEEE Std 1012-1998, "IEEE Standard for Software Verification and Validation", e.g., "Reusable software (in part or whole) includes software from software libraries, custom software developed for other applications, legacy software, or commercial-off-the-shelf (COTS) software. The V&V tasks of Table 1 are applied to reusable software just as they are applied to newly developed software. However, the inputs for these tasks may not be available for reusable software, reducing visibility into the software products and processes.""	

509	Lipsio-16	5.1.1	Para. 7	T	Memory leaks are the result of using C++ language inappropriately; they are not a risk of a COTS C++ compiler.	Eliminate "(ex. memory leaks in the C++ language)"	

549	Lipsio-3E	5.6.2.2	para. 1	T	"Industry standard COTS compiler and runtime interpreter both is not defined and assumes that, contrary to reality, something is fail-safe and fool-proof by virtue of being in common use."	"Require all tools, including compilers and interpreters, to be validated and verified in the same manner as application software."	

554	Lipsio-43	5.6.2.3	Para. 1	T	""COTS software is not required to be inspected"" is contrary to such other mission-critical methodologies as those used by the FDA and FAA, and contradicts what is specified in section 5.1.3.3.2"	"Eliminate the section, or, better yet, reverse its sense."	

555	Lipsio-44	5.6.2.3	Para. 1	T	There is implied a lack of testing in "COTS products require updates due to a detected security breach or vulnerability"; nothing that requires an update should pass testing.	"Mandate that testing preclude any security breach or vulnerability; mandate compliance with section 4.3.11 ("Previously developed or purchased software") of IEEE Std 1228-1994, "IEEE Standard for Software Safety Plans".  Mandate COTS be subject to the specifications of IEEE Std 1008ª-1987 (R1993), "IEEE Standard for Software Unit Testing".  Add reference to IEEE Std 982.1ª-1988, "IEEE Standard Dictionary of Measures to Produce Reliable Software".  "	

556	Lipsio-45	5.6.2.3	Para. 1	T	There is implied a lack of testing in "the most recent version of the COTS product incorporating all security patches" "; nothing that requires an update should pass testing.	"Bring into conformance with Annex D ("V&V of reusable software") of IEEE Std 1012-1998, "IEEE Standard for Software Verification and Validation", e.g., "Reusable software (in part or whole) includes software from software libraries, custom software developed for other applications, legacy software, or commercial-off-the-shelf (COTS) software. The V&V tasks of Table 1 are applied to reusable software just as they are applied to newly developed software. However, the inputs for these tasks may not be available for reusable software, reducing visibility into the software products and processes.""	

562	Lipsio-4B	6.4.4.1	Para. 2	T	"COTS hardware must have been tested to the rigor required of non-COTS components; if the supplier has not done this, then COTS hardware must be treated like any other component."	"Change paragraph to "COTS systems or components must be documented by their suppliers to have been tested to at least the same rigor as required of voting devices as specified hereinbelow; else, the said COTS components shall be tested in a like manner to any other component.""	

564	Lipsio-4D	6.6.2	para. 3 & 4	T	""Unmodified, general purpose COTS non-voting software ...is not subject to code examination...is not subject to the full code review and testing" is contrary to such other mission-critical methodologies as those used by the FDA and FAA, and contradicts what is specified in section 5.1.3.3.2."	"Eliminate the sections; ensure compliance with section 4.3.11 ("Previously developed or purchased software") of IEEE Std 1228-1994, "IEEE Standard for Software Safety Plans"."	

596	Lipsio-6D	3.26		E	"Second sentence is not part of the definition.  Whether or not my later comments on COTS are accepted, "These devices and software are exempted from certain portions of the qualification testing process so long as such products are not modified in any manner for use in the voting system" does not belong in the definition."	Delete the second sentence.	

609	Lipsio-7A	5.1.1	Para. 7	E	It is unclear if "vendors" means "COTS vendors" or "voting equipment vendors" in "vendors must adequately describe the control methods they have employed to ensure these risks have been mitigated."	Change vendors" to "COTS vendors" or "voting equipment vendors".	

615	Lipsio-80	5.1.3.6.5		E	COTS software was already covered in 5.1.1.	Eliminate "and software" from the first paragraph and eliminate item "a".	

624	Lipsio-89	5.6.2		E	""The software used by voting systems is selected by the vendor" appears to mean "COTS is selected"; else, it contradicts the subsequent sentence.  Change the opening words from "The software" to "The COTS software"."		

648	MercuriD50 - 013 (formerly mercuri-034)	4.6	Add bullet at end	General	It needs to be specified how updates to software are going to be supplied and performed.	"* Documentation describing how an update is to be certified and performed, should there be a declared or discovered defect in the voting system, software, hardware, or firmware, or any COTS products used in or in the development of the system that could compromise its operation as an election device."		"NC - Out of scope.   This is determined by the relevent election authority, such as NASED or the individual state officials."	The lack of any specification regarding updates and configuration management is a serious security flaw that must be addressed by the standard.

657	MercuriD50 - 022 (formerly mercuri-048)	6.6.2	Paragraphs 2-4	General	The decision by the FEC to exempt COTS products from inspection has created a serious security flaw.  It should not be imperative that the IEEE standard continue to reflect this inappropriate practice.  All exemptions for COTS product review should be removed from this standard.	Remove all exemptions for COTS product review from this standard on the grounds that such pose a serious security flaw.  COTS products shall be presented in their entirety for open review in the same way that vendor software is examined.		"NC - Only unmodified COTS is exempted.  This is a drastic change that permeates the spec and cannot be considered at this time.  Also, if required, a vendor cannot control COTS source availability which would also limit vendor choices in system design. "	"Unmodified COTS is not exempt from serious security flaws, as evidenced in the continual update patches that must be downloaded for Microsoft operating systems, for example.  The exemption for COTS products was erroneous in the FEC document and is ludicrous here.  This must be changed."

699	MercuriD50 - 064 (formerly mercuri-143)	5.6.1.1	Section	General	Concerns addressing use of COTS products need to be added.	"COTS products, especially software libraries, are a vulnerable attack point and must be subject to risks assessment prior to use in voting products.  Configuration management should include vendor updates and alerts when flaws are detected that could compromise election operations or cast ballot data integrity.  Object code modules should be provided such that compiled versions of programs can be compared."		"NC - This is covered in section 5.1.1 as shown by the following excerpt:   ""COTS products require updates due to a detected security breach or vulnerability.  The voting system vendor must provide a method to assess the impact of COTS updates on the voting system, as well as a method for providing notice and distribution of updates to purchasers."""	"COTS products themselves should be subject to thorough evaluation, not just their updates. COTS provide a significant security risk. This must be addressed by the working group. "

713	MercuriD50 - 078 (new)	7.13		Technical	"Provision is made in the standard for update for COTS products releases, but there is no such provision for updating or decertifying non-COTS voting system components if such have been revealed to be insecure."	"System changes that have resulted from identification of insecure voting system components must be propagated to all systems currently deployed.  (This might be more appropriate in the configuration management section, or a different section under maintenance.)"			

722	RGH 006	5.1.1	last paragraph	E	There is a change of gears just past the middle of the paragraph.	"Paragraph break with the sentence beginning ""COTS products require updates""""	RGHKR001		

723	RGH 007	5.1.1	last paragraph	T	Memory leaks in C++ is not an example of an inherent risk in COTS products.	"More appropriate would be ""security vulnerabilities in Microsoft products""."	RGHKR002		

788	RGH 072	5.4	Second paragraph above clause 5.4.1.	T	"COTS equipment will be entrusted with counting votes but is exempted from this standard with a ""proven record of performance""? OEMs of voting eqipment also have ""proven"" track records but must still test to this standard? This seems unreasonable."	Either require COTS equipment to comply to the same standards as all other voting equipment or remove the paragraph altogether. 	RGHMD010

833	RGH 117	5.6.2.3		T	""""COTS software "must be the most recent verion of the COTS product """     The most recent version is not always stable enough to deploy and may not be compatible with the other aspects of the application.   The vendor must have the latitude to employ the COTS versions and upgrades at the appropriate time."	Remove this clause.	RGHKC016

847	schneidewind - 001	6.6.2	Pg. 107	T 	"How do you know that the COTS software has not been modified?	COTS software should not be exempt from code inspection.	"	"Eliminate the exemption.	"	

848	schneidewind - 002	3	"Pg. 10	Line 26	Definition 26"	T	"COTS Hardware and software should not be exempted from qualification testing.		This exemption should not be included in Definitions. The exemption is not a definition. 	"	"Eliminate the exemption.	"	

851	schneidewind - 005	5.6.2.3	Pg. 70	T	"Why specify that COTS software must be designed in a modular or object oriented fashion and not inspect it for compliance?	"	Either eliminate the requirement or inspect for compliance.	

852	schneidewind - 006	6.4.4.1	Pg. 100	T	"Why exempt COTS hardware from environmental testing?	"	Require environmental testing of COTS hardware.	

853	schneidewind - 007	6.6.2	Pg. 107	T	"COTS software must work in conjunction with the voting application software. Therefore, it should be subjected to the same rigor of testing as the application software.	"	Eliminate the exemption of COTS software from the testing requirement.	

863	Simons - 002	5.1.1	"the sentence that reads, ""The security countermeasures implemented by an IT system typically use functions of the underlying products and depend upon the correct operation of those products and their security functions."""	G	This is a far too vague and does nothing to address the security issues.	"Replace sentence with the following: ""Underlying products, such as operating systems, database systems, firewalls, network devices, web browsers, smart cards, biometric devices, general purpose application components, libraries, and hardware platforms, that are crucial to the correct and secure operation of the entire system must be thoroughly tested.  This includes COTS systems.  In addition, there must be a line by line code review of ALL software that interacts with the voting system in any fashion.  This is required because of the potential risk of malicious code."""	

878	Simons - 017	5.1.3.4.2	the entire section	G	There is no way to adequately test against all possible bugs and malicious code in COTS.	Add the requirement that all COTS used in any voting system must be open source.	

889	Sklein-007	5.6	Para 5.6.1.1	T	Unmodified COTS must be evaluated at the source code level to protect against the threats identified in 5.3.2.1 (A).	"Delete "Unmodified third-party software is not subject to code examination; however,"   and replace it with "All third party software shall be subject to source code an d other examination to preclude the presence of trap doors, hard-coded passwords, vulnerabilities and other non-deliberate errors, deliberate errors allowing the introduction of malicious code, and malicious code of any kind, especially malicious code intended to trigger upon use of the software in voting systems.""	

926	Sklein-044	5.6.2.3	First paragraph	T	COTS must meet the requirements of 5.1.3.1	"In the second sentence, after "security requirements defined in" insert "Section 5.1.3.1 and"."	

927	Sklein-045	5.6.2.3	First paragraph	T	COTS virus detection programs are not available for all operating systems.	"In the second sentence, replace the comma after "security patches" with "and".  Replace "and must be tested" by ".  In complying with the requirement of 5.1.3.1, the vendor must document how the COTS has been defended against the threats identified in 5.1.2.3 (A-1), (A-3), (B-1) and (B-2), such as by testing"."	

933	Sklein-051	Section 3	Item 26	T	Unmodified COTS are not exempt from evaluation to preclude the threats identified in 5.3.2.1 (A).	Delete the second sentence of the definition.	

938	Sklein-056	5.1.3	All	T	Voter verified paper needs to be mandatory under certain circumstances	"Add to the section created under comment SK-4 above:  A voter verified paper audit trail is mandatory for any system in which any of the following conditions is found:  1.  Either the system software or any COTS used as either a system component or development tool, including compilers, libraries, and other tools, is too complex to clearly and thoroughly evaluate at the source code level to ensure absence of backdoors and other malicious code or means of introducing malicious code.  2.  All other security, accuracy, integrity, and availability requirements are not satisfied clearly, easily, and without any question or requirement for interpretation.  3. -  There are any reports or significant suspicions that similar technology may have failed to record all ballots exactly as cast.  4. - There is any question whatever about the ability of all using jurisdictiions to easily and completely satisfy all assumptions regarding supervision of machines and relevant personnel at all times machines are in use, regarding fully secure storage of machines between elections, and regarding other procedures intended to prevent tampering.  This latter condition is triggered if there is any significant complexity, budgetary impact, requirement for personnel qualification or training, or other impediment to full and complete satisfaction of all assumptions and recommended procedures."	

939	Sklein-057	5.6.2.3	5.6.1.1	T	"COTS evaluated should include compilers, libraries, and any other software tools used in system development and capable of introducing backdoors or other malicious code."	"COTS to be evaluated shall include compilers, libraries, and any other software tools used in system development and capable of introducing backdoors or other malicious code."	

945	VCW-02	5.1.1	2nd to last para	Editorial	The COTS products may also be subject to a security evaluation themselves; such evaluations can support the  voting system evaluation process. 	"delete second space before ""voting system"""	

977	wfw -001	Section 3 Definitions	26	Editorial	"COTS, whether modified or not must be tested at least to system level."	I would drop the last sentence.	

982	PPLX-001	3	Section 3.  Definition # 26	E	"In discussing the definition of COTS, this section goes on to say, "These devices and software are exempted from certain portions of the qualification testing process so long as such products are not modified in any manner for use in the voting system."  In general it is not a good idea to discuss policy in a definition.  In particular, doing so here raises the question, which portions of the testing process are "certain" portions from which testing is exempted."	Remove the text in quotes.	

1016	PPLX-035  	5.6.2.3	5.6.2.3 Software Modularity and Programming	T & E	"This section of the draft has this language:	"However, COTS software is not required to be inspected for compliance with this requirement but must be the most recent version of the COTS product incorporating all security patches," [emphasis added]	This section may be ambiguous.    Must the latest version always be incorporated or only the latest version of security patches?  What if the security patch is not relevant to the particular operation.		In any case, forcing the latest version of COTS software is a configuration control nightmare and will result in endless re-qualification.  One interpretation of this section is that software written to run on Windows 2000 must be rewritten and re-qualified to run on Windows XP even if it runs perfectly well on Win2000.  An even worse interpretation requires vendors to update hard disk controllers with new firmware and drivers every time a new software version is available.  We don't think this is intended nor desirable.		Note the term "Module".  The term Module is used here as it is used in the FEC VSS and we believe this usage to be non-standard.  A module should be a collection of related subroutines and functions.  A module may contain more than one subroutine.   But the proposed text says that a Module must have one entry and one exit.  A Subroutine or Function should have one entry (it is debatable if it should have one exit), but this should not be the requirement for a "Module."		We also wish to question the wisdom of requiring a single exit.  Such a requirement generally indicates good structure, but a blind adherence to this precept results in code that is hard to maintain and hard to read.		Consider the following artificial code fragment:		If A=B then  'If A equals B then we must test if C equals D		'Test to see if C equals D and if it does, not further processing is necessary	 If C = D then	  'When C = D do some processing	 {some code processing}	Exit sub  'We are done	End if	'Now execute code that is relevant if A = B but C does not equal D	{some code processing}		end if			If we do not allow an exit statement, the processing must set a flag and that flag must be tested to avoid running the code that is not intended. An Else statement does not always  solve this problem.  Exit statements, when used judicially and within an overall good structure enhance maintainability and readability.	"	"This section has several problems.  The module usage should be changed to subrouting or function, remove the strict requirement of only one exit per subroutine or function.  Change so the most recent version of COTS is not required."