Proposed
Resolutions for most COTS Comments for Draft 5.0 of IEEE P-1583
--- Sklein-051, wfw
- 001, Dill-7, df1, schneidewind – 002, PPLX-001, and
Lipsio-6D
Comments: In section 3.26, in the definition of COTS,
the second sentence doesn’t belong in a definition.
Proposed Resolution:
Remove “COTS” from the
definitions section.
In "Abbreviations and
Acronyms" add to the definition of “COTS” this text: “(subsystems or
components; software, electrical, electronic, mechanical, et cetera).”
--- MercuriD50 - 013 (formerly mercuri-034)
Comment: The principle
should be that COTS products should be evaluated to the same level of assurance
as non-COTS components. Documentation of this evaluation shall be available to
the ITA, which shall review it to ensure that the evaluation was adequate. The COTS product must be identical to the
product that was reviewed.
Proposed change: COTS should not be exempted, but comparable levels of evaluation
with a non-COTS components in the same role should be accepted if they are
documented and verified.
Discussion: All who commented agreed, but several
remarked that certification and administrative procedures are out of scope, and
a couple of people wanted more verbiage.
The proposed resolution attempts to account for these.
Proposed Resolution:
At the end of section 4.6, under
the bullet “software and firmware”, add these sub-bullets:
F
Description of how an update is
to be performed, together with guidelines on how certification may be affected,
should there be, declared or discovered, any defect in the voting system's
software, hardware, or firmware, or any COTS products used in the development
of the system that could compromise its operation as an election device.
F
Notice that should such a defect
as described in the previous bullet be found, the owners of the product should
be notified and device should, or at least may, be decertified until the
correcting update shall have been applied.
At the end of section 7.12.1, add
these bullets:
f)
Include a procedure to inform
equipment owners should there be, declared or discovered, any defect in the
voting system's software, hardware, or firmware, for any COTS products used in
the development of the system that could compromise its operation as an
election device.
g)
Include a procedure for when a
defect of the sort described in the previous bullet is found, to initiate
amending the V&V procedures so as to flag the found defect, and to
subsequently perform full regression testing.
h)
Include a procedure to provide owners
of the product with the required update subsequent to the retesting mandated in
the previous bullet.
--- RGH 072
Comment: COTS equipment will
be entrusted with counting votes but is exempted from this standard with a
"proven record of performance"? OEMs of voting equipment also have
"proven" track records but must still test to this standard? This
seems unreasonable.
Proposed change: Either require COTS equipment to comply to
the same standards as all other voting equipment or remove the paragraph
altogether.
Proposed Resolution:
(with
Bob Oliver dissenting) In the second
paragraph above clause 5.4.1, change “a documented record of performance under
conditions defined in the Standards” to “have been tested by their suppliers to
the rigor mandated in these Standards, with documentation of such testing being
supplied with the COTS product.”
--- VCW-02
Comment: The COTS products
may also be subject to a security evaluation themselves; such evaluations can
support the voting
system evaluation process.
Proposed change: delete second space before "voting system"
Proposed Resolution:
Accept proposed change; turn over
to the editing committee with the suggestion that they excise all redundant
spaces within sentences.
--- Corry-023 and RGH 006
Comments: There is a change
of gears just past the middle of the paragraph. [5.1.1, last
paragraph]. Last three sentences
should be separate paragraph.
Proposed changes: Paragraph break with the sentence beginning “COTS products require
updates”. Start new paragraph: [As] COTS
products require updates due to a detected security breach or vulnerability
[the] voting system vendor must provide
a method to assess the impact of COTS updates on the voting system, as well as
a method for providing notice and distribution of updates to purchasers[,
testing facilities, and election officials and boards]. Where COTS products are known to be
inherently risky ([e.g.,] memory leaks in the C++ language), vendors must
adequately describe the control methods they have employed to ensure these
risks have been mitigated.
Proposed Resolution:
Split the last paragraph of
section 5.1.1 into 2 paragraphs; begin the 2nd paragraph with the sentence:
“Guidance on how to securely configure COTS products requires ...”. Note that the same
paragraph will be modified because of the following change.
--- RGH 007 and Lipsio-16
Comments: Memory
leaks in C++ is not an example of an inherent risk in COTS
products. Memory leaks are the result of
using C++ language inappropriately; they are not a risk of a COTS C++ compiler.
Proposed change: More appropriate would be “security vulnerabilities in Microsoft
products”. Eliminate “(ex. memory leaks in the C++ language)”.
Proposed Resolution:
Eliminate the example “(ex.
memory leaks in the C++ language)”.
--- Corry-022
Comment: COTS may be
properly installed and configured but still not meet requirements unless latest
security patches are installed.
Proposed change: Notwithstanding the fact that system certifiers can rely upon the
prior validations of the individual components of the system [ ] provided they
are properly installed and configured [with the latest security patches], there
must still be an evaluation of the integrated system to make certain that
security holes have not been left or created during the integration process.
Discussion: Several of those commenting noted potential
conflict with other comments on instability and the impacts of latest versions and
patches, so the long parenthetical clause is being added to the original
proposed change to clarify that.
Proposed Resolution:
In section 5.1.1, on page 20, 3rd
paragraph, 3rd sentence, after “properly installed and configured” add the
words “with the latest security patches (available at the time of system design
or update; not necessarily to be construed as requiring retroactive
installation of patches in tested, released versions of code)”
--- Lipsio-7A
Comment: It is unclear if
“vendors” means “COTS vendors” or “voting equipment vendors” in “vendors must
adequately describe the control methods they have employed to ensure these
risks have been mitigated.”
Proposed change: Change vendors” to “COTS vendors” or “voting equipment vendors”.
Discussion: All those who
commented accepted the change but most did not specify whether it should be
changed to “COTS vendors” or to “voting equipment vendors”; of those who did
specify, one wanted “voting equipment vendors” while rest wanted both. Therefore, both will be specified on the theory
that COTS software will need to meet the specifications of in-house software,
although the voting equipment vendor may perform the describing for the COTS
products.
Proposed Resolution:
In paragraph 7 of section 5.1.1,
change “vendors” to “COTS vendors or voting equipment vendors”.
--- PPLX-035
Comment: This section of the
draft has this language: “However, COTS software is not required to be
inspected for compliance with this requirement but must be the most recent
version of the COTS product incorporating all security patches,” …
Proposed change: This section has several problems.
The module usage should be changed to subroutine or function, remove the
strict requirement of only one exit per subroutine or function. Change so the most recent version of COTS is
not required.
Discussion: The comment was
quite long and the answers were short. Every commentator basically agreed with
the changes; most wrote “delete entire
section”, which does not seem to contradict what the other commentators specified,
but since this applies to much more than COTS, only the complained about parts
are being deleted.
Proposed Resolution:
See proposed resolution of RGH
117 for resolution of all comments on section 5.6.2.3.
--- Sklein-044
Comment: COTS must meet the
requirements of 5.1.3.1
Proposed change: In the second sentence, after “security requirements defined in”
insert “Section 5.1.3.1 and” ; see proposed resolution
of RGH 117 for resolution of all comments on section 5.6.2.3.
Proposed Resolution:
In the second sentence of section
5.6.2.3, after “security requirements defined in” insert “Section 5.1.3.1 and”.
--- Sklein-045
Comment: COTS virus
detection programs are not available for all operating systems.
Proposed change: In the second sentence, replace the comma after “security patches”
with “and”. Replace “and must be tested”
by “. In complying with the requirement
of 5.1.3.1, the vendor must document how the COTS has been defended against the
threats identified in 5.1.2.3 (A-1), (A-3), (B-1) and (B-2), such as by
testing”.
Proposed Resolution:
Accept; see proposed resolution
of RGH 117 for resolution of all comments on section 5.6.2.3.
--- Lipsio-43
Comment: “COTS software is
not required to be inspected…” is contrary to such other mission-critical
methodologies as those used by the FDA and FAA, and contradicts what is
specified in section 5.1.3.3.2
Proposed change: Eliminate the section, or, better yet, reverse its sense.
Proposed Resolution:
Reverse the sense; see proposed resolution
of RGH 117 for resolution of all comments on section 5.6.2.3.
--- schneidewind
- 005
Comment: Why specify that
COTS software must be designed in a modular or object oriented fashion and not
inspect it for compliance?
Proposed change: Either eliminate the requirement or inspect for compliance.
Discussion: All those who
commented agreed, although they were divided as to whether to eliminate the
section or to mandate compliance.
However, after this section editing for other accepted comments, it had
already been eliminated.
Proposed Resolution:
Eliminate the requirement; see
proposed resolution of RGH 117 for resolution of all comments on section
5.6.2.3.
--- RGH 117
Comment: "…COTS
software …must be the most recent version of the COTS product …" The most recent version is not always
stable enough to deploy and may not be compatible with the other aspects of the
application. The vendor must have the
latitude to employ the COTS versions and upgrades at the appropriate time.
Proposed change: Remove this clause.
Proposed Resolution:
Change the first two sentences of
section 5.6.2.3 to the following:
Voting system application
software shall be designed in a modular or object oriented fashion. COTS software components shall incorporate all
security patches released as of the time the software is built and shall comply
with the security requirements defined in the Software Security sections
(5.1.3.1 and 5.1.3.4). The vendor shall,
in compliance with the requirements of 5.1.3.1, document how each COTS
component has been defended against the threats identified in 5.1.2.3 (A-1),
(A-3), (B-1) and (B-2), such as by testing, external controls, et cetera. COTS systems or components shall be
documented by their suppliers to have been tested to at least the same rigor as
required of voting devices as specified in this Standard. The COTS supplier, or the vendor using the
COTS component in their product, shall deliver documentation of this
testing. The said documentation may
include formal certification (such as Common Criteria) or other recognized
independent evaluation at levels appropriate to the voting system evaluation. The COTS systems or components either shall not
be changed from documented, certified, tested versions, or, if a software
component, shall be able to be built by the ITA from provided COTS modules. COTS components not meeting these criteria
shall be tested in a like manner to any other component.
--- Corry-139
Comment: Systems that are
simply cobbled together (kluge might be a better description) from COTS
components must not be exempted from environmental testing. I've had too many
problems with little doohickies hung on some piece of
otherwise great equipment that caused problems when fielded.
Proposed change: Delete first sentence of second paragraph.
Proposed Resolution:
In section 6.4.4.1, delete the
first sentence of the second paragraph.
--- schneidewind
– 006 and Lipsio-4B
Comments: Why exempt COTS
hardware from environmental testing? COTS
hardware must have been tested to the rigor required of non-COTS components; if
the supplier has not done this, then COTS hardware must be treated like any
other component.
Proposed change: Require environmental testing of COTS hardware. Change paragraph to “COTS systems or
components must be documented by their suppliers to have been tested to at
least the same rigor as required of voting devices as specified hereinbelow;
else, the said COTS components shall be tested in a like manner to any other
component.”
Discussion: All agreed; incorporated some re-writing and
extra verbiage by two of those who commented.
Proposed Resolution:
Replace the first sentence of the
second paragraph of section 6.4.4.1 with: “Systems designed exclusively
with system-level COTS hardware whose manufacturer can document that the system
has been tested to at least the same rigor as required of voting
devices as specified in this Standard and whose configuration has not
been modified in any manner need not be subjected to this segment of hardware
testing.”
--- Lipsio-4D, MercuriD50 – 022, schneidewind
- 007
Comments: “Unmodified, general
purpose COTS non-voting software ... is not subject to code examination ... is
not subject to the full code review and testing” is contrary to such other
mission-critical methodologies as those used by the FDA and FAA, and
contradicts what is specified in section 5.1.3.3.2.”; “The decision by the FEC
to exempt COTS products from inspection has created a serious security
flaw. It should not be imperative that
the IEEE standard continue to reflect this inappropriate practice. All exemptions for COTS product review should
be removed from this standard.”; COTS software must
work in conjunction with the voting application software. Therefore, it should
be subjected to the same rigor of testing as the application software.
Proposed change: Eliminate the sections; ensure
compliance with section 4.3.11 (“Previously developed or purchased software”)
of IEEE Std 1228-1994, “IEEE Standard for Software Safety Plans”; “Remove all
exemptions for COTS product review from this standard on the grounds that such
pose a serious security flaw. COTS
products shall be presented in their entirety for open review in the same way
that vendor software is examined.”; Eliminate the
exemption of COTS software from the testing requirement.
Discussion: Bob Oliver disagreed; he and others have not
seen IEEE Std 1228-1994, so the resolution is pending
upon their examination of that standard
Proposed Resolution:
Eliminate paragraphs 2, 3, and 4
of section 6.6.2, replacing them with: ‘COTS software components shall comply
with section 4.3.11 (“Previously developed or purchased software”) of IEEE Std
1228-1994, “IEEE Standard for Software Safety Plans”’.
--- Dill-35
Comment: If COTS hardware or
software is in the trusted subset, it must be treated exactly like software or
hardware designed by the vendor.
Proposed change: Specify that the COTS exclusion only applies to system components
outside the trusted subset.
Proposed Resolution:
Irrelevant
in light of resolution proposed for Lipsio-4D, MercuriD50 – 022, and schneidewind - 007