Vendor Truth Serum

Long Term Goals

Enable DHS, other agencies, and ultimately the private sector share information on software engineering tool capabilities in order to achieve the goal of protecting America’s software-based infrastructure.

Ultimately, it would be valuable to determine what the cost to government is of not sharing results of testing. Another economic path to examine would be to examine the soundness of a particular tool and to estimate the impact of that tool’s failure to appropriately secure, verify, or validate the software under test.

Background for Long Term Goals

Software engineering tools are an important technology for building secure software and cyber-physical systems. There is a class of tools targeting software security, stability, and correctness. Experience is that any single tool covers a relatively small percentage of potential errors. A mix of tools should increase coverage. One needs to know what the tools cover in order to make a rational decision of which tools a software engineering project needs. Likewise, one needs to know where the vulnerabilities are that no tools can check. For example, not one tool on the market detected the Heartbleed OpenSSL bug.[i]

In order to understand the capabilities of a suite of tools, enterprises undertake in-house product testing. For example, the Department of Homeland Security Science and Technology Division (DHS S&T) has undertaken making tools available, particularly for its SWAMP (Software Assurance MarketPlace) project. However, DHS is contractually prohibited from publicizing what the tool capabilities are. This means that even though DHS may know the capabilities of a tool, other departments and agencies in the U.S. government have to go through the expense of testing all of the extant tools to learn what tools a given project provide the requisite coverage. DHS, NIST, NSA, IARPA all have tool testing initiatives, driven by the fact that when one tests a tool, they cannot share the results with other agencies. This situation is even worse for the private sector, which DHS has the mandate to help protect.

Angles of research include investigating whether software engineering tools come under the False Claims Act[ii] and whether the GSA would need to modify the Federal Acquisition Regulations to mandate information sharing.

Intermediate Term Objectives

Enable the Federal government to reduce the cost and increase the reliability of software systems by eliminating or reducing duplicative tool testing and by improving the state of the software engineering tools marketplace by sharing results of tool testing.

Schedule of Major Steps:

Assemble an electronic collection of government software contracts from DHS and elsewhere [2nd month]
Reviewing such contracts for relevant non-disclosure terms and generating a survey of terms [3rd month]
Investigating whether such terms are common in non-government software contracts [3rd month]
Researching relevant case law as to the terms' enforceability [4th month]
Gather documentation on the soundness of static code analysis tools published by vendors [2nd month]
Validate if results from tool study correlate with data published by vendors about tools [4th month]
Host half-day workshop, including representatives from DHS, GSA, government contracting officers, and representatives of Federal enterprise software engineering users. Agenda: present the results of the research, learn from participants their experience with these terms, and discuss possible approaches [5th month or good date]
Final report based on collected research and results of workshop [2 months after workshop]


[i] Kupsch, J.A., & Miller, B.P., Why Do Software Assurance Tools Have Problems Finding Bugs Like Heartbleed?, Continuous Software Assurance Marketplace, 22 Apr. 2014,

[ii] 31 U.S.C. §§ 3729 - 3722