In an agile development model, software is developed in incremental, rapid cycles with the goal of continuous improvement, fast flaw remediation and improved customer experience. Agile models advocate adaptive planning, evolutionary development, early delivery, fast iterations, and rapid response to change.
Can modern product assurance programs be designed to keep pace with agile development? At Lightship Security, we think so. In fact, we’ve made it our mission: Certification at the speed of development. We believe that a shift to “agile assurance” is a necessary step towards restoring trust and credibility to the cyber supply chain.
In this post, we are concerned with options to improve assurance outcomes provided by internationally adopted product certification programs. As a company, our focus is on developing solutions that support a modernized approach to product certification. This includes moving our industry towards agile assurance using contextual automation and supporting methodologies – we want to disrupt product certification as we know it.
And what is the state of product certification today? Our observations:
- Slow moving standards. The pre-eminent standards for products certification today are Common Criteria (ISO 15408) for IT products and FIPS 140-2 (ISO 19790) for cryptographic modules. As with most standards, these are updated at a glacially slow pace. Little has changed since their inception in the 90’s (based on work from the 70’s and 80’s). Even so, they are the best standards that we have: Common Criteria in particular boasts 28 participating nations and a worldwide network of accredited laboratories – immense value that we, as an industry, must better tap into and leverage through a modernized approach to product certification.
- Little innovation. Even as industry and governments look for ways to improve standards and align them with today’s development methods and products, there remains very little focus on optimizing the “how” of evaluation and certification. Promising steps have been made with the introduction of collaborative Protection Profiles and Automated Cryptographic Validation Testing (ACVP). More innovation, especially on the test execution side is required for assurance to become agile.
- Static assurance. Today’s certifications apply to a specific version of a product. This point-in-time certification approach has been used for many years and is perhaps one of the most glaring problems of today’s product certification outcomes. The kinds of technologies that go through these certifications are being continually patched, updated and modified. There is currently no effective way to provide up-to-date certifications for deployed products that receive regular updates.
- Divergent requirements. Although mutual recognition agreements are in place, it is not uncommon for a vendor to have to perform multiple certifications or ‘top up’ evaluations to meet country-specific requirements due to a perceived lack of assurance or coverage from the original effort. Such divergence often comes down to an effort-vs-assurance trade off which can be addressed by automated solutions that allow us to increase the requirement set without increasing effort.
What can be done to move towards agile assurance? Our approach:
- Automate. As a test lab, we have been able to drastically reduce the time and effort required to perform complex security testing via automation. This will and must become the industry standard.
- Integrate. Integrate the assurance process with the development process – continuous testing and remediation against requirements using integrated automation tools that complement existing development and QA processes. We are working with our customers to embed Greenlight in these real-world environments – and are seeing promising results.
- Expand. Looking to the future, expand the role of automation by modernizing product certification processes:
– Machine readable requirements specifications;
– Automated evidence collection and submission; and
– Reduce reliance on documentation analysis and replace with automated alternatives (i.e. revisit the assurance paradigm in light of technology advances).
Even within the framework of existing standards and programs, we believe that automation and integration will result in better outcomes for all stakeholders:
- More Products. Increased throughput due to automation will result in more certified products in the marketplace for consumers.
- Faster Certification. Automation and integration will result in a faster and more consistent time-to-market for vendors along with related cost efficiencies.
- Better Assurance. Automation redraws what is possible in terms of the depth, breadth, complexity and repeatability of testing and analysis. Coupled with integration, the restraints of a labour-intensive point-in-time approach can be removed.
At Lightship Security, we have developed an extensible test automation platform to make our role as a Common Criteria test lab as efficient as possible. We believe that such an approach holds promise beyond this niche. Ultimately, automation will be needed to handle the complexities and sheer volume of security testing required for Internet of Things, Industrial Control Systems, smart medical, connected vehicles, etc. We look forward to moving product certification towards agile assurance and helping to restore trust and credibility to the cyber supply chain.
Lachlan has 20+ years of extensive product security certification experience, including roles as a government certifier, lab evaluator and vendor consultant. As the Director of Cyber Labs, Lachlan has overall responsibility for our Canadian and US Common Criteria laboratories.