Undertaking a Common Criteria (CC) evaluation should not be an opaque process from a timing, process or cost perspective. In this post, the testing experts at Lightship provide 6 practical tips to ensure that you are getting the best value and outcomes for your certification dollar. The following is targeted primarily at Protection Profile (PP) based evaluations, but most also apply to Security Target (EAL) based projects.
1. Ask your lab before contracting if you, as the vendor, will need to develop any test harnesses or use your own resources do any of the testing and show the results. With few exceptions, your lab should be able to do close to 100% of the Protection Profile testing with their processes and tools. They should not put the onus on your team to do their work. Set expectations in advance on your team’s required level of involvement to avoid surprises.
2. Confirm how much pricing contingency the lab is building into their model for testing. Historically, labs did not know how many rounds of testing they would need to do, because the testing was done at the end of an evaluation project with little advance insight into any possible issues. This resulted in labs building in a significant risk premium. If you, as a vendor, undertake a Functional Gap Assessment approach to ensure product readiness before formal testing, you can confidently enter into a contract with a lab that only includes one full pass of testing. Don’t pay for unnecessary testing cycles.
3. Did you know that labs must now do an Entropy Analysis for most CC evaluations? Check with your lab to ensure they can work with you to extract the necessary samples and do the required qualitative or quantitative detailed analysis for the target Certification Scheme. This should be done early in the process to prevent any surprises or non-conformities that may require fixes later in the process.
4. Ask your lab how they do their gap analysis. If it’s a paper-based exercise or checklist, be aware that the process will likely miss granular details that may end up costing re-development cycles and slow your time to market. A lab that relies solely on a paper-based gap analysis may only uncover additional problems during the official testing phase, at which point you are forced to remediate the problem. In our experience, the best way to determine gaps is to execute actual test cases against the target early and often to dramatically reduce re-development risk.
5. Confirm with your lab how long the whole process will take end to end before signing a contract. Be wary of large ranges of time. Armed with the results of a Functional Gap Assessment, the lab should be able to commit to a fairly specific duration for testing and finalization. There are some caveats around specific CC-scheme policies, such as the US scheme requiring last minute technical interpretations or requirements to be applicable right up until submission. In Lightship’s experience, a standard NDcPP formal evaluation can be completed in 60 days or less if the lab has the FGA results as inputs and is able to be “one and done” with formal testing. Don’t agree to an extended multi month process without understanding why it will take so long and slow your time to market.
6. Check with your lab on the ownership of the project deliverables. Be wary of labs that don’t provide the consulting or documentation deliverables as works for hire. You have paid for the work and should have ownership of the documents for future use with that lab or another of your choosing.
Armed with this information, you can negotiate the schedule, price and outcomes of your next CC evaluation with confidence. We hope it helps.
Jason has been involved in the leadership of different cyber security companies, including being responsible for the accreditation, management and profitable growth of several government-accredited IT security laboratories. Jason drives the Lightship vision of modernizing the product certification landscape.