Comparing CAVP and ACVP – Test Harness Implications

As the legacy CAVP algorithm testing program is put to pasture and the newer automated testing program – ACVP takes its place, there are several questions, concerns and design considerations to think about. Foremost among them is how testing will work going forward and what it means for vendors and their existing implemented algorithm test harnesses.

One of the questions that often comes up here at Lightship is how does an existing test harness need to change to support the JSON formatted ACVP test vector sets and also to produce the needed response files. Because many vendors have an existing test harness based on the legacy program, they now have to redesign their harness for this new program which becomes mandatory on July 1, 2020.

Despite there being significant changes to the containers in which the test data resides from CAVP to ACVP, it is often the case that aside from using a structured format (JSON), the contents remain the same.

Let’s take a look at a simple example: AES ECB.

In the current CAVP program, a registration of support for AES ECB in a single direction (encrypt) and single size (128) would result in 6 separate vector set files:

Each of these files (which we might also describe as test groups) contain a list of key/value pairs used as inputs to drive the tests with a COUNT (which could be called a test case ID) associated with it. Depending on the advertised support, the number of test cases within each file will vary. In our example case, the break down looks as follows:

• GFSbox128: 7 tests
• KeySBox128: 21 tests
• MCT128: 1 tests
• MMT128: 10 tests
• VarKey128: 128 tests
• VarText128: 128 tests

If we now take a look at the exact same algorithm registration but with ACVP, we are give a JSON structure that appears as follows:

{
  "vsId": 1,
  "status": "",
  "message": "",
  "algorithm": "ACVP-AES-CBC",
  "isSample": true,
  "mode": "",
  "revision": "1.0",
  "testGroups": [...]
}

This is the header information associated with the vector set. We will focus on the testGroups section.

Each test group is made up of a key tgId or test group ID (wait … we’ve used that term before) , some additional meta information about the test group nature, and an array of objects beneath the tests key. The meta data within the test group object identifies what type of test it is. Each item within the tests array contains test data and a key called tcId or test case ID (wait again, I’ve seen that term before as well). Already you can sense some of the similarities to the structure described above.

Looking at the contents what we end up having is the following structure:

As you can see, once the structure hierarchy has been parsed and understood, the underlying tests follow the same pattern as before. In your test harness, once the surrounding structures are peeled away, you are left with the same iteration of calls into the Implementation Under Test (IUT) for tests.

Conclusion

While the process of migrating to ACVP might seem daunting and the CAVP swan-song seem saddening, neither should be. Yes, the formats and arrangements have changed, but the underlying requirements of your test harness have not drastically changed.

We cannot emphasis enough that this simplistic example is not meant to support a decision to build a converter. There are a number of nuances and aspects to the CAVS and ACVP formats that are not demonstrated by this example. The short term bandaid on the bullet hole will be more time that it’s worth over the long term. ACVP will provide a much more extensible and thorough method of validating your cryptographic algorithms going forward and the investment in properly enhancing your harness to accept ACVP is the sound engineering solution.

As always, if you have any questions or require help with your ACVP migration or FIPS 140-2/3 needs, feel free to reach out to our testing experts and we’d be happy to help. Happy algorithm testing.