Or, maybe it is. In reality, the answer is that all automation is scripting but not all scripting is automation. Automation is really a maturation or evolution of scripting. Calculators script the mathematical principles defined by Thales, Pythagoras, Euclid and Archimedes. To-do applications script the act of making a list of tasks on a piece of paper and scratching them off. The directions given by Google Maps on a road trip script the job normally performed by the person with a paper map sitting in the passenger seat.
Automation is Protective
Scripts become automation when they become complex, resilient and robust. Consider the script:
if dark:
lights = on
else:
lights = off
This seems like a perfectly reasonable script, but it’s not automation. There are many situations in which this script would be ineffective. What if it’s time for bed? What if there’s a storm outside? What if there’s a firework display outside? What if there is no one home? What if you live in the Arctic? What if…, what if…, what if…? It is with the elegant, comprehensive handling of the what ifs that scripting graduates to automation.
Take a tool such as Kubernetes (https://kubernetes.io/). This is an “open-source system for automating deployment, scaling and management of containerized application”. Are the 66,520 commits, 4,366,875 lines of code, and 1,693 committers merely contributing to the creation of a script?
In the Common Criteria context, this is all too relevant. Take this simple test case from NDcPP:
Test 1: The evaluator follows the guidance documentation to configure a notice and consent warning message. The evaluator shall then, for each method of access specified in the TSS, establish a session with the TOE. The evaluator shall verify that the notice and consent warning message is displayed in each instance.
At first blush, a solution to this could be:
login to interface set banner log out of interface log into interface and check banner
A perfectly reasonable script for achieving the goal of the test, but far from something that could be considered valuable automation. There is an onslaught of what ifs:
What if:
- the interface is broken
- I can’t set the banner
- I can’t logout of the interface
- I can’t log back into the interface
- there are 17 different interfaces on this particular TOE
- I need to test the combination of setting the banner in each of the 17 interfaces
- …
The list goes on. The proper, consistent handling of the what ifs is where a script becomes automation.
Automation is Repeatable and Repetitive
Secondly, automation is meant to be run constantly and continuously. The script above could be expected to be run every picosecond, but that seems wasteful and unproductive. Automation on the other hand thrives in such an environment. It is also in that environment that most badly implemented automation falls over. Anyone who has written any automation can attest to the experience of writing something that they feel is well designed, only to realize that once it’s run every minute for a week by 50 other individuals, it suddenly it begins to crumble. Where scripts do not scale, automation does.
In our push to automate the prescriptive testing of CC, we want to be able to run the complex automation ad nauseam with the same result each time. In fact, as development spin builds, CC automation runs should occur in lockstep. An amount of fuzzing and randomness is also tremendously valuable in this case. It enables a breadth of testing that will be more likely to catch those edge cases as opposed to a script that is always testing that same set of data.
Automation is Orchestration
Automation also often involves the orchestration of more than one sub-system. Scripts are usually confined to a single node. Using the example above, that script might be a part of your home automation system which also ensures the sprinklers are off, the doors are locked, the security system is enabled, the furnace is set to an appropriate level, the garage door is closed, etc…
With NDcPP, this could be an LDAP server, a logging backend or a TLS client or server. Each of these need to be in place and controlled for the evolution of script to automation.
Automation at Lightship
At Lightship Security, we’ve designed our Test Automation Platform – Greenlight to deliver a system that doesn’t just script protection profile test activities, but actually automates them. This automation is complex, comprehensive, scalable and results in massive time savings while reducing the what ifs of a traditional CC evaluation.
About the Author
Alex Thurston is a Senior Developer at Lightship Security and team lead for Greenlight.
Alex Thurston
Alex Thurston is Lightship’s lead developer. While relatively new to the certifications space, he has been involved in software development for over 10+ years. He loves all aspects of software development, loves to find modern approaches through code, and hates doing the same thing more than once.