A Look at NIST 800-204D: Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines

At this point, the name Solarwinds has become almost synonymous with the idea of a Software Supply Chain (SSC) attack. There is a huge amount of information surrounding this particular attack and I don't want to repeat it all here, so suffice it to say that threat actors infiltrated the SSC of a company called Solarwinds and inserted a backdoor into the very product that Solarwinds sells, and that product was then distributed to many of Solarwinds' customers. Not cool. Not cool at all.

But perhaps some good will come from it, because from then on, SSC attacks have been embedded in the consciousness of most cybersecurity practitioners. And now, at this point, a few years after this particular attack, we have some recommendations from NIST on how to secure the systems that build and update software products, in the hope that if these recommendations are followed, these systems will not be so easily subverted--as was the case in the Solarwinds and other similar attacks.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-204D.pdf

The document is easy to read in one sitting, and below I pull out some interesting ideas and recommendations. I also pull out all the control recommendations made in the paper.

Improper Repository Configuration

Computing is becoming increasingly complex. Even managing a source code repository is a complicated and difficult process, with more and more configuration options that need to be understood in order to use them safely. Furthermore, in many situations, organisations will be using SaaS-based source code repositories such as Github, which are constantly changing, mostly for the better, but also growing in complexity, with an ever-increasing number of configuration options, buttons and choices to be made, most of which affect the security of the repository.

In response to the difficulty of configuration, the field of 'posture management' has grown considerably over the past few years and is applied to all kinds of systems, from 'generic cloud' to Kubernetes and software-as-a-service tools such as Github.

Many security issues with source code will be due to repository misconfiguration. We see this later in the recommendations section.

Completely Separate Deployment Automation

It makes a lot of sense to me: the system that builds and integrates the code should not be the same as the one that delivers it to production. As an industry we understand segmentation quite well, perhaps too well, but here it would make sense to separate the two processes.

Forking

For whatever reason, I find the section in the document on "forking," especially with the scare quotes, vaguely humorous. I mean, this is how pull requests work - this is basically how software is developed now. There's nothing malicious about forking in and of its own. Bad actors didn't invent forking as some kind of new way to exploit Git repositories.

That said, the process of submitting code is a complex one, and something that can be social-engineered. As well, often tests are run on any code submission to check if the patch is valid, and that testing process can often be subverted in unexpected ways. Usually this possibility is based on placing too much trust in who can submit a patch and what level of testing is run on that patch by default.

Wrapping CI/CD Actions With Witness

No matter how many layers we build into our systems, no matter how big and complicated our "security onion" is, someone will eventually cut through those layers. The question then becomes how do we tell the story of what happened? Do we have some kind of forensic evidence that we can analyze to determine exactly what occured? We all know that those questions are going to come from management after the attack...What happened? What did they have access to? Did they ex-filtrate data? Etc. Etc.

If we wrap all the work that we do in pipelines with some kind of forensic tool, then at least we have the possibility of answering those questions.

The document specifically recommends a tool called Witness, which provides the following capabilities:

Attests - Witness is a dynamic CLI tool that integrates into pipelines and infrastructure to create an audit trail for your software's entire journey through the software development lifecycle (SDLC) using the in-toto specification.

Verifies - Witness also features its own policy engine with embedded support for OPA Rego, so you can ensure that your software was handled safely from source to deployment.

Obviously implementing this properly would be a considerable undertaking.

Threats to Software Update Systems

Again this reminds one of the Solarwinds attack. Not only do we have to build software safely but it also has to be distributed in a secure fashion, which is much more difficult than it sounds, and as the paper suggests, simply validating hashes is not enough. Most organizations will have to either buy or build their own updating mechanism.

Requirements

The document also makes several "REQ" recommendations for controls around the security of the software development lifecycle. The following are taken verbatim from the document.

PULL-PUSH Operations on Repositories

• PULL-PUSH_REQ-1: Project maintainers should run automated checks on all artifacts covered in the change being pushed, including unit tests, linters, integrity tests, security checks, and more.
• PULL-PUSH-REQ-2: CI pipelines should only be run using tools when confidence is established in the trustworthiness of the source-code origin of those tools.
• PULL-PUSH-REQ-3: The repository or source-code management system should run CI workflows in sandboxed environments or have built-in protection that incorporates a delay in CI workflow runs until approved by a maintainer with write access.
• PULL-PUSH_REQ-4: If no built-in protections are available, external security tools are required to evaluate and enhance the security posture of the SCM systems, detect and remediate misconfigurations, security vulnerabilities, and compliance issues.

Secure Code Commits

• COMMIT-REQ-1: Evaluate committed code for adherence to organizational policy, including the absence of secrets such as keys and API tokens. Detected secrets should be prominently displayed, and appropriate alerts generated upon detection of policy violations.
• COMMIT-REQ-2: Push protection features should be enabled for all repositories, including verification of developer identity/authorization, enforcement of developer signing of code commits, and file name verification.

Securing Workflows in CD Pipelines

• DEPLOY-REQ-1: A security scanning sub-feature should be invoked to detect the presence of secrets in the code, such as keys and access tokens.
• DEPLOY-REQ-2: Before merging pull requests, it should be possible to view details of any vulnerable versions through a form of dependency review.
• DEPLOY-REQ-3: Specify that the artifact being deployed must have been generated by a secure build process to be cleared for deployment.
• DEPLOY_REQ-4: Check for evidence that the container image was scanned for vulnerabilities and attested for vulnerability findings.
• DEPLOY-REQ-5: Periodically check release build scripts for malicious code, including scanning a container image for vulnerabilities as soon as it is built and ensuring tools used to interact with repositories are capable of integration with CD tools.

Secure CD Pipeline — Case Study (GitOps)

• GitOps-REQ-1: The process should rely on automation rather than manual operations.
• GitOps-REQ-2: Package managers should preserve all data on the packages that were released.
• GitOps-REQ-3: Changes should not be manually applied at runtime; instead, changes should be made to the relevant code and a new release triggered.
• GitOps-REQ-4: The Git repository, containing application definitions and configuration as code, should be pulled automatically and compared with the specified state of these configurations.

Conclusion

In terms of the SSC--a lot of this we know instinctively now and the area is starting to become embedded in the ecosystem and the cybersecurity mentality, at the very least the fear part of it. The fact that this document isn't hundreds of pages long is an advantage in that you can actually sit down and read it in a single session and then start to think about how much of it has been applied to the systems that are in place for managing, building and updating code and applications. The subtlety of much of this is difficult to see at first glance, but once we've started to get into the weeds of how it all works... the complexity can become staggering.