Securing the CI Irene Michlin, Principal Security Consultant

People, process, and technology Step 1: make sure your CI does not harm your security Step 2: only then it can be used to improve your security On people, I only have one piece of advice - Hire the right people and treat them right. The rest of the talk is on process and technology.

This is not “tools” talk The choice is enormous, not going to say “This is the recommended toolchain”. Not on the agenda, not on the hidden agenda. It’s rather what to look for in my tools, and how to put them together so that first of all I’m not harming my organisation’s security. And the secondary goal – I have this neat CI pipeline going, can I use it to improve my software’s security, amongst other things. Image from https://www.blazemeter.com/blog/ultimate-devops-tools-ecosystem-tutorial-part-1

Pull latest code + tests Basic CI cycle Feedback Dashboard or gadget Generate feedback Local build + test run Development machine Version Control repo CI Server Pull latest code + tests Developer Poll Write code + tests Commit To ensure we are on the same page for the rest of the talk.

Isolate your environment Phishing link in email => keylogger installed => source code gone (or backdoor deployed) Experimenting with development network => accounting department affected before EOY Extra challenges: remote work or BYOD What happens in dev, stays in dev Physical or virtual, dev machines must be segregated from the main corporate environment.

Version control server It has one job only – remove or disable everything else No shared or generic accounts Matching business process to close accounts SSO is better than proliferation of accounts per each tool. RBAC with as much granularity as does not interfere with your work. Needs administrative component to terminate the accounts as required.

Integration build server Who is responsible for keeping it up to date? Where do external components come from? Check vendor advice on compiler and linker options It only needs R/O access to source control Set local repo for external libraries.

Feedback mechanism IoT electronic toys are notoriously insecure Custom integration scripts - are you cutting corners? Who is listening? Unless it is a low-tech mechanism “All the developers are in the same room and they look at a physical dashboard when they feel like it”, you have to consider the security of the mechanism. The tools are held together with what is often called “glue skills” – the developers are putting together ad hoc scripts, punching holes in firewalls, hardcode passwords. Even if the script itself isn’t dangerous, if it’s world-writable but runs at high privilege, it has a high potential to become dangerous. My example is not an attack on a particular tool, literally I’ve searched for “broken build gadgets” and picked a first example.

Do no harm Do not acquire CI components « by accident » Not everything is secure out of the box Dormant account today is an attacker-controlled account tomorrow Glue skills are great to have in your team, but they can also be dangerous. Treat these scripts as first class citizens – under source control and under the same security standard as everything else. CI environments are rarely planned upfront. The tools are introduced one by one and sometimes people only learn about everything they have during an audit, which is a bit too late. Secure system lifecycle. The product choice and evaluation can be quick and informal, but it has to be a conscious choice. Installation and configuration – don’t enable features you don’t need Ensure account management lifecycle covers all the CI tools. And now we are to level 2 of this game: how to actually do good in security terms using CI.

From dormant to active Glue skills are great to have in your team, but they can also be dangerous. Treat these scripts as first class citizens – under source control and under the same security standard as everything else. CI environments are rarely planned upfront. The tools are introduced one by one and Secure system lifecycle. The product choice and evaluation can be quick and informal, but it has to be a conscious choice. Installation and configuration – don’t enable features you don’t need Ensure account management lifecycle covers all the CI tools. And now we are to level 2 of this game: how to actually do good in security terms using CI.

CI Maturity model The figure above is based on several popular maturity matrices for continuous integration, as well as NCC Group experience with variety of clients. Won’t have time to go through them all, only couple. There is now understanding in the software development industry that security features are not enough for building a secure product. To achieve an acceptable level of overall security, the product needs all the features to be secure: the whole attack surface of the product must be considered from a security point of view, not only features that are directly security related, such as login functionality. The same is true for securing continuous integration. In the matrix above, the dedicated Security row lists explicit security related activities that must happen at some point towards CI maturity. However, every cell on the other rows needs to be evaluated for the security of its tools and processes as well.

Code reviews No change too small Leave trivial checks to tools Not a separate task, but in DoD for each task Reject & rework is part of “normal” Frequently asked “are manual code reviews waste of time”? Not if done well. If your decision has only one possible outcome, then why bother with the decision? It’s waste of time. Another source of waste of time in review is “saving up” to end of week/end of iteration. It’s all about quick feedback.

Root-cause Analysis What happens to externally reported issues? The first security feedback to introduce What was missing in our CI process? => Improve People like to rush into bug bounty programs Don't trouble trouble until trouble troubles you

Chain of custody Can you trust your release notes? Has every “unit of work” in the release gone through all the checks? Was it modified since “time of check”? Whether you release each change individually in CD, or in batches, the same questions apply

“On commit” is great Automated coding standards checks Code complexity / code duplication Banned functions / APIs Dynamic analysis Static analysis Fuzzing https://www.nccgroup.trust/uk/our-research/securing-the-continuous-integration-process/ Don’t throw all the tools at the dev team at the same time. Fuzzers, static analysis, dynamic analysis all take time to learn and configure and initially will bring the productivity down, not up No two journeys towards maturity are the same. It may make sense for your organisation to take a practice from Intermediate or Advanced before doing everything in Beginner. It meant to inform, not to restrict. Hard to come with good questions on the spot, so I won’t force you. I’m here today, obviously, and if you have questions later, find me on twitter or by email.

Points of contact Irene Michlin Principal Security Consultant M: +44 (0) 7972 333 148 E: irene.michlin@nccgroup.trust T: @IreneMichlin