DevOps and Security

Software developers are “bonused and incentivized” for causing change. That’s how they make their money. Operations people inherit these changes and earn their money by maintaining stability.

As change is the natural enemy of stability, developers and operations are natural enemies. DevOps changes this by harmonizing and aligning their incentives.

Operations vs Developers

 But what about Security?

The recent “heartbleed” and “shellhock” exploits have shown that we’ve much to gain by speeding up our change-management, amplifying the feedback-loop and make Continuous Deployment a reality.

The most appealing arguments for marrying DevOps with Security would be that:

  • Security professionals are included into the discussion much earlier, and so get a chance to harden and analyze potential weaknesses. This allows them to continue to do their job in the ever decreasing windows of time before deployment (and not when it’s too late and the proverbial “fan cleaning” starts).
  • It gives an opportunity for Security to become a more audible voice within the R&D process and therefore improve the reliability of services.
  • Bring more awareness about Security to Developers and Operations and cross-pollinate ideas between them.
  • Many Security activities such as automated pen-testing, virtual patching and continuous vulnerability assessment, are already carried out with a DevOps mindset.

For those Security professionals still unsure if DevOps has a place in InfoSec, I urge you to take a look at the Netflix “Chaos Monkey“.

Netflix lives by the principle that you need to fail fast and early to get back on your feet and deliver corrections at a time when it’s least costly to do so.Chaos Monkey is a software which validates and audits the live system in a similar way a PenTester would try to break things before the bad guys will find these issues and compromise your system.

“Chaos Monkey is a service which runs in the Amazon Web Services (AWS) that seeks out Auto Scaling Groups (ASGs) and terminates instances (virtual machines) per group. The software design is flexible enough to work with other cloud providers or instance groupings and can be enhanced to add that support. The service has a configurable schedule that, by default, runs on non-holiday weekdays between 9am and 3pm. In most cases, we have designed our applications to continue working when an instance goes offline, but in those special cases that they don’t, we want to make sure there are people around to resolve and learn from any problems. With this in mind, Chaos Monkey only runs within a limited set of hours with the intent that engineers will be alert and able to respond.”

The idea that “If something hurts, then do it a lot” might sound like monkey business and isn’t at all intuitive. So let’s take a closer look.

Chaos Monkey is a forced multiplier which persistently finds weaknesses and exposes them – even more so in distributed systems. Forced multipliers can be both good and bad. If you have a strong system it sheds light on its stability and increases it’s value. If you have “A big ball of mud” however, well you will amplify that! Any mistakes and technical debts will be magnified. Hence fear of the Chaos Monkey would be a normal healthy response as it forces us to deal with technical debt first. The good news is that we need to address these weaknesses anyway. And better to set aside a clear budget of time now than when our system forces us.

Isn’t a magnification of weaknesses and bottlenecks exactly what we strive for, both in Operations and Security?

Many mutual goals and values

Security and DevOps do share common goals. Specifically all parties

  1. bring an appetite for tackling technical debt. We know that we need to first clear our past mistakes and technical debts before we can recover and improve our overall stability and security. Setting enough time for this is important when moving to DevOps
  2. share a love for instrumentation. Without measuring, monitoring we are blind and unable to improve.
  3. are obsessed with orchestration. Instead of moving a piece of code or a server move whole environments and instances of our infrastructure. orchestration allows us to eliminate the #1 fallacy of missing errors that make it past the test-phases because we can ensure that the test environment behaves the same as our live production environment.
  4. hate complexity. Complexity is the enemy of security and stability and therefore disliked by all of us. Reducing complexity is at the heart of DevOps as it is the only way to increase velocity and throughput

This model has worked so well, that meanwhile a whole range of monkeys (the Simian Army) has been unleashed (not only at Netflix) and are trying to wreak havoc on production systems:

  • Conformity Monkey: ensures certain things are in place or they will kill the instance.
  • Janitor Monkey: ensures that old systems are decommissioned in order to save costs
  • Security Monkey: targeting instances with automated tools or SQL injecting so that these issues can be immediately addressed
  • and many others …


Bringing Security and DevOps together reduces our attack surface. More importantly DevOps provides us with a way of making a better case for security to those in management who need to pay for it. Security becomes part of the design and sheds the image of only “restricting and costing money”. It’s in every security professionals interest to become part of your companies DevOps movement or maybe even spearhead it?

Further reading:

Valbonne Consulting provides Research & Consulting for emerging technologies in Internet/Web of Things (WoT/IoT/M2M) and Emerging-Tech. We specialise in decentralisation, security and privacy. We work across a variety of traditional industry verticals (Telecommunications, Automotive, Energy, ...). We support Open Source and technologies built on open standards.

Joachim Bauernberger

Passionate about Open Source, GNU/Linux and Security since 1996. I write about future technology and how to make R&D faster. Expatriate, Entrepreneur, Adventurer and Foodie, currently living near Nice, France.

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. I especially like the section on common goals and the use of various monkeys to uncover issues and address them.