How we manage 100+ Web Application Firewalls

This is about how we manage large numbers of ModSecurity WAF deployments, including CRS and custom rules.

ModSecurity CRS Berne Meetup

At VSHN we’re not only 100% committed to Free and Open Source software, we also collaborate actively with the communities of the software we use, deploy and enhance. This June, we had the pleasure to join the OWASP ModSecurity Core Rule Set Berne Meetup group online who had asked us back in November at yet another Meetup to showcase the way we use ModSecurity and the Core Rule Set (CRS).

ModSecurity logo
OWASP ModSecurity CRS

Motivation

As a Swiss company, we have customers that care deeply about application security, typically because they operate in the financial, insurance or legal sector. We manage different WAF brands for them, ModSecurity being the only 100% free and open source solution among those. To make deployments, updating of the WAF engine and the deployed CRS, as well as tuning their custom rules manageable, we need to do a great deal of automation. Our talk explained how we can serve several customers that operate tens to hundreds of WAFs, requiring their developers and development agencies to protect all public-facing applications by a Web Application Firewall (WAF).

A lot to do, cloud native, in the cloud

When you deploy hundreds of first-line defense appliances there is no point in trying to keep things up-to-date manually. There is too much to take care of, too much to overlook. Deployment configuration (Apache or Nginx .conf files, Kubernetes manifests), the ModSecurity container image itself, the Core Rule Set configuration files, plus allowing developers to include their own custom rules for convenient independence. This process must scale, so we automate on all ends.

ModSec CRS rules
Working with ModSecurity rules
  1. We use the official ModSecurity-CRS container image (the one that we volunteered for taking over maintenance almost a year ago). It now provides a myriad of environment variables that allows for cloud native-style configuration at startup time. A matching version of the CRS is included.
  2. Custom rules we allow to include flexibly as .conf files, which are mounted into the running container at startup time. This matches the process suggested by Christian Folini’s ModSecurity CRS tutorials, this time applied to a WAF running as a containerized application.
  3. To glue all this together, we add a bunch of Kubernetes manifests (.yaml files) to the equation. This way, we pass the responsibility of deploying and operating our WAFs on to Kubernetes, which is the „engine“ running in the cloud infrastructure we manage on behalf of our customers.

Configuration automation

It’s important to note that by this clean separation we can focus on rolling out configuration only. No images are built on the fly for updates. The Kubernetes (OpenShift) cluster pulls-in the updated ModSecurity-CRS image when we update the related reference in the deployment configuration that define the WAF Pods. This was all just the configuration part. Next comes the actual automation.

  1. Each WAF instance is represented in code by a Git repository. This serves both for automation and traceability, which is important in validated environments. You may want to call this, „configuration as code“. We apply the needed configuration changes as Git commits. Commits that we apply in an automated fashion, of course. We use the ModuleSync tool for this purpose, which originally comes from the Puppet community.
  2. To make this scale, we literally wrap ModuleSync in a programming loop and add a set of configuration files („WAF configurations“), neatly organized in file system folders. We call this Concierge, „Configuration management + CI“. All this lives in a single repository. This is where we work for triggering changes on our hundreds of WAFs.

Configurations repository

Let’s take a quick look at the file organization of each of those configurations:

Cookiecutter logo
Delicious code by Cookiecutter
  • managed_modules.yml – contains the list of Git repositories that will receive the changes. They will be kept in sync.
  • moduleroot/ – contains all files that will be synced to the managed modules. Those files are actually templates, they can contain placeholder values and similar things that templates can do.
  • .sync.yml – when this is found in a target repository the variables therein influence the behavior of the sync process. For example, we use this to allow different CI workflows and environments for each WAF. Naturally, this file and also the rest of a new WAF repository comes from a blueprint, which we use as an input for yet another tool that helps us automate: Cookiecutter, a project generator from the Python community.

Now, we simply make a change to one or more of the files in the WAF Concierge repository and – boom! – the automation triggers: Concierge applies changes with ModuleSync to all target repositories, all the WAFs we manage. Whether there are ten, hundred or more of them. Nice!

Automatic merge requests

This is where the (automation) story would end. Because the rest we had to do manually: Open each of the target repositories, create a merge request from the changes that were applied in a dedicated Git branch, and merge this branch, which would allow GitLab to take care of the rest. Just a few clicks, but for quite a few repositories! Enter the main strength of Open Source software: Could we make ModuleSync do the creation of the merge requests for us, in an automated fashion? Of course we could! We could add this feature as a contribution. – And that’s exactly what we did with pull request #175 on the ModuleSync project. That’s why we love Open Source, and it shows we actually care to give back. Because so everyone can profit!

Focus on security

Automation is a powerful tool. In the context of ModSecurity, it allows us to focus on getting the „first line of defense“ working reliably, with configurations that work without getting in our customer’s way. So they can focus on their business. https://www.slideshare.net/bittner/managing-100-wafs