Toward GitOps with Kubernetes

[Freshworks engineers share their #KubernetesLearnings to spare you the horror stories. This is the fifth blog in the series.]

One of the fascinating things that Kubernetes enabled us to do was move fast with our releases. Also, the declarative model of deployments in Kubernetes helped us to code, and, hence, version on git. In this blog, we tell you the story of where we were and how we got to very modular GitOps deployments.

Chef and OpsWorks

Early on, all our infra was hosted on OpsWorks, an AWS service that allows you to define your infrastructure as stacks. Every stack was divided into multiple layers and deployments were orchestrated using Chef cookbooks (or recipes). For now, we will not talk much about how our stacks are oriented—which would make for another blog—but we want to touch on how we get our release onto those stacks.

We used to, and still, follow a blue-green model of deployment. We would have a blue stack and a green stack on OpsWorks. Assuming blue is the current live color, to do a release we would bring up the green stack with this new release. We would then selectively switch traffic to the green stack using HAProxy reconfiguration until 100% traffic is on green.

Here is how our blue-green shell was oriented on OpsWorks.

Most of the heavy lifting was done by Chef recipes during the deployments and managed by OpsWorks. Coordination of the releases and traffic management were handled by an in-house deployment automation tool (DAT).

This diagram shows the workflow for the OpsWorks-based deployments.

Enter Docker

The world was moving to Docker and we did not want to be left behind. But we didn’t jump on the bandwagon just because of peer pressure. We saw what problems Docker was capable of solving and it made sense for us to shift to it.

One deployable artifact

One of the biggest benefits of moving to Docker is the ability to have one uniform deployable artifact (earlier, the application and its dependencies could not be packed together as a single image). This allowed us to lessen our dependency on the Chef cookbooks and, hence, reduce errors introduced by recipe changes. One other benefit was that we could move the underlying systems independently.

Tighter security

One of our early decisions with the move toward Docker was to make the app container environment less prone to external offenses. We ensured that our Docker container ran with tighter security constraints:

  • The root file system was read-only,
  • The container always ran in unprivileged mode (non-root).

Configs to move with code

In OpsWorks, we had all our configurations coming from the OpsWorks Settings’ Custom JSON, and Chef cookbooks would configure all the application YAMLs. When we moved to Docker, we moved all these configurations as part of our application codebase and stored them as EJSON files, which we would load into the YAMLs in the entry script of the App Containers.

This was a significant change because it almost nullified our dependency on the Chef recipes and, hence, on OpsWorks as well. Now we were truly ready to bid adieu to OpsWorks and embrace any new Docker orchestration framework, and that orchestration framework was obvious.

Kubernetes and GitOps

Kuberenetes did not just mean writing a huge deployment.yaml to declare our stack structure and doing a kubectl apply. We took a structured and modular approach to our deployment templates so we could build from a base more reminiscent of our OpsWorks stack structure. We did this for the following reasons:

  • Move to Kubernetes from OpsWorks would at one point be a hybrid infra (OpsWorks and Kubernetes),
  • HAProxy (still on OpsWorks) backends configuration need not change with our move to Kubernetes.

Kusmotize modularity

We used Kustomize to organize our deployment templates in an extensible and modular fashion. The deployments were organized into folder structures with a Kustomize way of inheriting from the bases. This allowed us to define different environments (dev, staging, prod, etc.) or regions (US-East-1, EU-Central-1), inheriting from a base definition, and applying patches on top of those base definitions. This is one structure we follow for defining our templates:

A couple of things to note in this structure is that everything defined in deployments/base/shell is inherited across environments (or regions). Also notice that some environments (or regions) have their own bases, which both inherit from the deployments/base/shell as well as have their own specific definitions. So bottom level kustomization.yaml (say, deployments/staging/stagingvpc-shell-main/blue) would look something like this:

From the deployment repo structure, you would notice that for every environment/region we would have blue and green. And yes, we still do blue-green deployments, which is what we are going to discuss in the GitOps section below.

GitOps Deploys

The entire deployment structure that we discussed above is maintained in Git repository. At the top level, we have Makefile, which defines calls into Kustomize when doing something like the following:

The above translates into the following Kustomize build command:

Since we have complex deployments with multiple shells, we still fall back on deployment automation tool for our deployment workflows, and DAT follows GitOps deployments. Here are the steps on how it does that for a single shell deployment on staging:

1. We branch off from the mainline into the release branch

2. Choose the non-live color, say, green, and make changes:

a. kustomization.yaml with release tags

b. replicas.yaml with number of replicas per deployment

3. Commit these change into release branch

4. Call make on the green folder.

Since all of the above steps are orchestrated through DAT, the tool also ensures that the traffic switch to the green stack is also initiated.

Keeping the current deploy state in Git (branch) ensures that the current state of the infrastructure is always traceable to the branch in our repository. Rollbacks would merely be a two-step process:

1. Check out the previous release branch from the git repo, and

2. Apply it to Kubernetes, which, too, currently is handled by DAT.

Here is a diagram of the deployment workflow.

In pursuit of a pure model

Though this is not a purist model of the GitOps, it does signify an important move toward having versioned application infrastructure. Having said that, our pursuit is to move toward a pure GitOps model. But the current iterations are defined by our needs to solve a problem, and right now this is working pretty well for us.