How we built a composable plug-and-play configuration for our Kubernetes Jenkins Agents
Jenkins is a popular, extensible automation server that can be used as a simple Continuous Integration (CI) server or turned into a Continuous Delivery (CD) hub for any project. Jenkins is popular because it can efficiently distribute work across multiple machines, helping drive builds, tests, and deployments across multiple platforms (web, mobile, private/public cloud, etc.) faster. Like many fast-moving engineering teams, we use Jenkins at Freshworks to orchestrate our CI/CD practices.
In the Jenkins architecture, a Jenkins Agent performs the heavy lifting of the work distributed to multiple agents. There are many ways to provision a Jenkins Agent: bare metal machines, Virtual Machines (VMs), dynamic EC2 instances, Docker containers, or Kubernetes clusters.
We recently moved to a new model of dynamically provisioning a Jenkins Agent with a few simple lines of code in the Jenkins pipeline using a Jenkins Shared Library – Plug and Play (PnP) pod template configuration for Jenkins pipeline. This article shares our motivation and the approaches we considered before settling on this.
Context
The integration between Jenkins and Kubernetes clusters is pretty robust. The following advantages prompted us to fully migrate our CI/CD pipelines from a host (VM) based Agent to a Pod-based Agent.
- A dynamic, light-weight Jenkins agent from Kubernetes can be provisioned on-demand within a few seconds
- We can enjoy a fresh and fully reproducible Jenkins agent environment for every build
- The resource/cost savings offered by Kubernetes, when combined with Spot instances, are significant
While spot instances work great for dev builds, we also needed the reliability of On-Demand (OD) instances for production builds.
Problem Statement(s)
We initially used to configure our pod template via the Jenkins UI, which is usually available from the following URL – https://<jenkins-domain>/configureClouds/
However, with this approach, we observed a few problems over time.
- The Pod Template configuration was too complex to configure using the Jenkins UI: A single Pod Template could easily have more than 30 configurable properties. While this demonstrates the great flexibility Kubernetes offers, there needs to be a more reliable UX.
- Multiple Pod Templates are challenging to manage: As the need for jobs increases to build/deploy various services, the number of configured templates increases proportionately. The Jenkins Configuration UI displays multiple pages for this section, even with just a few templates. It quickly becomes challenging to find the right place to make changes.
- Managing Pod Template configurations across Jenkins instances: If Jenkins instances are put into different environments, it becomes a problem to configure these templates across instances. No one would enjoy copying pages worth of configurations from one Jenkins to another.
- Lack of Determinism: When a project needs to use a particular Jenkins Agent, say “Agent A,” the way to configure this is to use the label for the agent (see “Labels” in the picture above) as a reference in the pipeline configuration. There is however no way to know exactly what Agent A is constituted of, whether Agent A is the same today as it was yesterday, or if Agent A is different in any way from other Jenkins instances we have set up.
- Lack of Traceability: As a project evolves, what it requires from the Agent evolves too. For example, v1.0.0 of a project might only need a standard Agent, while v1.5.0 needs an Agent that includes a Memcached container for testing. Sometimes the docker images used in the Agent must also be upgraded. Requesting the Jenkins administrator to modify the Agent or create a new one sounds simple. Very soon, however, there is no traceability for such changes.
- Controlling pod configuration using automation: A UI is great for a one-time setup, but engineers who need to update the setup more often will start to require automation. One example was when we planned to introduce versioning for our base images that used specific deploy tools, and we couldn’t easily update the base image version through automation.
- Modular/customizable configuration: Even though you have ten jobs in your pipeline, they might all use the same pod configuration but may not require the same amount of resources. Our setup today forces all jobs to use the configuration of the most extensive job.
- Manual Errors: This is inevitable when the configuration could be more transparent, and one is always prone to copy-paste errors while configuring docker image URLs.
Approaches
When we sought to address this challenge, we explored a few alternate approaches – each with its merits and demerits.
Approach 1 – PodTemplate as Yaml
Following is an example pipeline referenced from the examples found with the Kubernetes Jenkins plugin.
podTemplate(yaml: '''
apiVersion: v1
kind: Pod
spec:
containers:
- name: docker
image: docker:19.03.1
command:
- sleep
args:
- 99d
env:
- name: DOCKER_HOST
value: tcp://localhost:2375
- name: docker-daemon
image: docker:19.03.1-dind
securityContext:
privileged: true
env:
- name: DOCKER_TLS_CERTDIR
value: ""
''') {
node(POD_LABEL) {
git 'https://github.com/jenkinsci/docker-jnlp-slave.git'
container('docker') {
sh 'docker version && DOCKER_BUILDKIT=1 docker build --progress plain
-t testing .'
}
}
}
In this simple example, the YAML to define the Pod Template is 20 lines of code, while the build logic is five lines.
In the real world, a practical YAML can easily exceed 50 lines and feel too long to belong in the pipeline code. To make this worse, you find yourself repeating this for similar projects.
Different organizations follow different ownership models. There isn’t a single answer to who shall be responsible for the project pipeline, specifically for the Pod Template piece in your pipeline code.
If individual project teams own the pipeline (as we do here at Freshworks), it is impossible to simultaneously perform changes for all projects. For example, when upgrading the docker image for a security patch is needed, immediate action must ideally be undertaken simultaneously for all projects. Imagine the effort required by a dedicated, centralized team to modify hundreds of projects effectively under pressure.
Approach 2 – Move Pod Template to Jenkins Shared Library
When one finds common things across the code for multiple Jenkins pipelines, it is natural to move them to a Jenkins Shared Library.
We prefer to store our PodTemplates in a Jenkins shared library and load them through a libraryResource for flexibility and maintainability across different development teams.
// load library via @Library or other method
pipeline {
agent {
kubernetes {
yaml libraryResource('node-10.yaml')
}
}
...
}
Unless we have a single container with all our job requirements installed, we would likely need to load multiple templates for each stage of our pipeline (multiple agent definitions) or pre-define one giant pod definition (which is not very flexible).
Approach 3 – PnP Pod Template configuration
We eventually wanted to explore a new solution based on the following requirements:
- As mentioned in Approach 1, a pod definition in YAML format is preferred
- Works for both scripted pipelines and declarative pipelines
- Flexibility to provide various kinds of Agents
- Easy to use within the pipeline code and hide the details within the Library
- Easy to develop the Library to support the increasing demands of Agent configurations
The approach we came up with was to define fragments of YAML, each for a specific requirement related to an Agent, and merge them to compose a complete YAML for the Pod.
For example, consider the following build/deploy requirements a project may have.
- An Agent may need to do a NodeJS or a Rails build, which now requires a resource file named ‘node’ to include a container with a ‘node’ docker image and another file, ‘rails’ to include a ‘rails’ docker image.
- An Agent may require a Memcached container, while another might require a docker container to build docker files.
- An Agent may require less or more computing power, mapping to resource files named ‘small’/ ‘large’/ ‘fast’ to request appropriate CPU/memory resources.
At a high level, we would like to be able to simply add these requirements together, leading to a composition that might look like, for example, ‘node+medium+memcached’, and then expect the Library to return a merged YAML for the expected agent configuration.
We can achieve this in two ways –
- Custom Parser to merge the YAMLs:
vars/k8sPodTemplate.groovy
def call(Map opts = [:]) {
String pod = opts.get('pod', 'node_base')
def configurations = pod.split('\\+|-').toList()
def templates = []
for (configuration in configurations) {
String yaml = libraryResource 'podtemplates/' + configuration + '.yaml'
templates.add(yaml)
}
def merger = new YamlMerger() // This can be your own YAML merger
def final_template = merger.merge(templates)
return final_template
}
2. Reuse the Kubernetes plugin classes to merge the YAMLs:
vars/k8sPodTemplate.groovy import org.csanchez.jenkins.plugins.kubernetes.PodTemplate import org.csanchez.jenkins.plugins.kubernetes.pod.yaml.Merge import static org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.combine import org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder import org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave import org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud import io.fabric8.kubernetes.api.model.Pod import io.fabric8.kubernetes.client.utils.Serialization def addYamlExt(String string) { return string.endsWith(".yaml") ? string : string + ".yaml" } def call(Map opts = [:]){ String podString = opts.get('pod', 'node_base') def yamlFiles = podString.split('\\+|-').toList() String parentYaml = addYamlExt(yamlFiles[0]) parentYaml = libraryResource('podtemplates/' + parentYaml) PodTemplate parent = new PodTemplate() parent.setYaml(parentYaml) for (i = 1; i < yamlFiles.size(); i++) { String childYaml = addYamlExt(yamlFiles[i]) PodTemplate child = new PodTemplate() child.setYaml(childYaml) child.setYamlMergeStrategy(new Merge()) child.setInheritFrom("parent") PodTemplate result = combine(parent, child) parent = result } KubernetesCloud cloud = Jenkins.get().getCloud("KubernetesRunner") KubernetesSlave ks = new KubernetesSlave.Builder().cloud(cloud).podTemplate(parent).build() Pod pod = new PodTemplateBuilder(parent, ks).build() String yaml = Serialization.asYaml(pod) return yaml }
In the case of the custom parser, we need to write a parser and maintain it, while in the case of reusing the Kubernetes Jenkins plugin class, we need to ensure that during plugin upgrades, merges don’t unexpectedly break.
With this approach, our Jenkinsfile would like the following:
pipeline {
agent {
kubernetes {
yaml k8sPodTemplate(pod:
'node_base+medium+dind+coverity+memcached')
}
}
stages {
...
} // stages
}
Reserved Instances
Now that we have this basic foundation, we can easily extend this by introducing tags to enable Reserved Instances for production builds alone.
if ( (env.BRANCH_NAME && env.BRANCH_NAME=='release') || (params.deployTo && params.deployTo == 'release')) { yamlFiles.add('ondemand') }
Summary
Modern engineering teams today rely heavily on shipping software continuously, and building and deploying automation is at the core of it all. We were very eager to jump on the Kubernetes bandwagon for its obvious efficiency and flexibility benefits. We also required our pipelines to be highly configurable because we run many microservices as a team across multiple environments.
Our evolution naturally led us to plug-and-play composable Pod Templates for configuring Jenkins Kubernetes Agents, making them much easier to maintain as an added benefit. The extensibility of this setup was quickly proven as we encountered emerging problems, like the need to adopt spot instances and reserved instances in certain environments. We expect to continue building on this abstraction for months and years at Freshworks as our teams happily churn out multiple releases daily.
References
https://github.com/icheko/jenkins-shared-lib/issues/2
Secondary Author: Satwik Hebbar, Senior Director of Engineering at Freshworks
