Customizing Kubernetes Applications with Kustomize

If you run your software on Kubernetes, you probably don’t deploy a new version to a cluster from your local machine. Instead, you would push your changes to a version-control system. This would start a pipeline which builds the code, run some tests, build a Docker image, and at some point deploys a new version to a cluster. Typically, it doesn’t deploy directly to production, but to stage, pre-prod, or test.

You probably don’t want to use the same configuration in test and production. I mean, they should mostly be the same if you goal is to have a test environment that is as close to production as possible, but maybe run less replicas of a pod, or choose lower resource requests and limits in test.

Kustomize helps you the generate variations of base yaml files by applying patches for each environment. the generated yaml can used by kubectl apply. Let’s see how this works.

# base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: price-service-deployment
  labels:
    app: price-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: price-service
  template:
    metadata:
      labels:
        app: price-service
    spec:
      containers:
      - name: price-service
        image: price-service:12.34

In a typical Kustomize project, I would put the yaml above in the base directory. Bases are a set of yaml files describing a workload, or a group of resources, which are customized with overlays. Overlays are patches to the base yaml files to create variants for test or production environments, for example.

# overlays/production/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: price-service
  name: price-service-deployment
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: price-service
        resources:
          limits:
            cpu: 2000m
            memory: 512Mi
          requests:
            cpu: 1000m
            memory: 512Mi

The snippet above shows a patch which would add resource requests/limits and increase the number of replicas of the Deployment defined in the base.

Additionally, there is a meta data file kustomization.yaml in each of the directories mentioned before. It specifies which files are included in a base or overlay, the namespace to use for all resources, merge strategies, and many other options.

# overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
bases:
- ../../base
patchesStrategicMerge:
- deployment.yaml

The file above specifies sets the namespace of all generated resources to production. Now we can run kustomize and generate a variety of the base for production.

$ kustomize build overlays/production
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: price-service
  name: price-service-deployment
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: price-service
  template:
    metadata:
      labels:
        app: price-service
    spec:
      containers:
      - image: price-service:12.34
        name: price-service
        resources:
          limits:
            cpu: 2000m
            memory: 512Mi
          requests:
            cpu: 1000m
            memory: 512Mi

The resulting yaml can be applied to a Kubernetes cluster and resources are created in a production namespace. Resource requests/limits were added and the number of replicas set to 5. Using the other overlay, we can generate a variety for the test namespace and apply it to the same or another cluster.

We are now able to generate variates of yaml files tailored for different environemnts πŸŽ‰. See the Kustomize documentation for more advanced examples, how to change the Docker image, and many more.

See below the directory structure of the Kustomize project.

$ tree price-service
price-service
β”œβ”€β”€ base
β”‚Β Β  β”œβ”€β”€ deployment.yaml
β”‚Β Β  └── kustomization.yaml
└── overlays
    β”œβ”€β”€ production
    β”‚Β Β  β”œβ”€β”€ deployment.yaml
    β”‚Β Β  └── kustomization.yaml
    └── test
        β”œβ”€β”€ deployment.yaml
        └── kustomization.yaml

The example being used in this post is on GitHub.