Helm Versus Kustomize

| | |

Helm vesrus Kustomize

When Kubernetes first became popular, most engineers had to work with just native Kubernetes object manifests. Then when Helm arrived it became the de facto standard for everyone. It was the only real choice until Kubernetes (or “k8s” as it is commonly abbreviated) v1.14 came on the scene. 

It is human nature to examine and debate; for example, the ongoing vim vs emacs debate. Helm and Kustomize are similar enough to consider them related to each other. In many cases they are not adversarial, but instead they can even work together.

To compare the two, let’s first start with defining the following initial assumptions:

In computer science, declarative programming is a programming paradigm (a style of building the structure and elements of computer programs that expresses the logic of a computation without describing its control flow.)

  • Kubernetes is Complex: which means there are a lot of possible permutations to get from the start to the destination.
  • Kubernetes is Declarative: which means that given the final manifest kubernetes will determine how to get the cluster to the point where it matches your wants. Declarative meaning the config is unambiguous, deterministic and not system dependent.

In computer science, imperative programming is a programming paradigm that uses statements that change a program’s state. In much the same way that the imperative mode in natural languages expresses commands, an imperative program consists of commands for the computer to perform.

Tool Introductions

Helm and Kustomize are tools for generating deployable manifests for Kubernetes objects, which philosophically takes the task of generating the final manifests in two distinct forms.

Helm3 is an imperative templating tool for managing Kubernetes packages called charts. Charts are a templated version of your yaml manifests with a subset of Go Templating mixed throughout, as well it is a package manager for kubernetes that can package, configure, and deploy/apply the helm charts onto kubernetes clusters.

Helm Philosophically

Helm uses templating to build up your final manifest, which means there will be a lot of brackets and non-yaml. The idea of a chart is really the unique part from helm as it’s utilization as a discreet package that can be deployed.

Pros for Helm3

There’s a lot you can do in a helm chart that you can’t do with kustomize and patches. 

  • The helm template functions are powerful in their own right; you have conditionals, loops over ranges, you can define your own helpers, and then you have the whole sprig library at your fingertips, with this comes complexity and imperative templating. 
  • It’s a templating engine.
  • Helm is a package manager. It works just as yum or apt does but for kubernetes. 
  • It boosts productivity due to the large amount of existing helm charts already out there.

Cons of Helm3

  • More abstraction layers and learning required
  • Harder CI-CD implementation.
  • Less readable templates which inevitably leads towards less customizability over time.
  • Not natively supported and therefore requires an external dependency
  • Even though the package manager works well, it still requires a decent set of customizations applied at runtime to handle image updates and other specifics, which increases the complexity of the ci/cd process.
  • Helm has a pretty steep learning curve and it can take some getting used to
  • Folder Structure
|-- Chart.yaml
|-- charts
|-- templates
|   |-- NOTES.txt
|   |-- _helpers.tpl
|   `-- service.yaml
`-- values.yaml

Example service.yaml

apiVersion: v1
kind: Service
name: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
type: {{ .Values.service.type }}
- port: {{ .Values.service.externalPort }}
    targetPort: {{ .Values.service.internalPort }}
    protocol: TCP
    name: {{ .Values.service.name }}
    app: {{ template "fullname" . }}

Example _helpers.tpl

{{/* vim: set filetype=mustache: */}}
Expand the name of the chart.
{{- define "mychart.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
{{- define "mychart.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}

Create chart name and version as used by the chart label.
{{- define "mychart.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

Common labels
{{- define "mychart.labels" -}}
helm.sh/chart: {{ include "mychart.chart" . }}
{{ include "mychart.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}

Selector labels
{{- define "mychart.selectorLabels" -}}
app.kubernetes.io/name: {{ include "mychart.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}

Create the name of the service account to use
{{- define "mychart.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}}
    {{ default (include "mychart.fullname" .) .Values.serviceAccount.name }}
{{- else -}}
    {{ default "default" .Values.serviceAccount.name }}
{{- end -}}
{{- end -}}

Kustomize: is a declarative tool, which works with yaml directly and works as a stream editor like sed. It traverses a Kubernetes manifest to add, remove or update configuration options without forking.

  • It is a very K.I.S.S. approach and doesn’t add an additional abstraction layer at all. It permits you to add logic into YAML, that’s all.
  • It is a purely declarative approach to configuration customization.
  • It runs as a standalone binary, as a stream editor like sed, which makes it perfect for CI/CD pipelines.

Kustomize Philosophically

  • Kustomize tries to follow the philosophy you are using in your everyday job when using Git as VCS, creating Docker images or declaring your resources inside Kubernetes.
  • It is like kubernetes in the fashion that it is totally declarative, you declare what you want and the system provides it to you.
  • It is like Docker in the fashion that you have many layers and each of those is modifying the previous one.

Pro’s for Kustomize

  • Declarative, so it matches with the same ideologies of kubernetes itself.
  • Native in as of kubectl v1.14, which means no external tools needed.
  • Available as a standalone binary for extension and integration into other services.
  • Every artifact that Kustomize uses is plain YAML and can be validated and processed as such.
  • Kustomize encourages a fork/modify/rebase workflow.
  • It’s not a templating engine, but a yaml patching system.
$ cd "$( mktemp -d )" ;
  $ mkdir -p ./base ./overlays/{dev,qat,stg,prd} ;

Example: Kustomization.yaml

$ touch ./base/kustomization.yaml ./overlays/dev,qat,stg,prd}/kustomization.yaml ;
$ cat <<EOF > ./base/kustomization.yaml
> apiVersion: kustomize.config.k8s.io/v1beta1
> kind: Kustomization
> namespace: default
> resources:
> - base-manifest.yaml
$ cat <<EOF > ./overlays/{dev,qat,stg,prd}/kustomization.yaml
> apiVersion: kustomize.config.k8s.io/v1beta1
> kind: Kustomization
> namespace: kube-system
> bases:
>   - ../../base
> patchesStrategicMerge:
> - deployments-patch.yaml

Cons of Kustomize:

  • Does not have a lot of bells and whistles.
  • Does not follow the DRY principle, in any functional definition.


To boil it all down to its base elements, Helm encapsulates Kubernetes objects into a single deployable unit and hides a lot of the complexity. Kustomize exposes everything and allows for more surgical changes that can change anything in a Kubernetes manifest.

In my opinion Kustomize is preferable. It allows you more control, as well as maintaining your ability to learn Kubernetes objects. This should be a prerequisite to using either of these tools, but with Kustomize you also have the ability to use both.

By using helm to generate the initial template and using Kustomize to modify it for your environment, complexity would be increased without much gain.

A user named xnxn_ on reddit once said it succinctly: 

“A sufficiently complex chart becomes unmaintainable, and because chart authors have to accommodate all the ways in which the chart might be used (annotations, node selectors, new features…), generic charts trend toward go template soup.”

Helm’s template languages operate on text, not YAML, so you get nastiness like `indent` and `quote`, and validation tooling (like an IDE) can’t help you with the YAML fields until you render the template (contrast with Kustomize patches, which are subsets of YAML resources).

Closing Remarks

I think it is ideal to not just discover these tools by reading about them, but familiarizing yourself with both. I would recommend you spend a few hours trying both of the tools out and testing what they can and can’t do, which in turn will make you more efficient in your chosen profession.


Azure DevOps YAML Pipeline with Terraform

Azure DevOps YAML Pipeline with Terraform

In my last post, I discussed the power of the Azure DevOps YAML pipeline with all of its built in features.  Today, I would like to focus on a specific use case for the Azure DevOps YAML pipeline with Terraform.  By combining these two great technologies, engineers...