GitOps – Kubernetes The Easy Way

Part I: Best practices with Kubernetes management techniques

The management of Kubernetes should be easy, simple and clear. Well-known DevOps practices and ultimately automation should be accessible by the management technique. The first GitOps principle, the declarative approach, is the essential Kubernetes management technique to leverage the desired feature set.

Kubernetes Training

Learn the basics of Kubernetes and container orchestration in just 3 days! From building and managing a container cluster to key concepts and best practices for running applications stably and securely in a Kubernetes cluster.
ATIX is official Kubernetes Training Provider

Introduction

This is the first article of a series on GitOps and its paradigms to make the configuration management of Kubernetes easy. The four GitOps principles are:

  1. declarative approach: the entire system is described declaratively,
  2. version control: the canonical desired system state is versioned in Git,
  3. deployment automation: approved changes are automatically applied to the system,
  4. continuous delivery: software agents ensure correctness and alert on divergence.

This article introduces different Kubernetes management techniques and will emphasize on the importance and the value of the declarative approach. The declarative approach is the first GitOps principle and it recommends to describe the entire system declaratively. In the following sections, three Kubernetes management techniques are explained. One technique is an approach without Kubernetes object configuration files:

  • imperative commands.

The other two techniques are based on Kubernetes object configuration files:

  • imperative object configuration,
  • declarative object configuration.

Join the LIVE webinar, “Kubernetes the Easy Way”, on May 15th, 10:00 am!

Imperative Commands

Create objects

The simplest way of interacting with the Kubernetes API are imperative commands. This management technique works without Kubernetes object configuration files, but with imperative steps. Imperative commands can be classified in verb-driven and object-driven commands. For example, a Deployment object for the vote containers can be generated by an verb-driven, imperative command:

kubectl run vote --image=andywirtzk8s/voting-vote:v1.0 --port=8080 -l "app=vote" -r 3 -n voting

With this command, the vote Deployment object in the voting Namespace is created by Kubernetes. The object creation requires only a single step and no configuration file is needed. The image can be found at dockerhub. A Service object for the vote containers can be generated with:

kubectl expose deployment vote --name vote -n voting

With this command, the vote Service object in the voting Namespace is created by Kubernetes. An HorizontalPodAutoscaler object for the vote containers can be generated with:

kubectl autoscale deployment vote --min 2 --max 5 --cpu-percent 80 --name vote -n voting

With this command, the vote HorizontalPodAutoscaler object in the voting Namespace is created by Kubernetes. These three commands are verb-driven. The next command is object-driven. With the object-driven command, different Kubernetes objects can be created. For example, a ConfigMap object for the vote containers can be generated with:

kubectl create configmap options --from-literal A=Imperative --from-literal B=Declarative -n voting

With this command, the options ConfigMap object in the voting Namespace is created by Kubernetes. Figure 1 depicts the four imperative commands to create Kubernetes objects.

Create objects with imperative commands.

Figure 1: Create objects with imperative commands.

Update Objects

The management technique with imperative commands for interacting with the Kubernetes API can also be used to change Kubernetes objects. Three available imperative commands are verb-driven. For example, a Deployment object can be horizontally scaled with:

kubectl scale deployment vote --replicas 2 -n voting

With this command, the replicas field of the vote Deployment object in the voting Namespace is updated by Kubernetes. The change requires only a single step and no configuration file is needed. A Kubernetes object can be annotated with:

kubectl annotate deployment vote atix.de/technique='imperative commands' -n voting

With this command, the annotations field of the vote Deployment object in the voting Namespace is updated by Kubernetes. A Kubernetes object can be labeled with:

kubectl label deployment vote technique=imperative -n voting

With this command, the labels field of the vote Deployment object in the voting Namespace is updated by Kubernetes. Alternatively, an aspect-driven command can be used. With an aspect-driven command, different fields for different Kubernetes objects can be changed. For example, the field of a Kubernetes object can be changed with:

kubectl set image deployment vote vote=andywirtzk8s/voting-vote:v1.1 -n voting

With this command, the image field of the pod template of the vote Deployment object in the voting Namespace is updated by Kubernetes and a rolling update is triggered. In addition, there are two imperative commands with a more general approach to changing live Kubernetes objects. For example, a specific field of a Kubernetes object can be modified by a patch string with:

kubectl patch deployment vote -p '{"spec": {"minReadySeconds": 10}}' -n voting

With this command, the minReadySeconds field of the vote Deployment object in the voting Namespace is updated by Kubernetes. Here, understanding the JSON format of the Kubernetes API Resources is required. For bigger changes to a Kubernetes object, the live object can be opened in the default editor with:

kubectl edit deployment vote -n voting

With the editor, changes can be made and saved. For example:

...
      - image: andywirtzk8s/voting-vote:v1.1
        name: vote
        envFrom:
        - prefix: OPTION_
          configMapRef:
            name: options
        livenessProbe:
           httpGet:
             path: /
             port: http
        readinessProbe:
           httpGet:
             path: /
             port: http
        ports:
        - containerPort: 8080
          name: http
...

In this example, the fields envFrom, livenessProbe, readinessProbe and ports[0].name are added to the live Deployment object. After making these changes, saving the file and exiting the editor, the vote Deployment object in the voting Namespace is updated by Kubernetes and a rolling update is triggered. Here, understanding the YAML format of the Kubernetes API Resources is required. Figure 2 depicts the six imperative commands to update Kubernetes objects.

Figure 2: Update objects with imperative commands.

Figure 2: Update objects with imperative commands.

Delete Objects

To delete a Kubernetes object, there exists one imperative object-driven command. For example, a Kubernetes object can be deleted with:

kubectl delete deployment vote -n voting

With this command, the vote Deployment object in the voting Namespace is deleted by Kubernetes. Figure 3 depicts the one imperative command to delete Kubernetes objects.

Figure 3: Delete objects with imperative commands.

Figure 3: Delete objects with imperative commands.

Benefits and Challenges

The major advantage of the Kubernetes management with imperative commands is the simplicity. Imperative commands are simple, easy to learn and easy to remember. Configuration changes require only a single step. In addition, changes to Kubernetes objects can be made by multiple writers without conflicting.
The major disadvantage of the Kubernetes management with imperative commands is the lack of Kubernetes object configuration files. Without the Kubernetes object configuration files, automation, standardization and reproducibility is impossible. DevOps practices like source control, change review processes and audit trails associated with changes can not be accessed.
Table 1 lists the features of the imperative commands for managing Kubernetes objects. More reading on this management technique can be found under imperative commands.

Management technique Imperative commands
Simplicity
Multiple editors
Operate on directories
Source control
Change review processes
Audit trails
Standardization
Reproducibility
Idempotence
Automation

Table 1: Features of the imperative commands.

Imperative Object Configuration

Create objects

A more sophisticated way of interacting with the Kubernetes API is the imperative object configuration. This management technique works with Kubernetes object configuration files. For example, a Deployment object for the vote containers can be defined in a file:

FILE=${HOME}/configs/voting/vote/deployment.yaml

The content of the file is a YAML descriptor of the Deployment object:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    atix.de/technique: imperative object configuration
  labels:
    app: vote
    technique: imperative
  name: vote
  namespace: voting
spec:
  minReadySeconds: 10
  replicas: 3
  selector:
    matchLabels:
      app: vote
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: vote
    spec:
      containers:
      - image: andywirtzk8s/voting-vote:v1.0
        name: vote
        envFrom:
        - prefix: OPTION_
          configMapRef:
            name: options
        livenessProbe:
           httpGet:
             path: /
             port: http
        readinessProbe:
           httpGet:
             path: /
             port: http
        ports:
        - containerPort: 8080
          name: http

This is a YAML manifest representing the desired state of the Deployment object. This Deployment object can be generated with:

kubectl create -f ${FILE}

With this command, the vote Deployment object in the voting Namespace is created by Kubernetes. For the command to work, it is required that the vote Deployment object in the voting Namespace does not exist beforehand. Figure 4 depicts the one imperative object configuration command to create Kubernetes objects.

Figure 4: Create objects with imperative object configuration.

Figure 4: Create objects with imperative object configuration.

Update objects

The management technique of imperative object configuration for interacting with the Kubernetes API can also be used to change Kubernetes objects. For example, the Deployment object configuration can be replaced by:

kubectl replace -f ${FILE}

Depending on the change made in the YAML manifest of the vote Deployment object, Kubernetes acts to ensure the new desired state. If the replicas field is changed, the live Deployment object is scaled horizontally. If the image field is changed in the pod template of the Deployment object, a rolling update is triggered. Kubernetes replaces the live object even if there is no configuration change. This results in imperative, non-idempotent configuration management. For the command to work, it is required that the vote Deployment object in the voting Namespace does exist beforehand. Figure 5 depicts the one imperative object configuration command to update Kubernetes objects.

Figure 5: Update objects with imperative object configuration.

Figure 5: Update objects with imperative object configuration.

Delete objects

To delete a Kubernetes object, there exists one command within the imperative object configuration technique. For example, a Kubernetes object can be deleted with:

kubectl delete -f ${FILE}

With this command, the vote Deployment object in the voting Namespace is deleted by Kubernetes. Figure 6 depicts the one imperative object configuration command to delete Kubernetes objects.

Figure 6: Delete objects with imperative object configuration.

Figure 6: Delete objects with imperative object configuration.

Benefits and challenges

The major advantage of the Kubernetes management with imperative object configuration is the record of the desired state in form of Kubernetes object configuration files. With these Kubernetes object configuration files, standardization and reproducibility can be enabled. DevOps practices like the usage of a source control system, of change review processes and of audit trails associated with changes can be accessed.

The major disadvantage of the Kubernetes management with imperative object configuration is the missing feature of idempotent configuration management. Updates are done by replacing the live object, even if there is no configuration drift between the configuration file and the live object.

If multiple editors manage the same Kubernetes object, changes of one source might be lost by the replace command of the second source. Such updates to live objects must be reflected in the configuration files. Moreover, there are additional requirements for the create and replace commands to work. Imperative object configuration works best on files, not directories. All these issues make it difficult to perform or automate configuration changes.

Table 2 lists the features of the imperative object configuration for managing Kubernetes objects. More reading on this management technique can be found under imperative object configuration.

Management technique Imperative object configuration
Simplicity 🔶
Multiple editors
Operate on directories 🔶
Source control
Change review processes
Audit trails
Standardization
Reproducibility
Idempotence
Automation

Table 2: Features of the imperative object configuration.

Declarative Object Configuration

Create objects

The most sophisticated way of interacting with the Kubernetes API is the declarative object configuration. This management technique works with Kubernetes object configuration files. It supports operating on directories with Kubernetes object configuration files. For example, a directory containing YAML descriptor files of Kubernetes objects can be defined:

DIR=${HOME}/configs/voting/vote

In this directory, the YAML manifests representing the desired state of the Deployment, the Ingress and the Service object for the vote containers are stored:

.
├── deployment.yaml
├── ingress.yaml
└── service.yaml

The files can be found on Github. The three Kubernetes objects can be printed with:

kubectl diff -f ${DIR}/

With this command, the vote Deployment, the vote Ingress and the vote Service objects are printed by Kubernetes and can be reviewed. The output does not only show the fields defined in the YAML manifests, but is enriched by the default field values of Kubernetes. The three Kubernetes objects can be generated with:

kubectl apply -f ${DIR}/

With this command, the vote Deployment, the vote Ingress and the vote Service objects are posted to Kubernetes and created by Kubernetes in the voting Namespace. There are no requirements on whether the objects exist beforehand or not. Figure 7 depicts the two relevant declarative object configuration commands to create Kubernetes objects.

Figure 7: Create objects with declarative object configuration.

Figure 7: Create objects with declarative object configuration.

Update objects

The management technique of declarative object configuration for interacting with the Kubernetes API can also be used to change Kubernetes objects. The difference between the locally stored YAML manifests and the live Kubernetes object counter-parts can be printed with:

kubectl diff -f ${DIR}/

With this command, the configuration drift of the vote Deployment, the vote Ingress and the vote Service objects is printed by Kubernetes and can be reviewed. The output shows only the fields that are not in synchronization between the locally stored Kubernetes object configuration files and the live Kubernetes objects. Independent of whether a difference is shown or not, the three Kubernetes objects can be updated with:

kubectl apply -f ${DIR}/

With this command, three different scenarios have to be considered. In case one, the Kubernetes object did not exist beforehand. Here, it is created by Kubernetes as described in the last subsection. In case two, there is no difference between the YAML manifest and the live Kubernetes objects. Here, no action is performed. In case three, one or more fields in one or more Kubernetes objects have been changed. Here, a merge patch is calculated and the Kubernetes objects are patched by Kubernetes.

With this merge patch calculation, the desired system state is not only described declaratively in the YAML manifests, but also communicated to Kubernetes in a declarative way. This results in idempotent configuration management. For example, if the replicas field and the image field are changed in the locally stored Kubernetes object configuration files, the live Deployment object is scaled horizontally and a rolling update is triggered, while the live Ingress and Service objects remain untouched.

On the other hand, if a field is not defined in the YAML manifest file, it is ignored by the command. If some imperative changes have been made on default field values not defined in a Kubernetes object configuration file, this change will not be overwritten. Figure 8 depicts the two relevant declarative object configuration commands to update Kubernetes objects.

Figure 7: Create objects with declarative object configuration.

Figure 8: Update objects with declarative object configuration.

Delete objects

To delete Kubernetes objects, there exists one command within the declarative object configuration technique. For example, the three Kubernetes objects can be deleted with:

kubectl delete -f ${DIR}/

With this command, the vote Deployment, the vote Ingress and the vote Service objects in the voting Namespace are deleted by Kubernetes. Figure 9 depicts the one declarative object configuration command to delete Kubernetes objects.

Figure 9: Delete objects with declarative object configuration.

Figure 9: Delete objects with declarative object configuration.

Benefits and challenges

The major advantage of the Kubernetes management with declarative object configuration is the support for operating on directories containing Kubernetes object configuration files and automatically detecting the operation type per object. Kubernetes can automatically decide whether an object or field should be created, patched, deleted or remain untouched. This enables idempotent configuration management.

With the directories containing the Kubernetes object configuration files, automation, standardization and reproducibility can be enabled. DevOps practices like the usage of a source control system, of change review processes and of audit trails associated with changes can be accessed.

The Kubernetes management with declarative object configuration is the most powerful of the management techniques. It comes with the cost of a steeper learning curve, as the patch merge calculation can result in unexpected behavior.

Table 3 lists the features of the declarative object configuration for managing Kubernetes objects. More reading on this management technique can be found under declarative object configuration.

Management technique Declarative object configuration
Simplicity
Multiple editors
Operate on directories
Source control
Change review processes
Audit trails
Standardization
Reproducibility
Idempotence
Automation

Table 3: Features of the declarative object configuration.

Summary

The goal of this article is to show the value of the declarative approach to manage Kubernetes. Table 4 compares the features of the three different Kubernetes management techniques. More reading on these options can be found under management techniques.

Management technique Imperative commands Imperative object configuration Declarative object configuration
Simplicity 🔶
Multiple editors
Operate on directories 🔶
Source control
Change review processes
Audit trails
Standardization
Reproducibility
Idempotence
Automation

Table 4: Comparison of the features of the management techniques.

First, it can be differentiated between management via imperative commands and via Kubernetes object configuration files. In the imperative approach with imperative commands, actions are defined. In Kubernetes object configuration files, state is defined. In the configuration files, the desired system state is described declaratively.

Second, it can be differentiated between management via imperative object configuration and declarative object configuration. In the imperative approach with imperative object configuration, state is communicated with an imperative command. In the declarative approach with declarative object configuration, state is communicated with a declarative command. Based on the actual and the desired system state, necessary actions are derived in the declarative approach.

The management technique with imperative commands is good for first steps with Kubernetes in a development environment. But this imperative approach does not enable DevOps practices. Ultimately, automation is impossible.

The management technique with imperative object configuration is a hybrid method, as it describes the desired Kubernetes state declaratively, but it communicates the state to Kubernetes with an imperative command. This hybrid approach enables a lot of DevOps practices. But ultimately, the lack of idempotence makes automation impossible.

The management technique with declarative object configuration is the recommendation for production environments of Kubernetes. This declarative approach enables DevOps practices. And the ultimate goal of automation is possible.

The declarative object configuration is the most sophisticated management technique and the one with the steepest learning curve. To reduce its complexity, it is important to embed this declarative approach in a wider set of practices. This article series recommends the GitOps principles.

The first GitOps principle is the declarative approach. This paradigm advocates to describe the entire system declaratively. The declarative object configuration is the Kubernetes management technique enabling this first GitOps principle.

The following two tabs change content below.

Andy Wirtz

Andy Wirtz is a Senior IT Consultant at ATIX AG, Germany. He supports his clients in the setup and configuration of container platforms, in the deployment of cloud native services, and in the development of microservice applications. He is an expert in the technical practices of DevOps, Continuous Delivery and automation. He provides in-depth trainings, workshops and webinars about Kubernetes and OpenShift.

Latest posts by Andy Wirtz (see all)