New Adventures in ArgoCD: Automating Automation

ArgoCD is a great CICD tool for facilitating Git-based Kubernetes deployments. Helm is a widely used solution, with good reason, to simplify complex Kubernetes deployments and customize them to your needs. It’s even supported by Argo per se. You can’t ask for anything better than a combination of the two. Or can you?

Basically yes, but …

One of our customers decided to switch their management of a Kubernetes environment from the old shell-script-based deployments to ArgoCD. It was a good choice as the new solution has a wide range of advantages. Clear permissions, secrets kept secret, GitOps itself—yes, that transparency speaks for itself.
A good Helm Chart takes care of a lot, from Pods to ConfigMaps to Ingress Rules. It is deployed and before long, a complex application is functional. Fantastic, no doubt.
However, this is of little value if the logic of the sources used changes drastically, and the same is not rolled out manually or via Argo. And that was the problem in our case. If you used Helm commands to roll out the application in question, everything worked fine. When we tried to do the same with ArgoCD, the result was disastrous. Illogical names, empty links, nonexistent pods, and thus an application that simply did not work.

Automated Rollout of an Automated Automation Solution: Jenkins in Kubernetes

By the way, the application was Jenkins, which in itself is a well-established automation solution. It might sound surprising that instead of the usual Kubernetes solutions, such a classic solution is used. New automation platforms are invented in the cloud every week, so why Jenkins?

Well, its simple, really: it works, it’s extremely robust, and it evolves over time. It’s also a legitimate player in the Kubernetes environment. You can integrate it seamlessly through its infinite plug-ins, it can handle Git automatically, and you can set it up in a declarative way. Jenkins is not outdated. Jenkins is experienced. Especially for a company in the enterprise environment, it offers advantages, especially thanks to its integration capabilities.

Assuming, of course, that the application is successfully deployed, which brings us back to the core issue.

The Helm Charts

In our case, an upgrade was presented: the latest version of Jenkins, also modernized in appearance. But more importantly, it was also optimized beneath the surface,

even with very nicely and carefully written Helm charts. And that was just the problem. The charts were no longer one hundred percent compatible with the old logic (it changed the place where the declarative plug-in settings are included), but instead intended for classic Helm scenarios. (Approximately) this code

helm install -f myvalues.yaml myjenkins ./jenkins

converted the charts into an application, no question.

The problem was that the entire chart logic was designed for manual installation. And Argo is only as smart as all computers are in general: it does what it is asked to do. In this case: take the Helm charts and roll them out. Don’t ask any questions, don’t carry out any interaction, just roll them out (if you have to intervene all the time, it’s not about automation anymore, is it?). Here’s my repo, I want to be able to log into Jenkins within five minutes at myjenkins.mynamespace.mycompany.com. With admin rights if possible. With all my predefined pipelines and jobs and sources set up. It is possible and not very complicated either.

This is how our real project came about: we wanted to adapt the charts in such a way that they actually performed the task described above. We became aware of the most important difference between the classic Helm logic and a combination of Helm and Argo: for ArgoCD, everything has to be configured as explicitly as possible so that the program fulfills our request. All the tiny details have to be carefully set up in charts and values—under the condition that the environment-dependent parameters remain environment-dependent and that the original logic of the charts is not completely rewritten. After all, there will be further updates from the same team in the future, which should also be included here. Rewriting everything again and again would be anything but effective.

Accordingly, we compared the Helm manifests step by step, cleaning up and unifying the overlapping settings, adjusting the mount points of the resources, rewriting the configurations if necessary. We stayed as close to the original sources as possible.

And we were successful: after numerous trials and tests, we were able to provide Jenkins as it was needed— based on the Jenkins version we were given.

We are a little proud of ourselves at this point: our work was successful, Jenkins does what it is supposed to do. With the addition of a prepared Git repository, Jenkins can be deployed in our name space via ArgoCD. Here is my repository, I can log into Jenkins within five minutes at myjenkins.mynamespace.mycompany.com. With admin rights. And all my predefined pipelines and jobs and sources are set up.

If you have a similar situation, we’ll be happy to help you consolidate the two different automation solutions—then all users have to do is get started.

You can read more about this topic in the article by my colleagues Jan Bundesmann and Pascal Fries on heise.de (paywall).