Curious Pandas

Git as the source of truth

GitOps puts the desired state in a git repository and a controller keeps the cluster matching it. Watch three kinds of reconciler handle the same kubectl edit very differently.

16 min+40 XP for finishing

In the Kubernetes track you met the core idea: write down what you want, and a controller keeps making it true. GitOps takes one more step. The “what you want” lives in a git repository, and a controller in the cluster keeps pulling from it and applying it.

What you get:

  • Every change is a commit. It has an author, a reason and a review, and you can revert it.
  • The cluster can be rebuilt from the repo. Lose the cluster and point a new one at the same repository.
  • Hand edits become visible, because live state that doesn’t match git is drift.

That last point depends entirely on what’s doing the reconciling, and that’s where people get caught.

Three reconcilers, one kubectl edit

Flux is a CNCF GitOps toolkit with separate controllers for separate jobs. Two of them matter here:

  • A Kustomization applies plain manifests. The Flux docs say each reconciliation, every interval, applies the manifests again, “correcting any existing drift in the process”. With prune turned on, it also deletes objects that were removed from git.
  • A HelmRelease installs a Helm chart. Its interval re-checks the release against the spec. Comparing the live objects is a separate, optional feature: .spec.driftDetection is opt-in.
Visual guideWho puts it back?
Who puts it back?Desired state lives in a git repository. A Kustomization applies its manifests to the cluster again every interval, so a kubectl edit is corrected. A HelmRelease installs a chart and, without driftDetection, does not compare live objects, so the same edit stays until the next upgrade.git repositorydesired stateKustomizationapply againHelmReleaseno driftDetectionclusterlive statekubectl edita hand changemanifestson intervaldriftchart+valueson upgradeWho puts it back?Desired state lives in a git repository. A Kustomization applies its manifests to the cluster again every interval, so a kubectl edit is corrected. A HelmRelease installs a chart and, without driftDetection, does not compare live objects, so the same edit stays until the next upgrade.git repositorydesired stateKustomizationapply againHelmReleaseno driftDetectionclusterlive statekubectl edita hand changemanifestson intervaldriftchart+valueson upgrade
  1. The desired state lives in git. Every change is a commit, with an author and a reason.
  2. A Kustomization applies those manifests to the cluster again on every interval.
  3. A hand edit changes the cluster but not git. That gap is drift - and the next Kustomization run puts git back.
  4. A HelmRelease re-checks the release, not the live objects. Without driftDetection, the same edit stays until the next upgrade.

Pick a reconciler, scale the Deployment by hand, and watch what happens to your edit.

Git says one thing, the cluster says another
Git (desired)
Reconciler
Cluster (live)

    Pin your versions

    One more habit from the tilt repo, in a comment on every HelmRelease:

    Pinned, not “*”: Flux re-resolves an unbounded version on every reconcile, so the platform rebuilt later is not the one built today.

    GitOps promises you can rebuild from the repo. With a floating version, you can rebuild something. With a pinned one, you rebuild the same thing.

    Check yourself+10 XP

    A Flux Kustomization manages a Deployment. Someone runs kubectl scale --replicas=7 to handle a traffic spike. What happens?

    Check yourself+10 XP

    You remove a ConfigMap's file from git. The Kustomization has prune: false. What happens to the ConfigMap in the cluster?

    Check yourself+10 XP

    Why does the tilt platform pin every chart version instead of using '*'?

    Do it for real · your machine+50 XP bonus

    Everything above ran in your browser. Now do it on a real cluster withthe tilt platform. You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?

    On the running platform:

    flux get helmreleases -A          # every chart Flux owns, its version and status
    flux get sources helm -A          # the chart repositories they come from
    kubectl get helmrelease -A -o custom-columns='NAME:.metadata.name,VERSION:.spec.chart.spec.version,INTERVAL:.spec.interval'

    The gitops profile (PROFILE=gitops ./scripts/platform.sh up) adds Crossplane, Flagger, External Secrets and CloudNativePG, all reconciled the same way.

    Self-reported. We can't see your machine, and we don't want to. This is a bonus, never required.

    Sources

    Finished reading?

    Missions and quizzes above earn their own XP. This marks the lesson done.