A pipeline on every push

Every team has the same question a dozen times a day: is this change safe to ship? A pipeline answers it the same way every time. A push triggers it; it lints, installs, tests, builds, publishes an artifact, deploys to a staging environment, and finally to production. Anything that fails stops the line.

That’s the whole mechanism, and it’s worth exactly as much as its verdict. A pipeline that’s slow gets skipped. A pipeline that goes red for no reason gets ignored. A pipeline that ships something other than what it tested is lying to you. This lesson is those three ways to waste one.

Fast enough that nobody skips it

Most of a slow pipeline is downloading the same dependencies again. A cache saves them between runs under a key, and the usual key is a hash of the lockfile. GitHub’s docs describe the rule: if there is an exact match for the key, that’s a cache hit. Change the lockfile and the key changes with it, so you get a fresh download exactly when the dependencies really changed, and never otherwise.

Try it

A flaky test is a broken alarm

A flaky test passes and fails on the same code. Google measured this across its whole test corpus in 2016: about 1.5% of all test runs reported a flaky result, and almost 16% of tests had some flakiness. Their write-up names the real cost plainly: it is common to ignore legitimate failures in flaky tests because of all the false alarms, and developers sometimes dismiss a failure as flaky, only to find later that it was real.

Google does re-run failing tests as one of its mitigations, so retries aren’t wrong in themselves. The trap is treating the retry as the fix. The fix is removing the source of the flakiness - usually time, ordering or shared state - so that a failure means something again. In the widget, faking the clock and serialising the requests made “cart total” pass every time without the race and fail every time with it.

Build once, promote the artifact

The Continuous Delivery patterns page puts it as a rule: only build packages once. The reason, in its words: you want to be sure the thing you’re deploying is the same thing you’ve tested throughout the pipeline.

Rebuilding for production quietly breaks that. A floating dependency like ^1.4 resolves to whatever is newest at build time, so a rebuild an hour later can pull a release nobody tested. Promote the artifact by its digest instead - the same idea as pinning images by digest in Tags and digests.

cat .github/workflows/validate.yaml            # read the pipeline first
pip install pyyaml
python scripts/ci/check-chart-pins.py          # every Helm chart pinned to a version
python scripts/ci/check-route-hostnames.py     # every routed hostname has a certificate SAN
for k in $(find . -name kustomization.yaml -not -path "*/node_modules/*" -not -path "*/skeleton/*"); do
  kubectl kustomize "$(dirname "$k")" > /dev/null || echo "broken: $k"
done