Curious Pandas
Explorer0 XP

The dashboard that lies

Metrics, logs and traces, and the most dangerous graph of all, the empty one. Break the watching instead of the thing being watched, and find where the truth is written.

Runs in your browserOn your machine
16 min+40 XP for finishing

Observability is how you find out what a system is doing without stopping it. It rests on three signals:

  • Metrics: numbers over time, such as requests per second or memory used. Cheap to keep, good for graphs and alerts. On the tilt platform: Prometheus, drawn by Grafana.
  • Logs: what the program said, line by line. On the platform: Loki.
  • Traces: the path one request took through several services, and how long each hop took. On the platform: Tempo.

Every other lesson breaks the thing you’re watching. This one breaks the watching.

Three empty dashboards on a healthy cluster

This happened three times on the tilt platform:

  • Argo CD’s monitor asked for a port named metrics, but every Argo CD service publishes http-metrics.
  • Loki’s monitor selected the label app: loki, but the chart had moved to app.kubernetes.io/name.
  • cert-manager had no monitor at all.

Each time, Kubernetes was content, Prometheus reported nothing wrong, and eighteen panels read as a calm, healthy system.

How a scrape is wired

With the Prometheus Operator, you don’t edit Prometheus’s config. You create a ServiceMonitor, which says “scrape the Services with this label, on the port with this name”. Both are matched exactly. Get either wrong and the operator generates no scrape target, and it doesn’t consider that an error. Nothing failed, so nothing alerts.

Visual guideFrom ServiceMonitor to panel
From ServiceMonitor to panelA ServiceMonitor names a Service label and a port name, which must match a real Service exactly. The Prometheus Operator turns each match into a scrape target; no match gives no target and no error. Prometheus scrapes the pods behind the Service, and Grafana draws what Prometheus stored. Prometheus's targets page is the only place a missing target shows.ServiceMonitorlabel + port nameService helloport name: httpoperatorno match: no errorPrometheusthe truth: /targetspod helloserves /metricsGrafana panelempty = no target?must matchread bytargetsendpointsscrapesqueried byFrom ServiceMonitor to panelA ServiceMonitor names a Service label and a port name, which must match a real Service exactly. The Prometheus Operator turns each match into a scrape target; no match gives no target and no error. Prometheus scrapes the pods behind the Service, and Grafana draws what Prometheus stored. Prometheus's targets page is the only place a missing target shows.ServiceMonitorlabel + port nameService helloport name: httpoperatorno match: no errorPrometheusthe truth: /targetspod helloserves /metricsGrafana panelempty = no target?must matchread bytargetsendpointsscrapesqueried by
  1. A ServiceMonitor names a label and a port name. Both must match a real Service exactly - "metrics" is not "http-metrics".
  2. The operator turns each match into a scrape target. No match, no target - and it does not count that as an error.
  3. Prometheus scrapes /metrics on the pods behind the Service. Its /targets page is the one place a missing target shows.
  4. Grafana draws what Prometheus stored. No target means an empty panel on a perfectly healthy cluster.
Why is this graph empty?
ServiceMonitor hello
spec:
  selector:
    matchLabels:
      : 
  endpoints:
    - port:
Service hello (what actually exists)
metadata:
  labels:
    app.kubernetes.io/name: hello
spec:
  ports:
    - name: http
      port: 9898
What you would normally check
  • ✓ servicemonitor/hello exists, no events
  • ✓ pod hello 1/1 Running, Ready
  • ✓ logs: serving requests
Grafana: requests per second

Existence is not freshness

Look at what the lab’s grader checks. It asserts four separate facts on purpose, because the first can pass while the other three fail:

  1. the ServiceMonitor exists;
  2. the port it names actually exists on the Service;
  3. Prometheus has a healthy target;
  4. the data is fresh: the last scrape was recent.

“There’s a series called hello” and “something scraped hello a moment ago” are different questions. Ask the second.

Check yourself+10 XP

A Grafana panel is empty. The pods are Ready and the ServiceMonitor exists with no events. What's the quickest place to find the problem?

Check yourself+10 XP

A ServiceMonitor says port: metrics. The Service's port is named http and is number 9898. What happens?

Check yourself+10 XP

Which signal answers 'which of my five services made this one request slow?'

Do it for real · your machine+50 XP bonus

Everything above ran in your browser. Now do it on a real cluster with tilt lab 08-empty-dashboard (profile observability). You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?

PROFILE=observability ./scripts/platform.sh up
kubectl apply -f labs/08-empty-dashboard/servicemonitor.yaml
./scripts/platform.sh lab 08          # 4/4
./labs/08-empty-dashboard/break.sh
# open https://prometheus.localhost/targets and search for hello
./labs/08-empty-dashboard/fix.sh && ./scripts/platform.sh lab 08
Self-reported. We can't see your machine, and we don't want to. This is a bonus, never required.

Sources

Finished reading?

Missions and quizzes above earn their own XP. This marks the lesson done.