Curious Pandas
Explorer0 XP

Load balancers and proxies

One address, many servers. Watch round robin feed a crashed backend, watch least connections make it worse, then fix it with health checks - and learn why only an L7 proxy can route by path.

Runs in your browserOn your machine
16 min+40 XP for finishing

A popular site isn’t one server. It’s one address in front of many servers, with a load balancer or reverse proxy deciding which server gets each request. Clients never see the decision. That’s the point, and it’s also why a bad decision is so hard to spot from outside.

A balancer makes two choices for every request: which servers are allowed to take it, and which one of those gets it.

Which one: the algorithm

Round robin takes turns: 1, 2, 3, 1, 2, 3. It’s NGINX’s default. Least connections sends each request “to the server with the least number of active connections” (NGINX docs). That sounds smarter, because busy servers get a rest.

Here’s the catch. A crashed server that refuses connections instantly never has an active connection, so least connections thinks it’s the least busy server in the pool and keeps choosing it.

Which are allowed: health checks

A health check probes each backend on a timer and takes failing ones out of rotation until they pass again. Without one, the balancer keeps sending a dead server its share of requests. Real balancers let you tune how many failures it takes: NGINX’s passive check marks a server unavailable after max_fails=1 failure within fail_timeout=10s by default. This simulation pulls a backend after two failed checks and lets it back after two good ones.

NGINX’s defaults go further: a failed connection is retried on the next server (proxy_next_upstream error timeout), and one failure takes a server out for 10 seconds - which would hide a refusing backend from customers. The simulator turns both off, the way a plain layer 4 balancer or a pool with no checks behaves, so you can see what the checks protect you from.

Load balancersimulated traffic
Scenario:

      What a balancer can see

      Visual guideWhat each kind of balancer can read
      What each kind of balancer can readA layer 4 balancer forwards TCP connections and sees only addresses and ports, so it can spread connections but not route by what is asked for. A layer 7 proxy ends the connection, reads the HTTP request - host and path - and can send /api to one service and everything else to another.clientGET /api/ordersL4 balancersees IP and portany backendchosen blindL7 proxyreads host + pathapi-1serves /api/*TCPbytesHTTPpath /apiWhat each kind of balancer can readA layer 4 balancer forwards TCP connections and sees only addresses and ports, so it can spread connections but not route by what is asked for. A layer 7 proxy ends the connection, reads the HTTP request - host and path - and can send /api to one service and everything else to another.clientGET /api/ordersL4 balancersees IP and portany backendchosen blindL7 proxyreads host + pathapi-1serves /api/*TCPbytesHTTPpath /api
      1. A request for /api/orders leaves the client. Which server can answer it depends on the path.
      2. An L4 balancer forwards the TCP connection. The path is inside bytes it never reads - and inside TLS, it could not read them - so it picks a backend blind.
      3. An L7 proxy ends the connection and reads the HTTP request: the Host header and the path. Now /api can go to the API and / to the web servers.

      That’s why routing by hostname or path, as in “grafana.localhost goes to Grafana” or “/api goes to the API”, is a layer 7 job. In Kubernetes, the Gateway API’s HTTPRoute describes exactly those rules: hostnames, path matches and the backends they go to.

      Check yourself+10 XP

      One of three backends crashed and refuses connections. Which change stops customers seeing errors?

      Check yourself+10 XP

      You need /api to reach a different service from /. What must the balancer be able to do?

      Do it for real · your machine+50 XP bonus

      Everything above ran in your browser. Now do it on a real cluster with tilt lab 02-route-a-hostname (profile minimal). You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?

      The tilt platform’s front door is an L7 proxy: an Istio-managed Gateway listening for *.localhost (HTTPS on 443, plus port 80 for the redirect), and an HTTPRoute per app that matches on hostname. Read the rules that route grafana.localhost and prometheus.localhost to different backends:

      kubectl get gateway -n istio-system
kubectl get httproute -A
kubectl get httproute grafana -n monitoring -o yaml   # hostnames, matches, backendRefs

      Lab 02 then has you route a second hostname yourself.

      Self-reported. We can't see your machine, and we don't want to. This is a bonus, never required.

      Sources

      Finished reading?

      Missions and quizzes above earn their own XP. This marks the lesson done.