A mirror that compares nothing

Tests catch the bugs you thought of. When you replace a piece of a running system, the bugs that hurt are the ones nobody thought of: the odd input, the edge case the old code handled by accident, the behaviour a customer quietly depends on. The best evidence you have is to run the old and the new side by side on real requests and compare their answers before any user sees the new one.

GitHub did this when it rewrote how pull requests are merged. The first step was to “enable it for a tiny fraction (1%) of all the requests”. After four days of fixing what the comparisons found, the experiment ran “for 100% of the requests, for 24 hours, and with no mismatches nor slow cases during the whole run”. In total it “verified tens of millions of merge commits”, and along the way it found “2 serious merge-related bugs in the original Git implementation which had gone undetected for years”. The comparison found bugs in the old code, which is a reminder that the old system is not automatically the right answer.

That result is evidence because of the numbers next to it: how many were compared, and for how long. This lesson is about the setups where those numbers quietly come out as zero.

Three ways to get two answers

• In the application. GitHub’s Scientist library wraps the old code path (the control) and the new one (the candidate). It runs both, and “experiment.run will always return whatever the use block returns”: the user always gets the old answer while the new one is recorded.
• In the proxy. Envoy, Istio and NGINX can mirror each request to a second service. This needs no code change, which is why it is tempting.
• In a diff service. Twitter’s Diffy (archived in 2020) replayed requests against the old and new versions and reported the differences.

The proxy throws the answer away

Mirroring sends the copy. It does not compare anything. Istio says mirrored requests are sent “as ‘fire and forget’, which means that the responses are discarded.” Envoy’s docs use the same words: Envoy “will not wait for the shadow cluster to respond before returning the response from the primary cluster”. NGINX says: “Responses to mirror subrequests are ignored.”

None of these is a bug. Each proxy does exactly what it documents. But a team that turns mirroring on, sees the new system taking traffic, and watches its error graph stay flat has checked one thing only: that the new system answers. Whether it answers the same was never asked.

Configs that mirror nothing, or everything

The mirror itself has quiet failure modes, all of them valid configuration:

• A header-named target. Envoy’s cluster_header reads the mirror target from a request header, and “no shadow request will happen if the value is not found in headers”. If your clients never send the header, nothing is mirrored. The docs don’t describe any error for this case.
• A missing primary. Also from Envoy: “Shadowing will not be triggered if the primary cluster does not exist.”
• No fraction. Envoy’s runtime_fraction: “If not specified, all requests to the target cluster will be mirrored.” Istio’s mirrorPercentage works the same way: “If this field is absent, all traffic will be mirrored.” A new system sized for a 10% shadow gets all of production instead.

The comparison side has its own. Scientist’s README warns: “Because enabled? and run_if determine when a candidate runs, it’s impossible to guarantee that it will run every time.” It also says of its own first example: “The try blocks don’t run yet and none of the results get published.” A sampling rule that matches nobody compares nobody.

Try it

The bench below runs five minutes of a small shop’s traffic through an Envoy route. The YAML updates as you change the controls. Every result shows compared next to differed, and a run that compared nothing is shown amber as “not checked”, never as a pass.

Noise is not a difference, until it hides one

Two correct systems still disagree. Shopify rebuilt its storefront rendering this way, and wrote: “Time-related and randomness-related exceptions make it impossible to have exactly byte-equal responses, so we ignore certain patterns in the verifier service to relax the equivalence criteria.” Its verifier “uses a fixed time value during the comparison process and sets any random values to a known value”.

Diffy handled noise by measurement rather than by guesswork. Its README lists “A secondary instance running the same known-good code as the primary instance”, and any field that differed between the two copies of the old code was noise by definition. Scientist lets you ignore a mismatch with your own rule. Every one of those rules is also a place to hide a real bug. An ignore rule broad enough to silence the noise can silence a wrong price too. You’ll meet that one in the next lesson’s lab.

Writes: the proxy discards the answer, not the side effect

Comparing a read is safe. Comparing a write is not, because the copy runs too. Scientist “is only safe for wrapping methods that aren’t changing data”. Diffy’s README says: “For safety reasons POST, PUT, DELETE are ignored by default.” It takes a command-line flag to turn them back on.

A mirrored POST /checkout reaches the new system, which talks to the real payment provider and charges the card. The proxy throws away the response, but nothing undoes the charge. That last step is our inference from the documented behaviour, not a quote from a vendor, but it follows directly from “fire and forget”. The usual answer is to compare writes against a copy of the new system whose side effects are stubbed, or to leave writes out and say so in the “not compared” count.

What a canary adds

A shadow shows that the new system gives the same answers. A canary shows that real users are fine with its answers. The release strategies lesson covers how canaries promote and roll back. The SRE Workbook’s chapter on canarying adds three things about the comparison side that bite exactly the way a shadow does:

• Compare at the same time, not before and after. A heading in the chapter reads “Before/After Evaluation Is Risky”, and the text explains: “Because time is one of the biggest sources of change in observed metrics, it is difficult to assess degradation of performance with before/after evaluation.” Compare a canary population with a control population running at the same moment.
• Volume. “Terminating a canary deployment after receiving just a handful of queries doesn’t provide a useful signal.” A bug in one request in five hundred needs thousands of requests before you can say you looked for it.
• Windows. The chapter’s failure example comes from “using a metric that is computed hourly to evaluate a deployment that is only 30 minutes long”. The metric could not show the change, so it didn’t.

Two more refinements from practice. Kayenta, Google and Netflix’s canary analyser, compares the canary against a newly created baseline rather than the long-running fleet: “Creating a new baseline helps minimize startup effects and limit the system variations between it and the canary.” AWS deploys first to a single box, and notes that “issues introduced by a one-box deployment might not trigger the service’s high-severity rollback alarm”, which is why it gives that box alarms of its own. Each refinement exists for the same reason: a check that cannot see the change looks just like a check that passed.