Curious Pandas

Switching it off is the riskiest change

Removing an old system looks like housekeeping. It is a production change with its own blast radius, and two famous failures happened there. Find the caller nobody listed, switch the old system off the reversible way, and delete it last.

18 min+45 XP for finishing

At the end of a migration there’s an old system nobody wants to look after any more. Turning it off feels like the easy part. It isn’t, and Google’s guidance on deprecation starts from the reason: “code is a liability, not an asset”. Every line you keep has to be secured, upgraded and understood. But removing it is a change to production too, with its own ways to fail.

Two failures, mirror images

Knight Capital, 2012: something that should have been removed wasn’t. When a new release went out, according to the SEC, “one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers”. That server still held old, unused code, and a reused flag woke it up. Knight “lost over $460 million” in “a 45-minute period”.

Atlassian, 2022: the wrong thing was removed. A script was retiring an old app. “Instead of providing the IDs of the intended app being marked for deletion, the team provided the IDs of the entire cloud site where the apps were to be deleted.” The CTO’s update adds: “The script was executed with the wrong execution mode and the wrong list of IDs” - permanent deletion rather than the recoverable mode. Customers lost access for up to two weeks.

Announcing it is not removing it

HTTP has two headers for this. A Deprecation header says an endpoint is deprecated, and RFC 9745 is explicit about what that does: “The act of deprecation does not change any behavior of the resource.” A Sunset header gives a date, and RFC 8594 calls such dates hints. Both are useful. Neither tells you who is still calling. An endpoint can carry a deprecation header for a year and still have a caller nobody knows about: configured, and inert.

Google’s chapter makes the same point about process: deprecation needs owners and deadlines, and warnings people actually act on. It also describes a blunter tool, planned outages of increasing length, that find dependencies nobody listed. GitHub announced exactly that in 2021 when retiring weak SSH keys, old ciphers and unencrypted git:// connections: several short, scheduled brownouts before the final cut-off.

Disable, watch, then delete

Microsoft’s Well-Architected guidance turns this into a checklist. Define “clear indicators of inactivity across a full user or business cycle”. “Disable before deleting.” Then watch a window in which any unexpected activity resets the clock.

Try it

The same shop as the Strangler Lab, after the migration: every route the team knows about is already on the new system. Legacy is still on. Your job is to find what still needs it, then switch it off without breaking anyone. Reading legacy’s own access log over a full month works; so does a planned brownout - switching it off and seeing what fails, then switching it back on.

Strangler Labsimulated: invented traffic, one day per step, simplified data syncday 0

Façade rules first match wins; anything unmatched goes to legacy

Differences found

None yet - nothing has been compared.

Data

Stores never compared.

Legacy app

Legacy's access log, last 30 days
    wrong answers served0
    customers charged twice0
    cart items lost0
    failed requests0
      Check yourself+10 XP

      An endpoint has returned a Deprecation header for six months. Can you remove it now?

      Check yourself+10 XP

      What made Atlassian's 2022 deletion so hard to recover from?

      Sources

      Finished reading?

      Missions and quizzes above earn their own XP. This marks the lesson done.