Branch by abstraction, and delete the flag

The Strangler Lab replaced a whole system from the outside, one route at a time. Branch by abstraction does the same thing from the inside, for one component at a time - a payment provider, a storage layer, a library - without a long-lived branch that drifts away from everyone else’s work.

The steps

Paul Hammant introduced the name publicly in 2007, crediting Stacy Curl with it. The steps:

1. Put an abstraction in front of the component you want to replace, and move the code that uses it onto the abstraction.
2. Build the new implementation behind the same abstraction, merged into the main line as you go, switched off.
3. Switch callers over to the new implementation, gradually if you can.
4. Delete the old implementation.
5. Optionally, delete the abstraction too, if it only existed for the migration.

Martin Fowler’s rule for the whole process: “Ensure that the system builds and runs correctly at all times.” Danilo Sato’s three verbs for the same shape - expand, migrate, contract - come with a warning about stopping halfway: “you might end up in a worse state than you started”. The branching strategies lesson covers the flags that make step 2 possible; this lesson is about steps 4 and 5.

trunkbaseddevelopment.com also says when not to use it: if customers choose when to upgrade and old interfaces have to live on, you need versioning instead.

The trap: the switch outlives the change

The new implementation works, traffic is at 100%, and the team moves on. The old path and the flag stay in the code. Pete Hodgson’s advice: “Savvy teams view their Feature Toggles as inventory which comes with a carrying cost”, and they put a limit on how many can exist. Uber built a tool called Piranha to remove stale flags automatically, and reports removing around two thousand of them.

The worst case has a regulator’s report. In 2012, Knight Capital deployed new code that reused an old flag. One of its eight servers still had old, unused code behind that flag, and turning it on ran it. Google’s SRE book cites the case: code “gated by a flag that is always disabled, is a metaphorical time bomb waiting to explode, as painfully experienced by Knight Capital”.

Try it

Three flags. Decide which one is a liability.

A one-line silent failure, for real

GitHub compared its old and new merge code on real traffic and found a bug in the old one: a merge with 768 conflicts was reported as success, because shell exit codes keep only the lowest 8 bits, and 768 is a multiple of 256. You can see the same arithmetic in any POSIX shell:

sh -c 'exit 768'; echo $?    # prints 0: "success"

No error, no warning: the number just wrapped. It’s a reminder of why the compare-before-you-switch step matters even when you trust the old code.