Curious Pandas

How much can you lose?

Every disaster asks two questions - how much data is gone, and how long you are down. RPO and RTO are the answers you choose in advance, per system, and the business chooses them, not IT.

12 min+35 XP for finishing

A disk dies, a region goes dark, someone runs the wrong script. However it happens, the morning after comes down to two questions:

  1. How much did we lose? Everything written since the last good copy is gone.
  2. How long were we down? From the moment it broke to the moment it worked again.

You can’t choose when disaster strikes, but you can choose, in advance, how bad each answer is allowed to be. Those choices have names.

Two numbers, decided before anything breaks

NIST’s contingency planning guide, SP 800-34, defines them:

  • Recovery point objective (RPO): “the point in time to which data must be recovered after an outage”. In practice, the most data you can afford to lose, measured in time. An RPO of 1 hour means you must never lose more than the last hour’s work.
  • Recovery time objective (RTO): “the overall length of time an information system’s components can be in the recovery phase before negatively impacting the organization’s mission or mission/business processes”. In practice, how long you can be down.

Behind both sits a third number from the same guide, maximum tolerable downtime: “the amount of time mission/business process can be disrupted without causing significant harm”. That’s the business’s limit. RTO is the target you plan to, so it has to fit inside it.

Visual guideOne disaster, two numbers
One disaster, two numbersA timeline. The last backup ran at 02:00 on Wednesday. The disaster hits at 14:50; everything written between 02:00 and 14:50 is lost, and that gap is what RPO limits. Recovery runs until 17:50; that downtime is what RTO limits.last backupWed 02:00disasterWed 14:50back onlineWed 17:50RPOdata you can loseRTOtime you can be downdata at riskrecoveringmeasuresmeasuresOne disaster, two numbersA timeline. The last backup ran at 02:00 on Wednesday. The disaster hits at 14:50; everything written between 02:00 and 14:50 is lost, and that gap is what RPO limits. Recovery runs until 17:50; that downtime is what RTO limits.last backupWed 02:00disasterWed 14:50back onlineWed 17:50RPOdata you can loseRTOtime you can be downdata at riskrecoveringmeasuresmeasures
  1. The last good copy of the data was taken at 02:00. Everything after it exists in one place only.
  2. At 14:50 that one place fails. Nearly thirteen hours of orders, edits and payments are gone.
  3. RPO caps that gap. Here the backup schedule decides it: nightly backups can lose up to a full day.
  4. Then the clock runs until service is back. RTO caps that: how fast you can restore, not how fast you would like to.

AWS’s disaster recovery whitepaper puts the first half plainly: how often you run your backup “will determine your achievable recovery point”. The second half is the same idea: your recovery method determines your achievable recovery time.

The business sets them, per system

It’s tempting for IT to pick numbers that sound responsible (“everything backed up every 15 minutes!”). But the right RPO for online orders and the right RPO for the marketing site are different, because losing an hour of each costs very different amounts. So the targets come from the people who know what an hour of each system is worth, and IT’s job is to meet them at the lowest cost.

Try it with three systems. Set a backup schedule and a recovery method, then let the disaster hit.

One disaster, two numbersillustrative monthly prices

safe in the last backuplost since itdown while recovering
  • Pick a system and a setup, then let the disaster hit.
Check yourself+10 XP

Backups run every night at 02:00. What RPO can this schedule promise?

Check yourself+10 XP

Who should decide that online orders may lose at most 1 hour of data?

Check yourself+10 XP

A system's maximum tolerable downtime is 8 hours. Which RTO makes sense?

Sources

Finished reading?

Missions and quizzes above earn their own XP. This marks the lesson done.