Writing bytes and recovering a system are different claims
Every backup product ships a dashboard with green checkmarks. A green checkmark means a job ran and bytes landed somewhere. It does not mean you can rebuild a working system from those bytes, on hardware you do not currently have, inside a window the business will tolerate. Those are separate claims, and the dashboard only tests one of them.
This is the most over-trusted signal in infrastructure. Nobody argues with green. Nobody verifies it either, until the day the restore matters, and on that day it is usually not the backup that fails. It is everything around the backup: the encryption key nobody kept a copy of, the service startup order nobody wrote down, the copy speed nobody measured.
A backup you have never restored is an untested assumption. It might be correct. You simply have no evidence either way, and you are choosing to find out during an outage.
Decide RTO and RPO before you evaluate anything
Two numbers make a backup design evaluable. Without them, every conversation about backup is aesthetics.
RTO, recovery time objective: how long the business can be down before the damage becomes unacceptable. Four hours. Twenty-four hours. Three days. It is a duration.
RPO, recovery point objective: how much recent work the business can afford to lose. If you back up nightly at 02:00 and the server dies at 18:00, you have lost sixteen hours of work. That is your real RPO, whether or not anyone agreed to it.
Both numbers are business decisions, not technical ones. The engineer's job is to state honestly what each target costs and what the current setup actually delivers. A finance team that loses a day of transactions has a different problem than a marketing site that loses a day of blog edits, and they should not be paying for the same design. Once the numbers exist, they drive everything else: replication frequency, where the copies live, and how much standby capacity you keep warm. That is an architecture decision, not a backup software decision.
The uncomfortable version of this conversation is the useful one. Most organisations discover that their stated RTO is four hours and their actual capability is somewhere between two days and unknown.
3-2-1, and the copy your admin account cannot delete
The old rule still holds: three copies of the data, on two different media or systems, with one copy off-site. It survived because it defends against the failure modes that actually recur — a disk dies, a filesystem corrupts, a building floods, someone deletes the wrong directory.
What it did not anticipate is an attacker with your credentials. Ransomware operators do not merely encrypt production; they look for the backup system first, because a company that can restore does not pay. If your backup server is domain-joined, reachable from the compromised network, and administered by the same account that just got phished, you do not have three copies. You have three copies of one blast radius.
So add a fourth condition: at least one copy must be immutable and outside the authority of your primary credentials. In practice that means object storage with an enforced retention lock, an append-only repository where the backup client holds write-but-not-delete rights, or a replication target that pulls from production rather than accepting pushes into it. Pull-based replication is underrated here — the backup host initiates the transfer and production never holds a credential that can reach into the archive.
# Snapshot locally, then let the backup host pull the stream.
# Production never holds a credential that can delete the replica.
zfs snapshot -r tank/data@2026-07-26
zfs send -RI tank/data@2026-07-25 tank/data@2026-07-26 | \
ssh backup-host zfs receive -F tank/replica/data
The test for immutability is blunt: if your most privileged account can delete a copy today, an attacker holding that account can delete it too.
What actually breaks restores
In practice, failed recoveries cluster into a short list of causes, and almost none of them are "the backup file was corrupt."
Missing keys. Encrypted backups are correct practice, and the passphrase is stored in the password manager that ran on the server you are trying to restore. Keys must live somewhere that survives the loss of the environment they protect, and someone other than one person must be able to reach them.
Undocumented restore order. Modern systems are interdependent. The application will not start without the database, the database will not accept connections without its certificate, the certificate is issued by an internal CA that itself needs restoring, and the queue workers fail silently until the cache layer is up. None of this is hard. It is just unknown at 03:00 unless someone wrote it down, and it changes every time the platform changes, which is why restore documentation belongs under the same change control discipline as the infrastructure it describes.
Databases copied while running. Copying a live database directory with a file-level tool produces a file that restores cleanly and then refuses to start, or worse, starts with subtle inconsistency. Use the database's own mechanism, or snapshot a filesystem that gives you a crash-consistent point-in-time image.
# Logical dump: consistent across the whole database, single transaction
pg_dump --format=custom --compress=6 \
--file=/var/backups/appdb-$(date +%Y%m%d).dump appdb
# Physical base backup: the correct starting point for point-in-time recovery
pg_basebackup --pgdata=/var/backups/base --format=tar --gzip --wal-method=stream
Retention that quietly expired the copy you needed. Some damage is discovered late — a corrupted table, a bad migration, a slow-moving data deletion. If retention is thirty days and discovery takes forty, every copy you hold is already poisoned. Retention should be set against how long problems take to surface, not against storage cost alone.
Restores that work but take four days. This is the most common one. The data is intact, the process is sound, and pulling several terabytes back over the available link takes longer than the business assumed by an order of magnitude. Nobody measured it. Restore throughput is a hard number you can obtain in an afternoon, and it is frequently the number that invalidates the whole plan.
Two of these — expired retention and jobs that silently stopped running weeks ago — are detectable in advance if something is watching the backup system itself rather than just its dashboard. A backup that has not run in eleven days should page someone, which is exactly the kind of thing continuous monitoring exists to catch.
The drill: measure wall-clock time, do not estimate it
A restore drill is a scheduled, timed rebuild of a real system from backup onto infrastructure that is not the original. Restoring a deleted file proves file-level retrieval works. It proves nothing about your ability to rebuild the host, and the host is what you lose.
Cadence depends on how much change the system absorbs. Quarterly is a reasonable default for a business-critical system; twice a year is the floor. What matters more than frequency is that the drill starts from the same position a real disaster does: fresh infrastructure, no shell access to the original, no tribal knowledge in the room that is not also in the runbook.
Verify the archive before you rely on it, then restore into scratch space rather than over anything live:
# Verify the repository by actually reading the data, not just the index restic -r "$REPO" check --read-data # Restore into disposable scratch space, never over a live path mkdir -p /mnt/restore-drill restic -r "$REPO" restore latest --target /mnt/restore-drill
Then start a clock and do not stop it for anything, including the fifteen minutes spent finding a password. That interruption is part of your recovery time. The output of a drill is one honest number — wall-clock minutes from "begin" to "the application serves a real request" — compared against the RTO the business asked for. The rebuild step will also expose how much of your platform is reproducible from configuration versus assembled by hand over years, which is worth knowing regardless of which operating system you standardised on.
A restore drill record you can copy
Keep this in the repository next to the runbook, one file per system, updated at every drill. The two fields that matter most are at the bottom: the measured time and the date last tested.
RESTORE DRILL RECORD ==================== System under test: ____________________ Drill date: ____________________ Performed by: ____________________ Target RTO: ________ Target RPO: ________ PRE-FLIGHT [ ] Restore target is fresh infrastructure, not the original host [ ] Encryption keys retrieved from their documented location [ ] Drill credentials used, not production admin credentials [ ] Source is the off-site immutable copy, not the local one [ ] Runbook opened and followed as written (no improvising from memory) TIMED RUN (record wall clock, not elapsed guesses) Start ________ Base OS provisioned ________ Data restored ________ Services started in documented order ________ Application serving a real request ________ Stop ________ RESULT Measured restore time: ________ vs target RTO: ________ Data loss at cutover: ________ vs target RPO: ________ PASS / FAIL: ________ CORRECTNESS CHECKS [ ] Record counts match the source at snapshot time [ ] A known recent record is present [ ] Cron jobs and queue workers run [ ] Outbound integrations authenticate (keys restored, not just files) [ ] TLS certificates present and valid FINDINGS Undocumented steps discovered: ______________________ Steps slower than expected: ______________________ Missing artifacts (keys, certs, config, DNS): __________ Runbook updates required: ______________________ DATE LAST TESTED: __________ NEXT DRILL DUE: __________
A system whose "date last tested" field is blank does not have a backup. It has a backup job.
If you have nothing, start here
Do not write a disaster recovery programme. Programmes take a quarter to approve and produce a document nobody reads.
Pick the single system whose loss would hurt most. This month, restore it onto a scratch machine, start to finish, using only what is written down. Time it with a clock. Write down the measured number, the date, and every artifact you did not have when you needed it.
That one afternoon converts your backup from an assumption into a measurement, and the list of things you were missing is the actual work plan. Everything else — extending drills to other systems, tightening RPO, making a copy immutable — follows from a number you now have instead of one you were hoping for.
Want a second set of eyes on your recovery position? See how we approach infrastructure design or schedule a consultation.