Your backup job says OK. That doesn't mean you can restore

What "OK" actually means

When a backup job ends with exit code 0, all you know is that a process ran and wrote some bytes somewhere. You don't know whether the bytes are complete, whether they hang together, whether anyone can still read the destination, or whether you could turn them back into a working system before the business starts to hurt.

The ways it goes wrong are depressingly ordinary:

• A database was copied as live files while it was being written to. The copy won't start.
• Someone added a new upload directory last spring. The backup job was set up before that and never heard about it.
• The encryption key for the backup only exists on the server being backed up.
• The backup disk filled up three weeks ago, and the retention policy has been quietly deleting the last good copies since.
• The restore works, but takes nine hours, and the business assumed two.

None of that shows up in a job log. All of it shows up the first time you try a restore.

Two numbers per system

Before setting up any schedule, agree two figures for each system:

The first one decides how often you back up. The second decides how you back up, and you can only check it by timing a real restore.

Copies that hang together

Copy databases through the database, never by copying its files from underneath it:

# MySQL / MariaDB, InnoDB tables: a consistent snapshot without locking writes
mysqldump --single-transaction --routines --triggers --events \
          --default-character-set=utf8mb4 shop_db | zstd -T0 > shop_db.sql.zst

# PostgreSQL: custom format, so single tables can be restored
pg_dump --format=custom --file=app_db.dump app_db

--single-transaction only gives a consistent view for transactional engines. If an older shop still has MyISAM tables, they need a lock or a conversion.

For files we like restic. It deduplicates, so keeping a lot of versions is cheap, and it can check its own repository:

restic -r sftp:backup@offsite:/srv/restic/shop backup /var/www/shop \
       --exclude /var/www/shop/var/cache
restic -r sftp:backup@offsite:/srv/restic/shop check --read-data-subset=5%

3-2-1, plus one copy they can't reach

The old rule still holds: three copies, on two kinds of storage, one of them somewhere else. Ransomware adds a fourth requirement. At least one copy has to be out of reach of the server's own credentials, so either the storage refuses to change or delete old data (append-only, object lock), or the backup machine pulls the data and the production server has no way of logging in to it.

How we run a restore test

The routine is the same every time:

1. Pick a backup. Not always last night's.
2. Restore it somewhere isolated: a scratch virtual machine or a separate database instance. Never on the production host.
3. Start the application or query the database, and run the same checks each time. For a shop: count orders and customers, look at the timestamp of the latest order, load one product page.
4. Time every step, from deciding to restore to a system that works.
5. Throw the test system away.
6. Write it down.

What we write down

The record is short and always looks the same:

Two lines on that sheet matter more than the rest. The latest order tells you the real recovery point. The time taken tells you the real recovery time. If either is worse than what was agreed, we change the backup setup. A note in the record isn't enough.

How often

Every month for systems with transactions in them. Every quarter for the rest. And after any change to the backup configuration, the storage or the way the application stores its data. If a restore test has never failed, it's worth asking whether it has ever been tried on the hard case: rebuilding a whole server from nothing but the backup and the runbook.