A backup is a recovery claim

A successful backup job proves that a program wrote something somewhere. It does not prove that the copy contains one coherent state, that the decryption key is available, that the restore instructions still match the software, or that the returned service will satisfy a user.

Treat each backup as a claim about a future recovery. The claim becomes credible only when a clean environment can use the stored copy, the documented dependencies, and the available credentials to produce a service that passes known checks within an acceptable time.

Define the recovery outcome

“Back up the server” is not a useful boundary. A server may contain replaceable packages, generated caches, durable application state, credentials held elsewhere, and configuration that is authoritative in a repository. Copying all of it can make recovery slower without making it more complete.

Name the result that has to return. For a public service, that may be a reachable endpoint with accounts, records, and configuration no older than a stated point. For a workstation, it may be a small set of documents and settings on replacement hardware. The required result tells you what belongs in the recovery set.

Service boundary

The user-visible capability that must return

Recovery point

The maximum amount of recent change that may be lost

Recovery time

The deadline from declaring recovery to verified service

Recovery set

Data, configuration, software versions, keys, and dependencies required together

Authority

Who can declare, perform, verify, and end a recovery

Acceptance proof

The checks that distinguish a restored service from copied files

Keep this contract beside the system’s one-page service record. It should name exact locations and owners, not merely the backup product. A product can be healthy while the recovery set is incomplete.

Capture a consistent state

A filesystem copy taken while an application is writing may contain each file successfully and still represent no moment the application ever occupied. A database dump, coordinated snapshot, or brief write pause gives the copy a defined consistency boundary. The right mechanism depends on the data store; the requirement does not.

Record the boundary with the copy: start and finish time, application or schema version, source identity, snapshot or transaction position where available, included components, excluded components, byte size, and a digest or provider identifier. This manifest lets a future operator reject an incomplete or mismatched set before spending the recovery window on it.

• Coordinate related state. Capture data that must agree under one application-aware boundary.
• Include restore prerequisites. Preserve the configuration, schema tools, software version, and key references needed to interpret the data.
• Exclude rebuildable bulk. Caches, downloaded packages, and generated output need instructions, not repeated copies.
• Fail incomplete jobs. A missing component must make the recovery set unavailable rather than quietly partial.
• Make the manifest immutable with the set. Evidence that can change independently cannot identify what was captured.

Keep an independent recovery path

The failure that damages production may also damage its backups. A mistaken deletion can follow a synchronized path; compromised administrator credentials can reach every account they control; an expired subscription can remove both runtime and storage. At least one recovery copy must sit beyond the ordinary production failure and authority boundary.

Independence is more than another bucket name. It may mean a separate account, write-once retention, offline media, or credentials unavailable to the production runtime. The exact mechanism can stay small, but the production service should not be able to rewrite or delete every usable recovery point.

Failure boundary

A provider, device, region, or account failure does not remove every copy

Authority boundary

Production credentials cannot alter the protected copy or its retention

Key boundary

Decryption material remains available if the production secret store is lost

Access path

An operator can retrieve the set without depending on the failed service

Observation

Backup failures and destructive actions report outside the protected system

Test the emergency credentials before the incident, store their recovery procedure where the failed system cannot hide it, and limit their routine use. An isolated copy that nobody can decrypt is only a different kind of loss.

Restore somewhere clean

Restoring over the original machine hides assumptions. Old packages, cached credentials, hand-edited configuration, and surviving data can make an incomplete procedure appear successful. A clean target forces the recovery set and runbook to provide everything they claim to provide.

1. Choose one recovery point. Select it by the contract and manifest, not simply by the newest timestamp.
2. Provision an isolated target. Keep it unable to send production email, run scheduled jobs, or accept real traffic.
3. Fetch with emergency access. Exercise the independent credentials, network path, and decryption keys.
4. Restore in dependency order. Recreate durable stores, configuration, application versions, and derived state deliberately.
5. Apply forward changes explicitly. Run supported migrations from the captured version; do not assume the newest runtime can read old state directly.
6. Record every manual correction. Put it into the runbook or recovery set before calling the drill complete.

The clean target can be smaller than production when capacity is not under test. It must still exercise the same formats, dependencies, identities, and application path. A partial sample is useful only when the sampling rule preserves the relationships that verification needs.

Verify the returned service

File counts and digests prove that bytes arrived unchanged. They cannot prove that relationships survived, credentials work, indexes are usable, or the application interprets the restored state correctly. Verification must climb from storage evidence to service behavior.

• Check structure. Confirm expected schemas, versions, files, permissions, and required configuration.
• Check relationships. Validate constraints, reference counts, and a few known records across dependent stores.
• Check application reads. Use normal interfaces to retrieve representative old and recent data.
• Check a disposable write. Create, read, update, and remove a synthetic record without touching real external systems.
• Check identity. Prove that intended accounts can authenticate and unintended recovery-only paths remain closed.
• Check the public contract. Before any traffic switch, repeat the relevant outside-facing checks against the isolated target.

Keep the expected markers in the recovery contract: a few stable record identities, minimum counts or ranges, a known configuration value, and the exact health behavior. Avoid a checklist that can pass on an empty but syntactically valid system.

Measure the whole clock

Restore duration is not recovery duration. The user waits through detection, the decision to recover, access to credentials, target provisioning, data transfer, decryption, restoration, migrations, verification, and the final traffic change. Measure from the declared start to verified service, with each interval recorded separately.

Detect

Recognize that normal repair will not meet the service deadline

Authorize

Choose the recovery point and grant emergency access

Prepare

Provision an isolated target and required network paths

Restore

Retrieve, decrypt, load, migrate, and rebuild the recovery set

Verify

Run data, application, identity, and outside-contract checks

Return

Switch traffic, observe real use, and close emergency access

Run the drill often enough that software, people, and credentials cannot drift beyond the deadline. Rotate the operator occasionally. A procedure that works only from its author’s laptop contains an undocumented dependency.

Retire copies deliberately

Retention is part of the recovery design. Keeping every copy forever increases storage cost, secret exposure, and the chance that obsolete personal or regulated data survives beyond its intended life. Deleting too early removes the older points needed to escape delayed corruption or unnoticed loss.

Set retention from the failure window and the contract: how long corruption may remain unnoticed, which historical points are useful, and which obligations require or forbid continued storage. Expire the manifest, data, replicas, and unique keys as one tracked action. Verify provider-side deletion when the copy crosses an account or service boundary.

• The recovery outcome, acceptable loss, deadline, owner, and proof are written down.
• Each usable copy represents one consistent, manifested recovery set.
• At least one copy and its key survive the production failure and authority boundary.
• A clean target can be restored without hidden state from the original machine.
• Verification reaches application behavior, not only files and job status.
• The full recovery clock has been measured by more than one operator.
• Retention and retirement preserve useful history without keeping data indefinitely.

The useful question is not “Did the backup run?” It is “Which service can this exact set recover, how much will be lost, how long will it take, and what evidence says the result is ready?” Test that answer while the original system is still available to correct it.