Automating Kubernetes backups: a step-by-step guide

A backup that's never been tested for restore isn't, in any practical sense, a backup. That's where we start with every backup setup.

For stateful workloads in Kubernetes we use CSI-level volume snapshots plus logical database dumps — the snapshot is fast, but a logical dump is easier to verify and restore selectively.

The backup schedule is defined declaratively and lives in the same repository as the rest of the infrastructure — so backup configuration never lives only in one engineer's head.

A critical step is storing backups outside the cluster and outside the same cloud account running production: an unavailable cluster shouldn't also mean unavailable backups.

Once a quarter we restore a backup into an isolated environment and confirm the application actually starts on the restored data — without that step, there's no real confidence in the backups.