Database Backup and Restore

The db-backup Helm chart (manifests/apps/db-backup) backs up the CNPG Postgres cluster (nhn-postgres-cluster) and provides an on-demand restore path.

A CronJob dumps the entire cluster (all databases and roles) into a shared PVC every sixth hour. A SnapshotSchedule then takes a Trident volume snapshot of that PVC, providing a retention window of older dumps.

CNPG cluster (nhn-postgres-cluster)
   │  pg_dumpall -c every 6h
   │  (CronJob psql-backup)
   ▼
Backup PVC (pvc1-psql-volumesnap-backup, 4Gi, sc-trident-nfs)
   │  Trident volume snapshot, 10 min after each dump
   │  (SnapshotSchedule snapschedule-psql-backup)
   ▼
VolumeSnapshots (trident-csi-snapclass)
   │  On-demand restore
   ▼
Restore PVC → restore pod (psql -f) → CNPG cluster

Components

On-demand restore resources (not templated; applied manually when needed):

Backup

CronJob (psql-backup)

Runs the Bitnami Postgres client image and writes a cluster-wide logical dump:

PGPASSWORD=$POSTGRES_PASSWORD pg_dumpall -c \
  -f "/backup/pgbackup-$(date +%Y%m%d%H%M).pgdump" -v \
  -h nhn-postgres-cluster-rw.nhn-postgres-cluster.svc.cluster.local \
  -U $POSTGRES_USER \
&& ls -1t /backup/pgbackup*.pgdump | tail -n +4 | xargs -r rm -f

Each dump is written to a date-stamped file pgbackup-<YYYYMMDDHHMM>.pgdump (UTC, matching the snapshot naming convention). After a successful dump, the job keeps only the 3 newest files on the PVC and deletes the rest — long-term retention comes from the snapshots, so the PVC only needs to hold 3 dumps.

The -c (clean) flag makes the dump include DROP statements before each object, so a restore recreates roles and databases exactly as they existed at backup time.

Client/server version: The Bitnami client major version should match the CNPG server's Postgres version. Check the server version with:

kubectl -n nhn-postgres-cluster get cluster nhn-postgres-cluster \
  -o jsonpath='{.spec.postgresql.version}'

Backup PVC

pvc1-psql-volumesnap-backup is a 4Gi ReadWriteMany claim on sc-trident-nfs, labeled postgres: backup — that label is what the SnapshotSchedule selects on. The PVC holds the 3 newest date-stamped dumps, so size it accordingly (increase it in templates/pvc-backup.yaml if 3 dumps approach 4Gi).

Snapshot schedule (snapschedule-psql-backup)

Takes a Trident snapshot of the labeled backup PVC 10 minutes after each dump (10 */6 * * *), using snapshot class trident-csi-snapclass.

Both retention limits apply; since snapshots are created every 6 hours, maxCount: 5 is the effective limit — the oldest snapshot is at most ~24 hours old. If a longer window is needed, raise maxCount (or lower the dump frequency).

Note that the SnapshotSchedule is independent of the CronJob: if a dump fails, the next snapshot still runs and captures the previous (or partial) dump file. Monitor dump success (see Operations).

Credentials

The psql-backup-credentials secret (keys username, password) holds the CNPG superuser credentials. It is created by manifests/scripts/bootstrap-db-secret.sh, which copies them from the cluster secret nhn-postgres-cluster-superuser in the nhn-postgres-cluster namespace:

# Interactive (prompts for namespace, default db-backup-test)
./manifests/scripts/bootstrap-db-secret.sh

# Or pass the namespace explicitly
./manifests/scripts/bootstrap-db-secret.sh db-backup-test
BACKUP_NAMESPACE=db-backup-test ./manifests/scripts/bootstrap-db-secret.sh

The secret must exist in the namespace of both the CronJob (backup) and the restore pod (restore) — normally the same namespace. Restoring requires superuser privileges because pg_dumpall output contains role definitions and drops/recreates all databases.

Deployment

The chart is deployed by Argo CD through the App of Apps (manifests/appOfApps). Each environment's values.*.yaml lists the enabled applications:

The environment values files for the chart itself (values.dev.yaml, values.test.yaml, values.prod.yaml) are currently empty, so the chart defaults in values.yaml apply:

Namespace note: The chart renders its resources into .Values.namespace (default db-backup), while the on-demand restore manifests in restore/ hardcode the namespace db-backup-test. Before a restore, verify where the backup resources actually live in your environment and adjust the restore manifests' namespace fields to match:

kubectl get cronjob psql-backup -A
kubectl get pvc pvc1-psql-volumesnap-backup -A

Restore

Warning: A restore is destructive. The dump was created with pg_dumpall -c, so replaying it drops and recreates all roles and databases as they existed at backup time. Any data, roles, or schema changes made after the backup point are lost. Restore only into a disposable/test cluster, or after confirming the current state can be discarded.

Prerequisites

• The trident-csi-snapclass VolumeSnapshotClass exists (cluster-scoped; see restore/volumesnapshotclass.yaml).
• The psql-backup-credentials secret exists in the restore namespace.
• You know which snapshot to restore from (see step 1).

Step 1 — Choose a snapshot

kubectl -n db-backup-test get volumesnapshots
# NAME                                                              READY   SOURCEPVC                              RESTORESIZE   AGE
# pvc1-psql-volumesnap-backup-snapschedule-psql-backup-202609181210 true    pvc1-psql-volumesnap-backup            1Gi         6h

Pick the snapshot with the YYYYMMDDHHMM timestamp you want to restore.

Step 2 — Create the restore PVC from the snapshot

Edit manifests/apps/db-backup/restore/pvc-restore.yaml and set dataSource.name to the chosen snapshot, then apply:

spec:
  dataSource:
    name: pvc1-psql-volumesnap-backup-snapschedule-psql-backup-202609181210  # chosen snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

# If a restore PVC from a previous restore exists, delete it first
kubectl -n db-backup-test delete pvc pvc1-psql-volumesnap-restore --ignore-not-found

kubectl -n db-backup-test apply -f manifests/apps/db-backup/restore/pvc-restore.yaml

# Wait until the PVC is Bound
kubectl -n db-backup-test get pvc pvc1-psql-volumesnap-restore -w

Step 3 — Run the restore pod

Optionally scale down the API and terminate active connections first, so that DROP DATABASE/CREATE DATABASE statements are not blocked by open sessions.

kubectl -n db-backup-test apply -f manifests/apps/db-backup/restore/pod-restore.yaml

# Follow the replay (psql prints NOTICE/error messages as the script runs)
kubectl -n db-backup-test logs -f psql-restore

# The pod should end in Completed
kubectl -n db-backup-test get pod psql-restore

The pod runs restartPolicy: Never and replays the newest date-stamped dump found on the restore PVC (it falls back to the legacy fixed name pgbackup.pgdump for snapshots taken before date-stamped dumps existed):

f=$(ls -1 /restore/pgbackup-*.pgdump 2>/dev/null | sort | tail -n1);
[ -z "$f" ] && f=/restore/pgbackup.pgdump;
PGPASSWORD=$POSTGRES_PASSWORD psql \
  -h nhn-postgres-cluster-rw.nhn-postgres-cluster.svc.cluster.local \
  -U $POSTGRES_USER -f "$f"

Step 4 — Verify

# Port-forward to the cluster through the pg-proxy pod (local port 9999)
./manifests/scripts/db-port-forward.sh

# In another terminal:
psql -h localhost -p 9999 -U <superuser> -c "\l"   # databases present
psql -h localhost -p 9999 -U <superuser> -c "\du"   # roles present

Step 5 — Clean up

kubectl -n db-backup-test delete pod psql-restore --ignore-not-found
kubectl -n db-backup-test delete pvc pvc1-psql-volumesnap-restore

Operations

Failure modes: