Backup & Restore

How to back up a Definite On-Prem deployment and restore it. This is a first-version runbook — validate it against your own environment and bake it into your normal DR process.

What state exists, and who owns it

Definite On-Prem keeps durable state in three places. The deployment itself — the API, frontend, and job-runner pods — is stateless and re-created from the Helm chart, so it does not need backing up.

The two halves of the lakehouse must be backed up as a consistent set: the object store holds the Parquet data files, and the DuckLake catalog holds the metadata — snapshots, table definitions, and the file manifest — that makes those Parquet files queryable. The catalog is the dedicated ducklake_catalog Postgres database, so both lakehouse halves are external state. A backup of one without the other is not restorable. See architecture.md for the component map.

There is no app-managed backup job. Definite does not ship a backup CronJob or any in-cluster backup component. Backup and DR are an infrastructure concern: back up your Postgres instance (it holds both the application definite database and the ducklake_catalog catalog database) with your provider's automated backups / PITR, and rely on object-store durability + versioning for the Parquet data.

Before you back up

A point-in-time-consistent backup is easiest when nothing is writing. For a clean backup:

1. Pause scheduled work — stop the job runner so automations and agent runs do not write mid-backup:

   kubectl scale deploy/definite-job-runner --replicas=0 -n definite
   

2. Optionally scale the API to zero as well for a fully quiesced backup. For a hot backup, skip this and accept that in-flight queries may not be captured; Postgres dumps and object stores are individually consistent, but the catalog/data set is only guaranteed consistent when writes are paused.

3. Run a DuckLake checkpoint so catalog state is flushed cleanly:

   definite run maintenance run --operation checkpoint
   

Backup procedure

1. Postgres (application and DuckLake catalog databases)

Use pg_dump. Run it from a host that can reach the database (a jump pod, or your laptop if Postgres is routable).

pg_dump \
  --format=custom \
  --no-owner \
  --no-privileges \
  --file=definite-pg-$(date +%Y%m%d-%H%M%S).dump \
  "postgres://definite:${POSTGRES_PASSWORD}@<host>:5432/definite"

pg_dump \
  --format=custom \
  --no-owner \
  --no-privileges \
  --file=ducklake-catalog-pg-$(date +%Y%m%d-%H%M%S).dump \
  "postgres://definite:${POSTGRES_PASSWORD}@<host>:5432/ducklake_catalog"

Store the .dump file off-cluster. If your Postgres is a managed service (RDS, Cloud SQL, Azure), the provider's automated daily snapshots are an acceptable alternative — and on a correctly provisioned deployment they are already on by default (see prerequisites.md for the per-provider setup). Verify the automated backups include both the definite and ducklake_catalog databases and have a non-zero retention period.

2. Object store (lakehouse bulk data)

Copy the deployment's prefix to a separate backup location. Use the same prefix you set as lakehouse.prefix.

# S3 / MinIO
aws s3 sync s3://acme-definite-lake/lake/ s3://acme-definite-backup/lake-$(date +%Y%m%d)/

# GCS
gsutil -m rsync -r gs://acme-definite-lake/lake/ gs://acme-definite-backup/lake-$(date +%Y%m%d)/

Object-store versioning or a lifecycle-managed backup bucket is the most robust option — enable it if you can.

3. DuckLake catalog metadata

No separate step. The DuckLake catalog lives in the ducklake_catalog Postgres database, so it is captured by the Postgres dump in step 1 — back it up together with the application database (or in the same low-write window) so the catalog and the object-store data stay a consistent set.

4. Config

Keep a copy of config.yaml and the values of every secret env var it references (POSTGRES_PASSWORD, S3_ACCESS_KEY_ID, OIDC_CLIENT_SECRET, …) in your secrets manager. Without these you cannot re-deploy.

5. Resume

kubectl scale deploy/definite-job-runner --replicas=1 -n definite
# and the API if you scaled it down

Before an upgrade

definite upgrade (and definite init on an existing deployment) applies any pending schema migrations to the application database on API boot. A failed migration can leave the schema in a half-applied state, so always take an on-demand Postgres backup immediately before upgrading — a fresh, named snapshot, independent of the rolling automatic-backup retention.

# RDS
aws rds create-db-snapshot \
  --db-instance-identifier <instance> \
  --db-snapshot-identifier <instance>-preupgrade-$(date +%Y%m%d-%H%M%S)

# Cloud SQL
gcloud sql backups create --instance=<instance> \
  --description="pre-upgrade $(date +%Y%m%d-%H%M%S)"

# Azure Database for PostgreSQL — trigger an on-demand backup from the portal
# or `az postgres flexible-server backup create`.

# Self-managed — run the pg_dump from "Backup procedure" step 1 above.

Keep the snapshot until the upgraded deployment has been verified healthy (definite status, a known query, and a spot-check of application data).

Restore procedure

Restore in dependency order: external state first, then the catalog, then bring the deployment up.

1. Restore Postgres

Into clean, empty definite and ducklake_catalog databases:

pg_restore \
  --no-owner \
  --no-privileges \
  --dbname="postgres://definite:${POSTGRES_PASSWORD}@<host>:5432/definite" \
  definite-pg-<timestamp>.dump

pg_restore \
  --no-owner \
  --no-privileges \
  --dbname="postgres://definite:${POSTGRES_PASSWORD}@<host>:5432/ducklake_catalog" \
  ducklake-catalog-pg-<timestamp>.dump

2. Restore the object store

Sync the backed-up prefix back to the bucket the new deployment will use. The object paths must match what the catalog expects — restore to the same prefix the catalog was taken with, or you must rewrite catalog paths.

3. Restore the DuckLake catalog

The catalog and the object-store data must come from the same backup point. The catalog is the ducklake_catalog Postgres database, so it was restored in Postgres step 1 — no separate catalog restore is needed.

4. Re-deploy

definite doctor --config config.yaml
definite init --config config.yaml

definite init runs helm upgrade --install; the API applies any pending schema migrations on boot.

5. Verify

• definite status --config config.yaml — all pods Running/Ready.
• definite run maintenance stats — every expected lakehouse table is listed with sane snapshot counts.
• A known query through the UI or definite run returns expected rows.
• Spot-check application data: recent automations, saved queries, integrations.

Recovery objectives & cadence

Set these with your customer; sane starting points:

• Postgres: dump at least daily; managed-service PITR if available. This covers both the application and ducklake_catalog databases.
• Object store: enable bucket versioning and/or sync the bucket on the same schedule as the Postgres backup, ideally during a low-write window so the catalog and the data stay consistent.
• Test restores quarterly into a scratch namespace. A backup you have never restored is not a backup.

Back up your Postgres instance (the catalog story)

The DuckLake catalog is the ducklake_catalog Postgres database, living on the same Postgres instance as the application definite database. There is no app-managed catalog backup job — the catalog is covered by whatever you use to back up that Postgres instance, exactly as for the application database:

• Managed Postgres (RDS / Cloud SQL / Azure): automated daily backups and point-in-time recovery cover both databases on the instance. On a correctly provisioned deployment these are already on by default — see prerequisites.md for the per-provider setup. Verify the retention is non-zero and that the snapshots include the whole instance (so both definite and ducklake_catalog).
• Self-managed Postgres: run the pg_dump of both databases from step 1 of the manual procedure above (or use pgBackRest / WAL archiving for PITR).

To restore the catalog, restore the Postgres instance (or PITR it to a point in time) — the ducklake_catalog database comes back with it. See the restore procedure above; no separate catalog-restore step is needed.

Don't forget the object store

The catalog references the Parquet data files by path. Keep the bulk data durable independently of Postgres:

• Enable bucket versioning (or a lifecycle-managed backup bucket) on the prefix that holds the lake's Parquet files. Without versioning, a delete (or a partial sync) makes a catalog backup non-restorable — the catalog points at files that no longer exist.
• For a strict point-in-time set, line up the Postgres snapshot/PITR target and any bucket-versioning checkpoint in the same low-write window so the catalog and the data stay a consistent pair.

Notes & limitations

• There is no app-managed backup job and no backup component in the chart. Postgres and bulk-data backups are an infrastructure responsibility — use managed-Postgres automated backups / PITR (covers both the application and catalog databases) plus object-store versioning, as described above.
• There is no single definite backup CLI command yet — for a fully manual point-in-time backup of all pieces, the documented procedure above remains the authoritative path.
• Air-gapped restore is out of scope for v1.
• If catalog and object-store backups drift out of sync, the catalog is the source of truth for which files should exist; a definite run maintenance pass (stats, then delete_orphaned_files) can help reconcile a partially restored object store.
• For escalation when a restore does not come up clean, see support.md.