Compute Profiles
Compute profiles are named burst tiers the operator declares in
config.yaml. A user, a Fi thread, or a pipeline Python step picks a profile
by name at run time, and the workload runs on a Kubernetes pod sized for that
tier — usually on a dedicated node pool that scales to zero when idle, so big
compute costs nothing while nobody's using it.
The deployment ships with one profile, default, sized at the small
always-on fi.sandbox.resources defaults. The operator adds bigger profiles
(large, xlarge, gpu, …) by declaring them in config.yaml.
When to use a profile
| Use case | Profile path | What runs where |
|---|---|---|
| Heavy Python step in an automation (pandas joins, ML training, dbt-style transforms) | python step config.compute_profile | Job-runner leases a Sandbox of that profile and POSTs the script + requirements to it; the script runs on the burst pod, not in the job-runner. |
| SQL transformation model or automation SQL step | Transformation project/model Compute setting, or sql step config.compute_profile | The generated automation step carries the profile. Default runs on the job-runner's attached DuckLake connection; non-default dispatches through the burst SQL sandbox path. |
| Scheduled / ad-hoc DuckLake maintenance (expire_snapshots, cleanup_old_files, compact, …) | Settings → Maintenance → Compute profile (schedule-level), or compute_profile on POST /api/v1/lakehouse/maintenance/runs | The profile is stamped onto every maintenance step. Default runs in the job-runner pod's attached DuckLake connection; non-default runs each ducklake_* / CHECKPOINT statement on a disposable sandbox. |
| Heavy data-app dashboard query | App-level "compute_profile" in the app's app.json manifest | Each SQL resource query dispatches to a burst sandbox of that tier (requires Fi). Falls back to the API pod's attached DuckLake connection when the profile can't be honored. Interactive — see the cold-start caveat below. |
Fi session that crunches a big dataset in the agent's bash tool | Thread menu Compute profile, or compute_profile field on POST /api/v1/fi/threads/{id}/runs | A fresh Fi sandbox is bound to the profile's SandboxTemplate; the agent's bash and tool calls run on that pod for the active sandbox session. Existing live sandboxes keep their current profile until released, restarted, or reaped. |
| Heavy ad-hoc SQL — TPC-H joins, migrations, full-table re-materializations | compute_profile field on POST /api/v1/queries | Burst pod runs DuckDB directly against the lake catalog (ATTACH 'ducklake:postgres:<DSN>'); each call gets a fresh process so DuckDB's connection-scoped memory leak (duckdb/duckdb#15176, #18031) cannot accumulate. |
The /api/v1/queries endpoint accepts compute_profile from every caller
(SDK, frontend dropdown, CLI). Default profile runs on the API pod's
embedded DuckLake connection — which attaches the Postgres catalog and reads
parquet directly from object storage over httpfs. A
non-default profile dispatches the query to a disposable burst sandbox
chosen by compute_profiles.<name> (issue #458). See
Running heavy SQL workloads for the
cold-start / warm-pool tradeoff and when to bump the API/job-runner sizing
vs. add a burst profile.
Note —
sql,python, andmaintenanceautomation steps honorconfig.compute_profilevia the burst-sandbox path. The three sync step types (pg_sync,adbc_sync,mssql_sync) honor it differently: a non-default profile runs the whole sync step as a dedicated Kubernetes Job sized by the profile — see Sync-executor Jobs below. (WithjobRunner.syncJobs.enabled: false, pg_sync falls back to its older burst-sandbox path for portable sources, andadbc_sync/mssql_syncfall back to running in-pod.)
Sync-executor Jobs
Sync steps are where the run-to-completion workloads live (full-history
backfills, big catch-ups, forced full refreshes), and they get a different
surface than the service-like burst sandboxes: a batch/v1 Job per step
(#710). The profile contributes the same scheduling triple — resources,
node_selector, tolerations — so a profile name means the same hardware on
either surface. The pool/lifecycle knobs (warm_pool_size,
worker_idle_ttl_seconds, max_concurrent) don't apply to Jobs: there is no
warm pool to keep (cold-start is noise against a multi-hour run) and no idle
state to reap — the pod ends when the sync ends, and the burst node pool
scales back to zero.
What this buys over running the same sync on a burst sandbox:
- no sandbox idle reaper and no single multi-hour RPC to hold open;
- pod-death restarts come free (
backoffLimit) and resume from the sync's persisted chunk checkpoint / watermark; - it works with Fi disabled (core Kubernetes only), and with SSH-bastion / pasted-cert sources (the tunnel opens inside the executor pod);
- DuckDB inside the executor is memory-bounded by the profile's limits.
Job-level knobs (jobRunner.syncJobs.{enabled,activeDeadlineSeconds,backoffLimit})
live in values.yaml; see docs/automations.md → "Sync-executor Jobs" for
the full behavior.
Maintenance jobs
The managed maintenance schedule and ad-hoc maintenance runs are
schedule-level: one profile applies to every step in the run (set it in
Settings → Maintenance → Compute profile, or pass compute_profile to
POST /api/v1/lakehouse/maintenance/runs). The API validates the name against
the registry at write time and stamps it onto each step's config. At run time
the job-runner dispatches each maintenance statement to a disposable sandbox of
that tier, exactly like a sql step. Leave it on default to keep maintenance
running in-pod on the job-runner's attached DuckLake connection.
Data apps
A data app opts into burst by adding a top-level "compute_profile" to its
app.json manifest:
{
"version": 2,
"name": "Heavy Dashboard",
"compute_profile": "large",
"resources": { "...": "..." }
}
Every SQL resource the app serves then runs on a large sandbox instead of the
API pod's embedded DuckLake connection. The name is validated against the
registry on upload. Two caveats specific to data apps:
- Cold-start latency. Dashboard queries are interactive (they fire on load
and on filter changes). With
warm_pool_size: 0the first query after an idle period pays the pod cold-start. Give a data-app profile awarm_pool_size≥ 1 if responsiveness matters. - Type fidelity. The burst sandbox returns JSON rows, which are converted
back to Arrow with inferred types — a
DATEcolumn comes back as a string, not a date. Prefer the default profile for apps whose charts depend on precise column types, or cast columns explicitly in the resource SQL.
When the profile can't be honored (Fi disabled or unknown name) the query silently falls back to the API pod's embedded DuckLake connection so a dashboard never hard-fails over its profile choice.
Declaring profiles (operator)
compute_profiles: is a top-level map in config.yaml. Each entry is a name
plus the resources, scheduling hints, and warm-pool size that should back it.
compute_profiles:
large:
resources:
requests: { cpu: "4", memory: "16Gi" }
limits: { cpu: "8", memory: "32Gi" }
node_selector:
# Whatever label identifies your burst node pool — see below.
cloud.google.com/gke-nodepool: definite-burst
tolerations:
- { key: definite-app/burst, operator: Equal, value: "true", effect: NoSchedule }
warm_pool_size: 0 # cold-start; pool scales to zero when idle
max_concurrent: 2 # SQL worker slots for this profile
worker_idle_ttl_seconds: 300
xlarge:
resources:
requests: { cpu: "16", memory: "64Gi" }
limits: { cpu: "32", memory: "128Gi" }
node_selector:
cloud.google.com/gke-nodepool: definite-burst-xl
tolerations:
- { key: definite-app/burst-xl, operator: Equal, value: "true", effect: NoSchedule }
warm_pool_size: 0
max_concurrent: 1
worker_idle_ttl_seconds: 300
Apply with definite upgrade --config config.yaml. The chart renders one
SandboxTemplate per profile and one optional SandboxWarmPool. Profile names
must be DNS-label safe (lowercase alphanumeric + hyphens, max 40 chars).
Field reference
| Field | Type | Default | Notes |
|---|---|---|---|
resources | K8s resources map | fi.sandbox.resources | Standard {requests, limits} map. GPUs etc. flow through unchanged — the chart toYamls it. |
node_selector | {label: value} | none | Pins the pod to a specific node pool. See Per-cloud node pool setup. |
tolerations | list | none | Required when the pool carries a taint. Standard K8s shape. |
warm_pool_size | int | fi.sandbox.warmPoolSize | 0 = cold-start (no pre-warmed pods, pool scales to zero). Bigger numbers reserve idle pods at this profile's cost — usually keep at 0 for non-default profiles. |
worker_idle_ttl_seconds | int | fi.workerTier.idleTtlSeconds | How long an on-demand attached SQL worker stays warm after its last query before the API reaper deletes its SandboxClaim. Separate from warm pools. |
workspace_size | string | fi.sandbox.workspaceSize | Per-pod ephemeral PVC for /workspace. Bump for jobs that write large intermediate files. |
storage_class_name | string | fi.sandbox.storageClassName | StorageClass for the workspace PVC. |
max_concurrent | int | none | Maximum attached SQL workers for the profile. When the worker tier is enabled, concurrent profiled queries fan out across these slots; excess queries queue briefly, then return 503. |
A profile with no node_selector schedules on the default node pool —
useful for a slightly bigger sandbox that doesn't need its own pool.
Editing a profile's size at runtime (Settings → Compute)
config.yaml is the source of truth for which profiles exist and their
baseline size. For a quick CPU/memory change without a redeploy, an admin
can override a profile's resources from Settings → Compute (or the API).
node_selector, tolerations, warm-pool size, and the profile set remain
Helm-only — runtime overrides cover sizing only.
# Set large to 3→6 CPU / 12→24Gi. At least one of requests/limits cpu/memory
# is required. Admin token only.
curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"requests":{"cpu":"3","memory":"12Gi"},"limits":{"cpu":"6","memory":"24Gi"}}' \
https://onprem.example.com/api/v1/compute-profiles/large/resources
# → {"profile":{...,"resources_overridden":true}, "reconciled":true, "detail":null}
# Revert large to its config.yaml size.
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
https://onprem.example.com/api/v1/compute-profiles/large/resources
How it works:
- The override is persisted in Postgres (
definite.compute_profile_overrides) — it is the source of truth, not the live cluster object. - The API immediately reconciles it onto the profile's live
SandboxTemplate(spec.podTemplate.spec.containers[0].resources).reconciled:false(with adetailnote) means Fi is disabled or the template isn't rendered yet — the size is still saved and applies later. - Only new sandboxes pick up the change; warm/running pods keep their
current size. With
warm_pool_size: 0(the burst default) there are no warm pods, so the next claim gets the new size. - On API startup the persisted overrides are re-applied onto the
templates, so they survive a
helm upgrade(see below).
Overrides and helm upgrade
Because the API takes server-side-apply ownership of the overridden
resources field, a plain helm upgrade (Helm v4 / SSA) fails with a
field-ownership conflict on that field. definite upgrade already passes
--force-conflicts, which lets the chart reclaim the field (resetting it to
the config.yaml size); the API's boot-time reconcile then re-applies the
persisted override seconds later. If you invoke helm upgrade directly, add
--force-conflicts when any compute-profile size override is active.
Per-cloud node pool setup
The feature is cloud-agnostic — the chart uses standard nodeSelector /
tolerations. Only the label keys for node_selector change per cloud.
GKE (managed node pool)
gcloud container node-pools create definite-burst \
--cluster=definite --region=us-central1 \
--machine-type=n2-standard-16 \
--node-taints=definite-app/burst=true:NoSchedule \
--enable-autoscaling --num-nodes=0 --min-nodes=0 --max-nodes=4
In config.yaml:
node_selector:
cloud.google.com/gke-nodepool: definite-burst
GKE Autopilot autoscales managed node pools down to zero — the burst pool costs nothing while idle and cold-starts in ~2 minutes the first time it's used in a day.
EKS (managed node group)
eksctl create nodegroup --cluster=definite \
--name=definite-burst --instance-types=m5.4xlarge \
--node-taints=definite-app/burst=true:NoSchedule \
--asg-access --node-min=0 --node-max=4
In config.yaml:
node_selector:
eks.amazonaws.com/nodegroup: definite-burst
EKS (Karpenter — recommended for burst)
Karpenter spins nodes up in ~30s and tears them down when the last pod schedules off, which is the best fit for "burst, scale fully to zero".
NodePool CR:
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: definite-burst
spec:
template:
spec:
taints:
- key: definite-app/burst
value: "true"
effect: NoSchedule
requirements:
- { key: karpenter.sh/capacity-type, operator: In, values: [on-demand] }
- { key: node.kubernetes.io/instance-type, operator: In, values: [m5.4xlarge, m5.8xlarge] }
disruption:
consolidationPolicy: WhenEmpty
consolidateAfter: 30s
In config.yaml:
node_selector:
karpenter.sh/nodepool: definite-burst
AKS (managed node pool)
az aks nodepool add --cluster-name definite \
--name burst --node-vm-size Standard_D16s_v5 \
--node-taints definite-app/burst=true:NoSchedule \
--enable-cluster-autoscaler --min-count 0 --max-count 4
In config.yaml:
node_selector:
agentpool: burst
The chart will happily render a profile even if no matching node pool
exists yet — the sandbox pods just sit Pending until you create one.
kubectl get pods -n definite -l app.kubernetes.io/component=fi-sandbox
makes it obvious.
Using a profile at runtime
CLI
# SQL — non-default profiles run the query on a burst sandbox pod.
definite run query --profile large 'SELECT count(*) FROM giant_table'
Transformation projects
On the Transformations page, deployments with more than one profile show a
project-level Compute dropdown and per-model overrides. Saving either
setting writes a new transformation manifest version and recompiles the
linked automation so each generated SQL step contains the actual
config.compute_profile it will run with.
The CLI can set the same project default at apply time:
definite transform apply transformations --name finance-marts --compute-profile large
Manifest/API callers can set the same fields directly:
{
"name": "finance marts",
"compute_profile": "large",
"models": [
{
"name": "stg_orders",
"sql": "select * from raw.orders"
},
{
"name": "orders",
"sql": "select * from staging.stg_orders",
"depends_on": ["stg_orders"],
"compute_profile": "xlarge"
}
]
}
HTTP API
# Listing — workspace-wide; any authenticated user can see the registry.
# Each profile carries its effective `resources` (the admin override when set,
# else the chart-rendered size) and `resources_overridden`. `editable` is true
# when this deployment can persist per-profile size overrides.
curl -H "Authorization: Bearer $TOKEN" \
https://onprem.example.com/api/v1/compute-profiles
# → {"profiles":[
# {"name":"default","is_default":true,"max_concurrent":null,
# "resources":{"requests":{"cpu":"250m","memory":"1Gi"},
# "limits":{"cpu":"4","memory":"8Gi"}},
# "resources_overridden":false},
# {"name":"large","is_default":false,"max_concurrent":2,
# "resources":{"requests":{"cpu":"4","memory":"16Gi"},
# "limits":{"cpu":"8","memory":"32Gi"}},
# "resources_overridden":false}],
# "default":"default","editable":true}
# Set a Fi thread's preferred profile. The next fresh sandbox for that thread
# uses the matching SandboxTemplate.
curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-X PATCH \
-d '{"compute_profile":"large"}' \
https://onprem.example.com/api/v1/fi/threads/$TID
# One-off Fi run override on a large sandbox.
curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"prompt":"crunch a big dataset", "compute_profile":"large"}' \
https://onprem.example.com/api/v1/fi/threads/$TID/runs
# Unknown profiles 400 with a helpful message:
# {"detail":"unknown compute profile 'xxlarge'. Known: default, large"}
Pipeline Python step
A python step gains an optional config.compute_profile. When set to a
non-default profile, the job-runner skips the in-pod subprocess path and
instead leases a Sandbox of that profile, POSTs the script + requirements
to its /python-jobs RPC, and tears the claim down when the step
finishes.
Set config.attach_lake: true when the script needs
definite_lakehouse.query(), write_table(), or a direct DuckDB/DuckLake
attach. That opt-in forwards LAKEHOUSE_* and the active object-store
credentials to the sandbox. Python steps without it can still burst onto a
compute profile, but they do not receive lake credentials unless the field
is omitted and the resolved script imports or mentions definite_lakehouse
(backward compatibility for older generated sync definitions).
{
"id": "tpch-queries",
"type": "python",
"config": {
"script_name": "tpch_q1_q22",
"compute_profile": "large",
"attach_lake": true
}
}
Frontend
The Query page shows a small Compute dropdown next to the Run button
when more than one profile is configured. Single-tier deployments
(no compute_profiles: in config.yaml) see no UI change.
Running heavy SQL workloads
POST /api/v1/queries with a non-default compute_profile dispatches the
query to a disposable burst sandbox (issue #458). Each call:
- Leases a fresh
SandboxClaimagainst the profile'sSandboxTemplate(definite-fi-<profile>, labeldefinite.app/compute-profile=<name>). - Waits for the pod's RPC to come up, then POSTs the SQL to
/sql-queries. - Inside the pod, spawns a fresh
pythonsubprocess that opens its own embedded DuckDB connection, attaches the lake (ATTACH 'ducklake:postgres:<DSN>'), runs the query, prints a JSON envelope, and exits. - Tears the claim down on the way out — success, SQL error, or pod
failure all hit the same
delete_claiminfinally.
The subprocess-per-call isolation is the key: DuckDB's connection-scoped
memory leak (duckdb/duckdb#15176, #18031) accumulates
~564 MiB per INSERT … SELECT batch on a long-lived connection. A fresh
process every call gives it nowhere to accumulate, which is the leg the
opinionroute transactiondata migration was missing.
How burst SQL runs the query
Because the lakehouse runs the query plan on the caller, the burst pod
ATTACH 'ducklake:postgres:<DSN>' reads catalog metadata straight from the
dedicated ducklake_catalog Postgres database and runs the query plan
locally in its own embedded DuckDB process, reading parquet over httpfs.
That client-side-compute model is what gives burst SQL its leak isolation: a
fresh process per call has nowhere for the connection-scoped leak to
accumulate.
Credential note. Because the query plan runs on the caller, the burst compute pods need both object-store credentials (httpfs parquet reads) and the catalog connection (the
catalog_dsn). The API forwards these to the sandbox automatically. On EKS, S3 IRSA removes static keys from the API/job-runner pods; if you also run burst compute against S3, make sure the shared Fi sandbox ServiceAccount can read the lakehouse bucket too, or keep using the static S3 fallback for those sandbox workloads.
| Cold-start surface | What to do |
|---|---|
| First call to an unused profile (no warm pod) | 5–30 s of pod scheduling + image pull. Fine for migrations; rough for an interactive Query page. |
| You want interactive burst SQL with low latency | Set warm_pool_size: N on the profile so claims bind in milliseconds. Each call still spawns a fresh subprocess inside the pod, so the leak isolation is preserved even though the pod is reused. |
| One-off heavy job, no concurrency | warm_pool_size: 0. Pool scales to zero between jobs. |
Worker tier
The dispatch above spawns a fresh python subprocess per call inside the
burst pod — a new DuckDB connection that re-ATTACHes the lake and re-warms
its parquet/metadata cache every query. That's exactly what defeats the
connection-scoped leak, but it also throws away the warm cache and pays the
ATTACH cost on every query.
The worker tier trades a little of that isolation back for warmth. With
fi:
workerTier:
enabled: true
idleTtlSeconds: 300
reaperIntervalSeconds: 60
a non-default compute_profile query routes to a bounded pool of
long-lived attached workers in the sandbox. Each worker ATTACHes the lake
once and serves queries against the same warm connection, so repeat queries
skip the ATTACH and reuse cached parquet/metadata. max_concurrent controls
how many worker slots a profile may have; a profile with max_concurrent: 4
can run four profiled queries at once before later queries queue.
The worker tier is enabled by the default Helm values. Set
fi.workerTier.enabled: false to keep the one-shot subprocess-per-call path.
The credential model is identical — the lake env is still forwarded per
request exactly as today.
On-demand workers are not permanent. After the last query, a worker stays warm
for worker_idle_ttl_seconds (or fi.workerTier.idleTtlSeconds, default
300s), then the API reaper deletes its SandboxClaim. With warm_pool_size: 0,
this gives cold-start-on-first-use, warm-through-burst, then scale-back-to-zero
when idle.
To stay safe against DuckDB's connection-scoped leak, the worker recycles
its connection on a cadence (between queries, never mid-query): it closes and
re-ATTACHes once it has served enough queries, scanned enough bytes, or grown
its RSS past a fraction of the pod's memory limit. The thresholds are chart
defaults (fi.workerTier.recycleQueries / recycleBytes /
recycleRssFraction) rendered onto the sandbox pod and rarely need tuning.
Warm schedules (business-hours warm windows)
Scale-to-zero means the first query after an idle gap pays the full cold
start — on autoscaled clusters that's node provisioning plus a fresh workspace
volume, often 1–2 minutes for big profiles. When a team hits the same profile
every weekday morning (a heavy data app on large, say), declare a warm
schedule instead of eating that cold start daily:
# Keep one large worker pre-started Mon–Fri 6:00–18:00 New York time.
curl -sS -X PUT "$BASE/api/v1/compute-profiles/large/warm-schedule" \
-H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
-d '{
"windows": [{"days": ["mon","tue","wed","thu","fri"],
"start": "06:00", "end": "18:00"}],
"timezone": "America/New_York",
"min_warm": 1
}'
# → {"profile":"large","warm_schedule":{...},"active":true,
# "detail":"Saved. The warm keeper applies this on its next tick (~1 minute)."}
# Read it back (any authenticated user; `active` = a window is open now).
curl -sS "$BASE/api/v1/compute-profiles/large/warm-schedule" \
-H "Authorization: Bearer $TOKEN"
# Remove it — workers drain via the normal idle TTL.
curl -sS -X DELETE "$BASE/api/v1/compute-profiles/large/warm-schedule" \
-H "Authorization: Bearer $ADMIN_TOKEN"
While any window is active, the API's warm-schedule keeper (same cadence as
the worker reaper) pre-starts min_warm worker slots for the profile and
the idle reaper will not reap them; the first query of the morning lands on an
already-attached warm worker. Outside every window nothing is protected — the
normal worker_idle_ttl_seconds drains the tier back to zero after the last
real query, so a query still running at the window's end is never killed
mid-flight.
Semantics worth knowing:
- Windows are same-day (
start < end, minute precision,HH:MM24h) and evaluated in the schedule's IANAtimezone(DST handled). Express an overnight span as two windows. Up to 16 windows per profile. min_warmis clamped to the profile'smax_concurrent. PUT rejects a floor above capacity; if capacity later shrinks, the keeper clamps at enforcement time.- Admin-only to write; anyone can read. Writes are audited
(
compute.profile.warm_schedule.*). The schedule also appears onGET /api/v1/compute-profiles(warm_schedule,warm_schedule_active) and the live floor onGET …/{name}/worker-status(warm_floor). - It's declarative, not fired. There is no missed-run state: a restarted API re-evaluates the window on the next keeper tick (~60s) and converges — same model as the reaper itself.
- Cost: a warm window is billed like any running worker pod for the whole
window. One
large(8 CPU / 32Gi) worker kept warm 6:00–18:00 Mon–Fri is ~60 pod-hours/week — sizemin_warmand the window accordingly. - The
defaultprofile takes no schedule (it runs inside the API pod; there is nothing to pre-start).
When to bump the always-on pods vs add a burst profile
Lakehouse queries run as embedded DuckDB inside the API and job-runner pods, so the "baseline" knob is those pods' memory, not a separate lakehouse pod.
| Symptom | Right knob |
|---|---|
Steady-state interactive query load is hitting the API pod's memory limit | Raise resources.api.memory (and cpu if needed). Sizes the interactive baseline. |
Steady-state automation / maintenance SQL is hitting the job-runner's memory limit | Raise resources.job_runner.memory (and cpu if needed). |
| One job (migration, big join) blows the pod when it runs but the baseline is fine | Add a compute_profiles.<name> block sized for that job, then call it from /api/v1/queries (or from the script). The always-on pods stay cheap, and the burst pod can pin a bigger, scale-to-zero node pool (see per-cloud setup). |
Long-running INSERT … SELECT loops OOM a pod after a few batches | The DuckDB connection-scoped leak. Move each batch to a burst profile — fresh subprocess per call resets the leak. |
How SQL compute lands
What determines where SQL compute lands is the ATTACH the client uses. The
lakehouse is postgres_catalog:
ATTACH 'ducklake:postgres:<DSN>' AS LAKE (DATA_PATH '...')— DuckLake is attached on the client with its Postgres catalog backend: catalog metadata lives in a dedicatedducklake_catalogPostgres database (reusing the deploy's existing Postgres server), and parquet reads + the query plan run locally on the caller in embedded DuckDB via httpfs. Compute scales with whatever pod runs the query. The CLI deriveslakehouse.catalog_dsnfrompostgres.urlwhen omitted, changing only dbname toducklake_catalog.
This is the pattern ducklake#1151 added on 2026-05-12 and
DuckDB 1.5.3 (released 2026-05-20) wires up end-to-end. Because
the query plan always runs on the caller, burst SQL and compute profiles do
not require a separate lakehouse service — the API/job-runner pod runs
default, and a non-default profile runs the same ATTACH in a sandbox pod. The
worker tier reuses attached workers when enabled; one-shot burst SQL leases a
fresh pod per query when the worker tier is disabled. The CLI derives the
catalog DSN and the chart wires the API + job-runner setup SQL to this ATTACH
form (plus a CREATE SECRET store_creds (...) for httpfs reads). See the
config.md lakehouse section.
If the DuckLake Postgres catalog resolves to a private RFC1918 or IPv6 ULA IP,
add an exact sandbox egress allowlist in fi.sandbox.egress.private_cidrs so
the profile's sandbox or worker pod can reach it. The chart does not derive
this from the secret DSN; NetworkPolicy ipBlock rules need
operator-supplied CIDRs.
The DuckLake Postgres catalog is multi-writer and battle-tested, so there are no concurrency/maintenance beta gaps to work around.
Run heavy SQL with a profile-bound Python step
Another way to run a heavy SQL workload is to wrap it in a Python step (or Fi
run) with compute_profile: large. The sandbox image has DuckDB available, and
Python steps can opt into lake access with attach_lake: true. The SDK
(definite_lakehouse) exposes a write_table that handles >50k-row writes
through staging. The script runs on the burst node, attaches the DuckLake
Postgres catalog, reads inputs straight from object storage over httpfs, and
writes results back to the lake.
Example: TPC-H benchmarks
Generate TPC-H data once, then run the 22 queries on a burst pod:
# automation_scripts/tpch_gen.py — generates SF=10 TPC-H data and writes it
# to the lake. Run this once as a pipeline step with compute_profile: large.
import duckdb
from definite_lakehouse import write_table
con = duckdb.connect(":memory:", config={"memory_limit": "28GB"})
con.execute("INSTALL tpch; LOAD tpch; CALL dbgen(sf=10)")
for table in ("nation", "region", "part", "supplier", "partsupp",
"customer", "orders", "lineitem"):
df = con.execute(f"SELECT * FROM {table}").fetch_arrow_table()
write_table(name=f"tpch.{table}", arrow_table=df, mode="replace")
print("TPC-H SF=10 written to lake")
# automation_scripts/tpch_run.py — runs the 22 TPC-H queries against the
# DuckLake-backed tables, on the same burst profile. We attach the DuckLake
# Postgres catalog directly (the same path the API uses) and read parquet
# from the object store over httpfs, then time each query.
#
# Easiest path: let the SDK build the attach for you, which reads
# LAKEHOUSE_CATALOG_DSN / LAKEHOUSE_DATA_PATH and the active store creds
# (all forwarded when attach_lake: true) and wires up the store secret.
import json, time
from definite_lakehouse import attach # returns the configured DuckDB conn with LAKE attached
con = attach() # Postgres DuckLake catalog + httpfs parquet, SQL runs in this process
con.execute("INSTALL tpch; LOAD tpch") # provides tpch_queries()
timings = []
for q in con.execute("FROM tpch_queries()").fetchall():
qid, sql = q[0], q[1].replace("FROM ", "FROM LAKE.tpch.")
t0 = time.perf_counter()
con.execute(sql).fetchall()
timings.append({"query": f"q{qid}", "ms": round((time.perf_counter() - t0) * 1000)})
print(json.dumps({"timings": timings}, indent=2))
Register both as automation scripts (definite run script create --name tpch_gen --content tpch_gen.py, same for tpch_run), then drop
them into a pipeline:
# pipeline.yaml — create with: definite run automation create --definition pipeline.yaml
steps:
- id: generate
type: python
config:
script_name: tpch_gen
compute_profile: large
attach_lake: true
- id: bench
type: python
config:
script_name: tpch_run
compute_profile: large
attach_lake: true
A burst-pool node spins up on the first step, both steps reuse the same pool, and the node scales back down ~30s after the second step's sandbox lease is torn down.
For genuinely ad-hoc beefy SQL (one-off exploration, no pipeline
overhead), open a Fi thread with compute_profile: large and ask the
agent to run the query. Fi's bash tool inside the sandbox can invoke
duckdb directly — same compute path, no automation plumbing.
Limits (v1)
- Lake access from Python steps is explicit. A
compute-profile Python step only receives
LAKEHOUSE_*and object-store credentials whenconfig.attach_lake: trueis set. Setattach_lake: falseto force no lake env. adbc_sync/mssql_syncsteps do not route to burst sandboxes yet. They acceptconfig.compute_profilefor forward compatibility but run on their existing executor paths.pg_synchonors a non-default profile when the source connection is portable (no SSH bastion, no pasted SSL certs); non-portable sources stay in-pod and log that the profile was ignored.- Burst SQL needs the Fi sandbox subsystem. If the deployment has Fi
disabled (
fi.enabled: false), a non-defaultcompute_profileon/api/v1/queriesreturns 503 rather than silently falling back to the API pod's embedded connection — silent fallback would defeat the leak-isolation guarantee the feature exists to provide. - No authorization gating. Any authenticated user can pick any
declared profile. Cost is controlled by what the operator declares
in
config.yaml, and the worker tier enforcesmax_concurrentas the attached SQL worker slot count for each profile.
Default deployments
A deployment with no compute_profiles: block uses the default sandbox
settings from fi.sandbox.*. Add named profiles only when specific workloads
need different resources, scheduling hints, or warm-pool behavior.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
SandboxClaim stuck Pending, no pod scheduled | Node pool for the profile doesn't exist or its taint doesn't match the profile's tolerations | kubectl get nodes -L cloud.google.com/gke-nodepool (etc.) to confirm the pool exists; compare the pool's taint to compute_profiles.<name>.tolerations |
Worker query times out attaching ducklake:postgres while the default query path works | The profile's sandbox/worker pod cannot reach a private Postgres catalog endpoint because sandbox NetworkPolicy excludes RFC1918 by default | Add fi.sandbox.egress.private_cidrs: ["<catalog-ip>/32"] and keep private_ports at [5432] unless your catalog uses a custom port, then re-run definite upgrade |
GET /api/v1/compute-profiles returns only default after adding a profile | Helm release wasn't re-rendered, or FI_COMPUTE_PROFILES_JSON wasn't refreshed | definite upgrade --config config.yaml, then kubectl rollout restart deploy/definite-api -n definite |
Warm-pool sandboxes still on the old image after helm upgrade | Known agent-sandbox footgun — the controller keeps the warm pool's bound pods until the SandboxClaim is deleted | kubectl delete sandboxes.agents.x-k8s.io -n definite -l app.kubernetes.io/component=fi-sandbox — the pool refills with the new image. |
400: unknown compute profile 'foo'. Known: default, large | The frontend or CLI sent a profile name not in the chart-rendered registry | Check config.yaml's compute_profiles: keys and re-run definite upgrade if you added one but didn't deploy it. |