Compute Profiles

Compute profiles are named burst tiers the operator declares in config.yaml. A user, a Fi thread, or a pipeline Python step picks a profile by name at run time, and the workload runs on a Kubernetes pod sized for that tier — usually on a dedicated node pool that scales to zero when idle, so big compute costs nothing while nobody's using it.

The deployment ships with one profile, default, sized at the small always-on fi.sandbox.resources defaults. The operator adds bigger profiles (large, xlarge, gpu, …) by declaring them in config.yaml.

When to use a profile

Use caseProfile pathWhat runs where
Heavy Python step in an automation (pandas joins, ML training, dbt-style transforms)python step config.compute_profileJob-runner leases a Sandbox of that profile and POSTs the script + requirements to it; the script runs on the burst pod, not in the job-runner.
SQL transformation model or automation SQL stepTransformation project/model Compute setting, or sql step config.compute_profileThe generated automation step carries the profile. Default runs on the job-runner's attached DuckLake connection; non-default dispatches through the burst SQL sandbox path.
Scheduled / ad-hoc DuckLake maintenance (expire_snapshots, cleanup_old_files, compact, …)Settings → Maintenance → Compute profile (schedule-level), or compute_profile on POST /api/v1/lakehouse/maintenance/runsThe profile is stamped onto every maintenance step. Default runs in the job-runner pod's attached DuckLake connection; non-default runs each ducklake_* / CHECKPOINT statement on a disposable sandbox.
Heavy data-app dashboard queryApp-level "compute_profile" in the app's app.json manifestEach SQL resource query dispatches to a burst sandbox of that tier (requires Fi). Falls back to the API pod's attached DuckLake connection when the profile can't be honored. Interactive — see the cold-start caveat below.
Fi session that crunches a big dataset in the agent's bash toolThread menu Compute profile, or compute_profile field on POST /api/v1/fi/threads/{id}/runsA fresh Fi sandbox is bound to the profile's SandboxTemplate; the agent's bash and tool calls run on that pod for the active sandbox session. Existing live sandboxes keep their current profile until released, restarted, or reaped.
Heavy ad-hoc SQL — TPC-H joins, migrations, full-table re-materializationscompute_profile field on POST /api/v1/queriesBurst pod runs DuckDB directly against the lake catalog (ATTACH 'ducklake:postgres:<DSN>'); each call gets a fresh process so DuckDB's connection-scoped memory leak (duckdb/duckdb#15176, #18031) cannot accumulate.

The /api/v1/queries endpoint accepts compute_profile from every caller (SDK, frontend dropdown, CLI). Default profile runs on the API pod's embedded DuckLake connection — which attaches the Postgres catalog and reads parquet directly from object storage over httpfs. A non-default profile dispatches the query to a disposable burst sandbox chosen by compute_profiles.<name> (issue #458). See Running heavy SQL workloads for the cold-start / warm-pool tradeoff and when to bump the API/job-runner sizing vs. add a burst profile.

Notesql, python, and maintenance automation steps honor config.compute_profile via the burst-sandbox path. The three sync step types (pg_sync, adbc_sync, mssql_sync) honor it differently: a non-default profile runs the whole sync step as a dedicated Kubernetes Job sized by the profile — see Sync-executor Jobs below. (With jobRunner.syncJobs.enabled: false, pg_sync falls back to its older burst-sandbox path for portable sources, and adbc_sync / mssql_sync fall back to running in-pod.)

Sync-executor Jobs

Sync steps are where the run-to-completion workloads live (full-history backfills, big catch-ups, forced full refreshes), and they get a different surface than the service-like burst sandboxes: a batch/v1 Job per step (#710). The profile contributes the same scheduling triple — resources, node_selector, tolerations — so a profile name means the same hardware on either surface. The pool/lifecycle knobs (warm_pool_size, worker_idle_ttl_seconds, max_concurrent) don't apply to Jobs: there is no warm pool to keep (cold-start is noise against a multi-hour run) and no idle state to reap — the pod ends when the sync ends, and the burst node pool scales back to zero.

What this buys over running the same sync on a burst sandbox:

  • no sandbox idle reaper and no single multi-hour RPC to hold open;
  • pod-death restarts come free (backoffLimit) and resume from the sync's persisted chunk checkpoint / watermark;
  • it works with Fi disabled (core Kubernetes only), and with SSH-bastion / pasted-cert sources (the tunnel opens inside the executor pod);
  • DuckDB inside the executor is memory-bounded by the profile's limits.

Job-level knobs (jobRunner.syncJobs.{enabled,activeDeadlineSeconds,backoffLimit}) live in values.yaml; see docs/automations.md → "Sync-executor Jobs" for the full behavior.

Maintenance jobs

The managed maintenance schedule and ad-hoc maintenance runs are schedule-level: one profile applies to every step in the run (set it in Settings → Maintenance → Compute profile, or pass compute_profile to POST /api/v1/lakehouse/maintenance/runs). The API validates the name against the registry at write time and stamps it onto each step's config. At run time the job-runner dispatches each maintenance statement to a disposable sandbox of that tier, exactly like a sql step. Leave it on default to keep maintenance running in-pod on the job-runner's attached DuckLake connection.

Data apps

A data app opts into burst by adding a top-level "compute_profile" to its app.json manifest:

{
  "version": 2,
  "name": "Heavy Dashboard",
  "compute_profile": "large",
  "resources": { "...": "..." }
}

Every SQL resource the app serves then runs on a large sandbox instead of the API pod's embedded DuckLake connection. The name is validated against the registry on upload. Two caveats specific to data apps:

  • Cold-start latency. Dashboard queries are interactive (they fire on load and on filter changes). With warm_pool_size: 0 the first query after an idle period pays the pod cold-start. Give a data-app profile a warm_pool_size ≥ 1 if responsiveness matters.
  • Type fidelity. The burst sandbox returns JSON rows, which are converted back to Arrow with inferred types — a DATE column comes back as a string, not a date. Prefer the default profile for apps whose charts depend on precise column types, or cast columns explicitly in the resource SQL.

When the profile can't be honored (Fi disabled or unknown name) the query silently falls back to the API pod's embedded DuckLake connection so a dashboard never hard-fails over its profile choice.

Declaring profiles (operator)

compute_profiles: is a top-level map in config.yaml. Each entry is a name plus the resources, scheduling hints, and warm-pool size that should back it.

compute_profiles:
  large:
    resources:
      requests: { cpu: "4",  memory: "16Gi" }
      limits:   { cpu: "8",  memory: "32Gi" }
    node_selector:
      # Whatever label identifies your burst node pool — see below.
      cloud.google.com/gke-nodepool: definite-burst
    tolerations:
      - { key: definite-app/burst, operator: Equal, value: "true", effect: NoSchedule }
    warm_pool_size: 0          # cold-start; pool scales to zero when idle
    max_concurrent: 2          # SQL worker slots for this profile
    worker_idle_ttl_seconds: 300
  xlarge:
    resources:
      requests: { cpu: "16", memory: "64Gi" }
      limits:   { cpu: "32", memory: "128Gi" }
    node_selector:
      cloud.google.com/gke-nodepool: definite-burst-xl
    tolerations:
      - { key: definite-app/burst-xl, operator: Equal, value: "true", effect: NoSchedule }
    warm_pool_size: 0
    max_concurrent: 1
    worker_idle_ttl_seconds: 300

Apply with definite upgrade --config config.yaml. The chart renders one SandboxTemplate per profile and one optional SandboxWarmPool. Profile names must be DNS-label safe (lowercase alphanumeric + hyphens, max 40 chars).

Field reference

FieldTypeDefaultNotes
resourcesK8s resources mapfi.sandbox.resourcesStandard {requests, limits} map. GPUs etc. flow through unchanged — the chart toYamls it.
node_selector{label: value}nonePins the pod to a specific node pool. See Per-cloud node pool setup.
tolerationslistnoneRequired when the pool carries a taint. Standard K8s shape.
warm_pool_sizeintfi.sandbox.warmPoolSize0 = cold-start (no pre-warmed pods, pool scales to zero). Bigger numbers reserve idle pods at this profile's cost — usually keep at 0 for non-default profiles.
worker_idle_ttl_secondsintfi.workerTier.idleTtlSecondsHow long an on-demand attached SQL worker stays warm after its last query before the API reaper deletes its SandboxClaim. Separate from warm pools.
workspace_sizestringfi.sandbox.workspaceSizePer-pod ephemeral PVC for /workspace. Bump for jobs that write large intermediate files.
storage_class_namestringfi.sandbox.storageClassNameStorageClass for the workspace PVC.
max_concurrentintnoneMaximum attached SQL workers for the profile. When the worker tier is enabled, concurrent profiled queries fan out across these slots; excess queries queue briefly, then return 503.

A profile with no node_selector schedules on the default node pool — useful for a slightly bigger sandbox that doesn't need its own pool.

Editing a profile's size at runtime (Settings → Compute)

config.yaml is the source of truth for which profiles exist and their baseline size. For a quick CPU/memory change without a redeploy, an admin can override a profile's resources from Settings → Compute (or the API). node_selector, tolerations, warm-pool size, and the profile set remain Helm-only — runtime overrides cover sizing only.

# Set large to 3→6 CPU / 12→24Gi. At least one of requests/limits cpu/memory
# is required. Admin token only.
curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"requests":{"cpu":"3","memory":"12Gi"},"limits":{"cpu":"6","memory":"24Gi"}}' \
  https://onprem.example.com/api/v1/compute-profiles/large/resources
# → {"profile":{...,"resources_overridden":true}, "reconciled":true, "detail":null}

# Revert large to its config.yaml size.
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  https://onprem.example.com/api/v1/compute-profiles/large/resources

How it works:

  • The override is persisted in Postgres (definite.compute_profile_overrides) — it is the source of truth, not the live cluster object.
  • The API immediately reconciles it onto the profile's live SandboxTemplate (spec.podTemplate.spec.containers[0].resources). reconciled:false (with a detail note) means Fi is disabled or the template isn't rendered yet — the size is still saved and applies later.
  • Only new sandboxes pick up the change; warm/running pods keep their current size. With warm_pool_size: 0 (the burst default) there are no warm pods, so the next claim gets the new size.
  • On API startup the persisted overrides are re-applied onto the templates, so they survive a helm upgrade (see below).

Overrides and helm upgrade

Because the API takes server-side-apply ownership of the overridden resources field, a plain helm upgrade (Helm v4 / SSA) fails with a field-ownership conflict on that field. definite upgrade already passes --force-conflicts, which lets the chart reclaim the field (resetting it to the config.yaml size); the API's boot-time reconcile then re-applies the persisted override seconds later. If you invoke helm upgrade directly, add --force-conflicts when any compute-profile size override is active.

Per-cloud node pool setup

The feature is cloud-agnostic — the chart uses standard nodeSelector / tolerations. Only the label keys for node_selector change per cloud.

GKE (managed node pool)

gcloud container node-pools create definite-burst \
  --cluster=definite --region=us-central1 \
  --machine-type=n2-standard-16 \
  --node-taints=definite-app/burst=true:NoSchedule \
  --enable-autoscaling --num-nodes=0 --min-nodes=0 --max-nodes=4

In config.yaml:

node_selector:
  cloud.google.com/gke-nodepool: definite-burst

GKE Autopilot autoscales managed node pools down to zero — the burst pool costs nothing while idle and cold-starts in ~2 minutes the first time it's used in a day.

EKS (managed node group)

eksctl create nodegroup --cluster=definite \
  --name=definite-burst --instance-types=m5.4xlarge \
  --node-taints=definite-app/burst=true:NoSchedule \
  --asg-access --node-min=0 --node-max=4

In config.yaml:

node_selector:
  eks.amazonaws.com/nodegroup: definite-burst

Karpenter spins nodes up in ~30s and tears them down when the last pod schedules off, which is the best fit for "burst, scale fully to zero".

NodePool CR:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: definite-burst
spec:
  template:
    spec:
      taints:
        - key: definite-app/burst
          value: "true"
          effect: NoSchedule
      requirements:
        - { key: karpenter.sh/capacity-type, operator: In, values: [on-demand] }
        - { key: node.kubernetes.io/instance-type, operator: In, values: [m5.4xlarge, m5.8xlarge] }
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s

In config.yaml:

node_selector:
  karpenter.sh/nodepool: definite-burst

AKS (managed node pool)

az aks nodepool add --cluster-name definite \
  --name burst --node-vm-size Standard_D16s_v5 \
  --node-taints definite-app/burst=true:NoSchedule \
  --enable-cluster-autoscaler --min-count 0 --max-count 4

In config.yaml:

node_selector:
  agentpool: burst

The chart will happily render a profile even if no matching node pool exists yet — the sandbox pods just sit Pending until you create one. kubectl get pods -n definite -l app.kubernetes.io/component=fi-sandbox makes it obvious.

Using a profile at runtime

CLI

# SQL — non-default profiles run the query on a burst sandbox pod.
definite run query --profile large 'SELECT count(*) FROM giant_table'

Transformation projects

On the Transformations page, deployments with more than one profile show a project-level Compute dropdown and per-model overrides. Saving either setting writes a new transformation manifest version and recompiles the linked automation so each generated SQL step contains the actual config.compute_profile it will run with.

The CLI can set the same project default at apply time:

definite transform apply transformations --name finance-marts --compute-profile large

Manifest/API callers can set the same fields directly:

{
  "name": "finance marts",
  "compute_profile": "large",
  "models": [
    {
      "name": "stg_orders",
      "sql": "select * from raw.orders"
    },
    {
      "name": "orders",
      "sql": "select * from staging.stg_orders",
      "depends_on": ["stg_orders"],
      "compute_profile": "xlarge"
    }
  ]
}

HTTP API

# Listing — workspace-wide; any authenticated user can see the registry.
# Each profile carries its effective `resources` (the admin override when set,
# else the chart-rendered size) and `resources_overridden`. `editable` is true
# when this deployment can persist per-profile size overrides.
curl -H "Authorization: Bearer $TOKEN" \
  https://onprem.example.com/api/v1/compute-profiles
# → {"profiles":[
#      {"name":"default","is_default":true,"max_concurrent":null,
#       "resources":{"requests":{"cpu":"250m","memory":"1Gi"},
#                    "limits":{"cpu":"4","memory":"8Gi"}},
#       "resources_overridden":false},
#      {"name":"large","is_default":false,"max_concurrent":2,
#       "resources":{"requests":{"cpu":"4","memory":"16Gi"},
#                    "limits":{"cpu":"8","memory":"32Gi"}},
#       "resources_overridden":false}],
#    "default":"default","editable":true}

# Set a Fi thread's preferred profile. The next fresh sandbox for that thread
# uses the matching SandboxTemplate.
curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -X PATCH \
  -d '{"compute_profile":"large"}' \
  https://onprem.example.com/api/v1/fi/threads/$TID

# One-off Fi run override on a large sandbox.
curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"prompt":"crunch a big dataset", "compute_profile":"large"}' \
  https://onprem.example.com/api/v1/fi/threads/$TID/runs

# Unknown profiles 400 with a helpful message:
# {"detail":"unknown compute profile 'xxlarge'. Known: default, large"}

Pipeline Python step

A python step gains an optional config.compute_profile. When set to a non-default profile, the job-runner skips the in-pod subprocess path and instead leases a Sandbox of that profile, POSTs the script + requirements to its /python-jobs RPC, and tears the claim down when the step finishes.

Set config.attach_lake: true when the script needs definite_lakehouse.query(), write_table(), or a direct DuckDB/DuckLake attach. That opt-in forwards LAKEHOUSE_* and the active object-store credentials to the sandbox. Python steps without it can still burst onto a compute profile, but they do not receive lake credentials unless the field is omitted and the resolved script imports or mentions definite_lakehouse (backward compatibility for older generated sync definitions).

{
  "id": "tpch-queries",
  "type": "python",
  "config": {
    "script_name": "tpch_q1_q22",
    "compute_profile": "large",
    "attach_lake": true
  }
}

Frontend

The Query page shows a small Compute dropdown next to the Run button when more than one profile is configured. Single-tier deployments (no compute_profiles: in config.yaml) see no UI change.

Running heavy SQL workloads

POST /api/v1/queries with a non-default compute_profile dispatches the query to a disposable burst sandbox (issue #458). Each call:

  1. Leases a fresh SandboxClaim against the profile's SandboxTemplate (definite-fi-<profile>, label definite.app/compute-profile=<name>).
  2. Waits for the pod's RPC to come up, then POSTs the SQL to /sql-queries.
  3. Inside the pod, spawns a fresh python subprocess that opens its own embedded DuckDB connection, attaches the lake (ATTACH 'ducklake:postgres:<DSN>'), runs the query, prints a JSON envelope, and exits.
  4. Tears the claim down on the way out — success, SQL error, or pod failure all hit the same delete_claim in finally.

The subprocess-per-call isolation is the key: DuckDB's connection-scoped memory leak (duckdb/duckdb#15176, #18031) accumulates ~564 MiB per INSERT … SELECT batch on a long-lived connection. A fresh process every call gives it nowhere to accumulate, which is the leg the opinionroute transactiondata migration was missing.

How burst SQL runs the query

Because the lakehouse runs the query plan on the caller, the burst pod ATTACH 'ducklake:postgres:<DSN>' reads catalog metadata straight from the dedicated ducklake_catalog Postgres database and runs the query plan locally in its own embedded DuckDB process, reading parquet over httpfs. That client-side-compute model is what gives burst SQL its leak isolation: a fresh process per call has nowhere for the connection-scoped leak to accumulate.

Credential note. Because the query plan runs on the caller, the burst compute pods need both object-store credentials (httpfs parquet reads) and the catalog connection (the catalog_dsn). The API forwards these to the sandbox automatically. On EKS, S3 IRSA removes static keys from the API/job-runner pods; if you also run burst compute against S3, make sure the shared Fi sandbox ServiceAccount can read the lakehouse bucket too, or keep using the static S3 fallback for those sandbox workloads.

Cold-start surfaceWhat to do
First call to an unused profile (no warm pod)5–30 s of pod scheduling + image pull. Fine for migrations; rough for an interactive Query page.
You want interactive burst SQL with low latencySet warm_pool_size: N on the profile so claims bind in milliseconds. Each call still spawns a fresh subprocess inside the pod, so the leak isolation is preserved even though the pod is reused.
One-off heavy job, no concurrencywarm_pool_size: 0. Pool scales to zero between jobs.

Worker tier

The dispatch above spawns a fresh python subprocess per call inside the burst pod — a new DuckDB connection that re-ATTACHes the lake and re-warms its parquet/metadata cache every query. That's exactly what defeats the connection-scoped leak, but it also throws away the warm cache and pays the ATTACH cost on every query.

The worker tier trades a little of that isolation back for warmth. With

fi:
  workerTier:
    enabled: true
    idleTtlSeconds: 300
    reaperIntervalSeconds: 60

a non-default compute_profile query routes to a bounded pool of long-lived attached workers in the sandbox. Each worker ATTACHes the lake once and serves queries against the same warm connection, so repeat queries skip the ATTACH and reuse cached parquet/metadata. max_concurrent controls how many worker slots a profile may have; a profile with max_concurrent: 4 can run four profiled queries at once before later queries queue.

The worker tier is enabled by the default Helm values. Set fi.workerTier.enabled: false to keep the one-shot subprocess-per-call path. The credential model is identical — the lake env is still forwarded per request exactly as today.

On-demand workers are not permanent. After the last query, a worker stays warm for worker_idle_ttl_seconds (or fi.workerTier.idleTtlSeconds, default 300s), then the API reaper deletes its SandboxClaim. With warm_pool_size: 0, this gives cold-start-on-first-use, warm-through-burst, then scale-back-to-zero when idle.

To stay safe against DuckDB's connection-scoped leak, the worker recycles its connection on a cadence (between queries, never mid-query): it closes and re-ATTACHes once it has served enough queries, scanned enough bytes, or grown its RSS past a fraction of the pod's memory limit. The thresholds are chart defaults (fi.workerTier.recycleQueries / recycleBytes / recycleRssFraction) rendered onto the sandbox pod and rarely need tuning.

Warm schedules (business-hours warm windows)

Scale-to-zero means the first query after an idle gap pays the full cold start — on autoscaled clusters that's node provisioning plus a fresh workspace volume, often 1–2 minutes for big profiles. When a team hits the same profile every weekday morning (a heavy data app on large, say), declare a warm schedule instead of eating that cold start daily:

# Keep one large worker pre-started Mon–Fri 6:00–18:00 New York time.
curl -sS -X PUT "$BASE/api/v1/compute-profiles/large/warm-schedule" \
  -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{
    "windows": [{"days": ["mon","tue","wed","thu","fri"],
                 "start": "06:00", "end": "18:00"}],
    "timezone": "America/New_York",
    "min_warm": 1
  }'
# → {"profile":"large","warm_schedule":{...},"active":true,
#    "detail":"Saved. The warm keeper applies this on its next tick (~1 minute)."}

# Read it back (any authenticated user; `active` = a window is open now).
curl -sS "$BASE/api/v1/compute-profiles/large/warm-schedule" \
  -H "Authorization: Bearer $TOKEN"

# Remove it — workers drain via the normal idle TTL.
curl -sS -X DELETE "$BASE/api/v1/compute-profiles/large/warm-schedule" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

While any window is active, the API's warm-schedule keeper (same cadence as the worker reaper) pre-starts min_warm worker slots for the profile and the idle reaper will not reap them; the first query of the morning lands on an already-attached warm worker. Outside every window nothing is protected — the normal worker_idle_ttl_seconds drains the tier back to zero after the last real query, so a query still running at the window's end is never killed mid-flight.

Semantics worth knowing:

  • Windows are same-day (start < end, minute precision, HH:MM 24h) and evaluated in the schedule's IANA timezone (DST handled). Express an overnight span as two windows. Up to 16 windows per profile.
  • min_warm is clamped to the profile's max_concurrent. PUT rejects a floor above capacity; if capacity later shrinks, the keeper clamps at enforcement time.
  • Admin-only to write; anyone can read. Writes are audited (compute.profile.warm_schedule.*). The schedule also appears on GET /api/v1/compute-profiles (warm_schedule, warm_schedule_active) and the live floor on GET …/{name}/worker-status (warm_floor).
  • It's declarative, not fired. There is no missed-run state: a restarted API re-evaluates the window on the next keeper tick (~60s) and converges — same model as the reaper itself.
  • Cost: a warm window is billed like any running worker pod for the whole window. One large (8 CPU / 32Gi) worker kept warm 6:00–18:00 Mon–Fri is ~60 pod-hours/week — size min_warm and the window accordingly.
  • The default profile takes no schedule (it runs inside the API pod; there is nothing to pre-start).

When to bump the always-on pods vs add a burst profile

Lakehouse queries run as embedded DuckDB inside the API and job-runner pods, so the "baseline" knob is those pods' memory, not a separate lakehouse pod.

SymptomRight knob
Steady-state interactive query load is hitting the API pod's memory limitRaise resources.api.memory (and cpu if needed). Sizes the interactive baseline.
Steady-state automation / maintenance SQL is hitting the job-runner's memory limitRaise resources.job_runner.memory (and cpu if needed).
One job (migration, big join) blows the pod when it runs but the baseline is fineAdd a compute_profiles.<name> block sized for that job, then call it from /api/v1/queries (or from the script). The always-on pods stay cheap, and the burst pod can pin a bigger, scale-to-zero node pool (see per-cloud setup).
Long-running INSERT … SELECT loops OOM a pod after a few batchesThe DuckDB connection-scoped leak. Move each batch to a burst profile — fresh subprocess per call resets the leak.

How SQL compute lands

What determines where SQL compute lands is the ATTACH the client uses. The lakehouse is postgres_catalog:

  • ATTACH 'ducklake:postgres:<DSN>' AS LAKE (DATA_PATH '...') — DuckLake is attached on the client with its Postgres catalog backend: catalog metadata lives in a dedicated ducklake_catalog Postgres database (reusing the deploy's existing Postgres server), and parquet reads + the query plan run locally on the caller in embedded DuckDB via httpfs. Compute scales with whatever pod runs the query. The CLI derives lakehouse.catalog_dsn from postgres.url when omitted, changing only dbname to ducklake_catalog.

This is the pattern ducklake#1151 added on 2026-05-12 and DuckDB 1.5.3 (released 2026-05-20) wires up end-to-end. Because the query plan always runs on the caller, burst SQL and compute profiles do not require a separate lakehouse service — the API/job-runner pod runs default, and a non-default profile runs the same ATTACH in a sandbox pod. The worker tier reuses attached workers when enabled; one-shot burst SQL leases a fresh pod per query when the worker tier is disabled. The CLI derives the catalog DSN and the chart wires the API + job-runner setup SQL to this ATTACH form (plus a CREATE SECRET store_creds (...) for httpfs reads). See the config.md lakehouse section.

If the DuckLake Postgres catalog resolves to a private RFC1918 or IPv6 ULA IP, add an exact sandbox egress allowlist in fi.sandbox.egress.private_cidrs so the profile's sandbox or worker pod can reach it. The chart does not derive this from the secret DSN; NetworkPolicy ipBlock rules need operator-supplied CIDRs.

The DuckLake Postgres catalog is multi-writer and battle-tested, so there are no concurrency/maintenance beta gaps to work around.

Run heavy SQL with a profile-bound Python step

Another way to run a heavy SQL workload is to wrap it in a Python step (or Fi run) with compute_profile: large. The sandbox image has DuckDB available, and Python steps can opt into lake access with attach_lake: true. The SDK (definite_lakehouse) exposes a write_table that handles >50k-row writes through staging. The script runs on the burst node, attaches the DuckLake Postgres catalog, reads inputs straight from object storage over httpfs, and writes results back to the lake.

Example: TPC-H benchmarks

Generate TPC-H data once, then run the 22 queries on a burst pod:

# automation_scripts/tpch_gen.py — generates SF=10 TPC-H data and writes it
# to the lake. Run this once as a pipeline step with compute_profile: large.
import duckdb
from definite_lakehouse import write_table

con = duckdb.connect(":memory:", config={"memory_limit": "28GB"})
con.execute("INSTALL tpch; LOAD tpch; CALL dbgen(sf=10)")
for table in ("nation", "region", "part", "supplier", "partsupp",
              "customer", "orders", "lineitem"):
    df = con.execute(f"SELECT * FROM {table}").fetch_arrow_table()
    write_table(name=f"tpch.{table}", arrow_table=df, mode="replace")
print("TPC-H SF=10 written to lake")
# automation_scripts/tpch_run.py — runs the 22 TPC-H queries against the
# DuckLake-backed tables, on the same burst profile. We attach the DuckLake
# Postgres catalog directly (the same path the API uses) and read parquet
# from the object store over httpfs, then time each query.
#
# Easiest path: let the SDK build the attach for you, which reads
# LAKEHOUSE_CATALOG_DSN / LAKEHOUSE_DATA_PATH and the active store creds
# (all forwarded when attach_lake: true) and wires up the store secret.
import json, time
from definite_lakehouse import attach  # returns the configured DuckDB conn with LAKE attached

con = attach()  # Postgres DuckLake catalog + httpfs parquet, SQL runs in this process
con.execute("INSTALL tpch; LOAD tpch")  # provides tpch_queries()

timings = []
for q in con.execute("FROM tpch_queries()").fetchall():
    qid, sql = q[0], q[1].replace("FROM ", "FROM LAKE.tpch.")
    t0 = time.perf_counter()
    con.execute(sql).fetchall()
    timings.append({"query": f"q{qid}", "ms": round((time.perf_counter() - t0) * 1000)})
print(json.dumps({"timings": timings}, indent=2))

Register both as automation scripts (definite run script create --name tpch_gen --content tpch_gen.py, same for tpch_run), then drop them into a pipeline:

# pipeline.yaml — create with: definite run automation create --definition pipeline.yaml
steps:
  - id: generate
    type: python
    config:
      script_name: tpch_gen
      compute_profile: large
      attach_lake: true
  - id: bench
    type: python
    config:
      script_name: tpch_run
      compute_profile: large
      attach_lake: true

A burst-pool node spins up on the first step, both steps reuse the same pool, and the node scales back down ~30s after the second step's sandbox lease is torn down.

For genuinely ad-hoc beefy SQL (one-off exploration, no pipeline overhead), open a Fi thread with compute_profile: large and ask the agent to run the query. Fi's bash tool inside the sandbox can invoke duckdb directly — same compute path, no automation plumbing.

Limits (v1)

  • Lake access from Python steps is explicit. A compute-profile Python step only receives LAKEHOUSE_* and object-store credentials when config.attach_lake: true is set. Set attach_lake: false to force no lake env.
  • adbc_sync / mssql_sync steps do not route to burst sandboxes yet. They accept config.compute_profile for forward compatibility but run on their existing executor paths. pg_sync honors a non-default profile when the source connection is portable (no SSH bastion, no pasted SSL certs); non-portable sources stay in-pod and log that the profile was ignored.
  • Burst SQL needs the Fi sandbox subsystem. If the deployment has Fi disabled (fi.enabled: false), a non-default compute_profile on /api/v1/queries returns 503 rather than silently falling back to the API pod's embedded connection — silent fallback would defeat the leak-isolation guarantee the feature exists to provide.
  • No authorization gating. Any authenticated user can pick any declared profile. Cost is controlled by what the operator declares in config.yaml, and the worker tier enforces max_concurrent as the attached SQL worker slot count for each profile.

Default deployments

A deployment with no compute_profiles: block uses the default sandbox settings from fi.sandbox.*. Add named profiles only when specific workloads need different resources, scheduling hints, or warm-pool behavior.

Troubleshooting

SymptomCauseFix
SandboxClaim stuck Pending, no pod scheduledNode pool for the profile doesn't exist or its taint doesn't match the profile's tolerationskubectl get nodes -L cloud.google.com/gke-nodepool (etc.) to confirm the pool exists; compare the pool's taint to compute_profiles.<name>.tolerations
Worker query times out attaching ducklake:postgres while the default query path worksThe profile's sandbox/worker pod cannot reach a private Postgres catalog endpoint because sandbox NetworkPolicy excludes RFC1918 by defaultAdd fi.sandbox.egress.private_cidrs: ["<catalog-ip>/32"] and keep private_ports at [5432] unless your catalog uses a custom port, then re-run definite upgrade
GET /api/v1/compute-profiles returns only default after adding a profileHelm release wasn't re-rendered, or FI_COMPUTE_PROFILES_JSON wasn't refresheddefinite upgrade --config config.yaml, then kubectl rollout restart deploy/definite-api -n definite
Warm-pool sandboxes still on the old image after helm upgradeKnown agent-sandbox footgun — the controller keeps the warm pool's bound pods until the SandboxClaim is deletedkubectl delete sandboxes.agents.x-k8s.io -n definite -l app.kubernetes.io/component=fi-sandbox — the pool refills with the new image.
400: unknown compute profile 'foo'. Known: default, largeThe frontend or CLI sent a profile name not in the chart-rendered registryCheck config.yaml's compute_profiles: keys and re-run definite upgrade if you added one but didn't deploy it.