Architecture

Overview

Definite on-prem is a single-tenant, self-hosted distribution of Definite, deployed via a Rust CLI that wraps Helm. One YAML config in, working deployment out.

                       ┌────────────────────────────────────────┐
                       │           Customer Kubernetes          │
                       │  ┌────────────┐    ┌───────────────┐   │
                       │  │  Frontend  │◄──►│      API      │   │
                       │  │  (React)   │    │   (FastAPI)   │   │
                       │  └────────────┘    └───────┬───────┘   │
                       │                            │            │
   user ───https───►   │   Ingress (cert-manager)   │            │
                       │                            ▼            │
                       │              ┌─────────────────────┐   │
                       │              │   Job Runner        │   │
                       │              │  (Python pipelines) │   │
                       │              └─────────────────────┘   │
                       │                                         │
                       │   Embedded DuckDB + DuckLake run        │
                       │   inside the API & Job Runner pods —    │
                       │   no separate query server.             │
                       └─────────────────┬─────────────┬────────┘
                                         │             │
                            ┌────────────▼─┐     ┌─────▼──────┐
                            │  Postgres    │     │ Object     │
                            │ app DB +     │     │ store      │
                            │ DuckLake     │     │ (parquet,  │
                            │ catalog      │     │  httpfs)   │
                            └──────────────┘     └────────────┘

Components

CLI (Rust)

Single static binary. Three jobs:

1. Preflight (definite doctor): probe Postgres, object store, Kubernetes, LLM provider. Fail loud and early.
2. Deploy (definite init, definite upgrade): map config.yaml → Helm values → helm upgrade --install.
3. Helm export (definite export-helm): emit the raw chart for operators who prefer to review or apply it with helm directly.

The CLI never owns long-running state. It shells out to kubectl and helm. Everything it does is auditable.

Helm chart

The actual deployment artifact. Versioned alongside the CLI. Templates:

• api.yaml - Deployment + Service for the FastAPI service
• frontend.yaml - Deployment + Service for the React UI
• job-runner.yaml - Deployment for the Postgres-backed pipeline scheduler
• ingress.yaml - Routing + TLS
• secret.yaml - Consolidated Secret (managed by chart) for inlined credentials
• configmap.yaml - Non-secret runtime config

API (Python + FastAPI)

Single-tenant. No team_id, no RLS. Reads from Postgres for application state, runs lakehouse queries in an embedded DuckDB connection that attaches the DuckLake catalog, and calls the LLM provider for Fi.

Access control is enforced here, in the API — it is the only gateway to the lakehouse. Three independent layers: application roles (viewer/editor/admin), per-resource content sharing, and table-level data-access roles. See permissions.md.

Frontend (React)

Home, Fi, query, catalog, data apps, automations, docs, integrations, and settings.

Lakehouse (embedded DuckDB + DuckLake, Postgres catalog)

There is no separate lakehouse query pod. The DuckLake catalog lives in a dedicated Postgres database (ducklake_catalog, reusing the deployment's existing Postgres server). The API and job-runner each open an embedded DuckDB connection that attaches the catalog directly (ATTACH 'ducklake:postgres:<DSN>'); the query plan runs locally in that process and bulk parquet data is read from the configured object store over httpfs. Compute therefore scales with whatever pod is running the query.

Job runner

Postgres-backed worker pool. Cron + retries + idempotency. Not Temporal. A SQL-only design keeps the operational surface small.

Compute profiles (burst tiers)

The three core deployments — api, frontend, job-runner — are small and always-on (the lakehouse is embedded DuckDB inside the api and job-runner pods, not a separate deployment). For workloads that need bigger compute on demand (heavy Python pipelines, Fi sessions crunching a large dataset, ad-hoc SQL benchmarks like TPC-H) the operator declares compute profiles in config.yaml:

compute_profiles:
  large:
    resources:
      requests: { cpu: "4", memory: "16Gi" }
      limits:   { cpu: "8", memory: "32Gi" }
    node_selector:
      cloud.google.com/gke-nodepool: definite-burst
    tolerations:
      - { key: definite-app/burst, operator: Equal, value: "true", effect: NoSchedule }
    warm_pool_size: 0     # cold-start; pool scales to zero when idle

The chart renders each profile as a SandboxTemplate plus an optional SandboxWarmPool. At runtime a Fi thread or a pipeline Python step picks a profile by name; the workload runs on a Kubernetes pod sized for that tier, usually on a dedicated burst node pool that scales to zero when nobody's using it. A deployment with no compute_profiles: block uses the default sandbox sizing.

Where SQL compute lands follows from the postgres_catalog lakehouse design: the DuckLake catalog lives in Postgres, every client attaches it directly in embedded DuckDB, and the SQL plan executes on whatever pod runs the query while parquet is read from object storage over httpfs. The default compute profile runs queries in the API or job-runner pod's embedded connection; a non-default profile dispatches the query to a burst sandbox that opens its own embedded DuckDB and attaches the same catalog.

The full operator + user guide — including per-cloud node-pool setup (GKE / EKS / Karpenter / AKS) and a TPC-H benchmark recipe — is in compute-profiles.md.

External dependencies (customer-provided)

Product boundaries

Definite on-prem is designed around these boundaries:

• One organization per deployment; it is not a multi-tenant control plane.
• Semantic modeling is built into the lakehouse workflow rather than a separate Cube-style service.
• Data movement is built from integrations, sync steps, automations, and Fi-assisted connector workflows rather than a full packaged connector library.