Embeddings & Vector Search

Vector search over DuckLake columns: embed a text column once, then rank its rows against a search query by cosine similarity — all on DuckDB, no separate vector database and no GPU node pool.

Why this shape

Embedding work splits cleanly into two workloads of very different weight, and the design follows that split:

So there is no standalone embedding pod. The API embeds queries itself; batch column embedding is enqueued as an ordinary python automation step.

The model runs on CPU. Embedding models are ~100–350M params — the default BAAI/bge-small-en-v1.5 is 384-dim and ~130 MB on disk. fastembed (ONNX runtime, no PyTorch) is used so the image grows by ~hundreds of MB rather than ~2 GB.

Configuration

# helm/values.yaml
embeddings:
  model: "BAAI/bge-small-en-v1.5"   # any fastembed-supported model

The model name is read by the API (query embedding) and baked into the batch step (column embedding) so the two always agree. The API pod's memory limit is 3Gi to hold the loaded model; a larger model needs a further bump.

The model is fetched from HuggingFace on first use. Air-gapped installs must pre-seed the fastembed cache (mount it, or bake it into the image) — the first search otherwise fails with a 503 embedding model unavailable.

Storage model

The batch step writes a table with three columns:

• the source id column (unchanged name + type)
• the source text column
• embedding — the vector as a JSON-array VARCHAR

embedding is a string, not a native FLOAT[], because the lakehouse write SDK maps list/array Arrow types to VARCHAR. Search casts it back with CAST(embedding AS FLOAT[]) and ranks with the core list_cosine_similarity function — a brute-force scan, no vss extension and no HNSW index.

That is correct and fast up to a few hundred thousand rows. Past that, materialize a native-typed table and add an HNSW index.

API

Embed a column (batch)

POST /api/v1/embeddings/columns
{
  "schema": "main",
  "table": "documents",
  "id_column": "id",
  "text_column": "body",
  "dest_schema": "main",          // optional; defaults to source schema
  "dest_table": "documents_emb",  // optional; defaults to <table>__embeddings
  "batch_size": 256               // optional
}

Enqueues an ad-hoc job-runner run (a one-step python pipeline) and returns the run plus the destination table. Poll the run via GET /api/v1/automations/runs/{run_id}.

Search

POST /api/v1/embeddings/search
{
  "schema": "main",
  "table": "documents_emb",   // the embedded table
  "id_column": "id",
  "text_column": "body",
  "query": "some dog I need to find",
  "limit": 10
}

Embeds query in-process and returns rows of (id, text, similarity), best match first.

Misc

GET /api/v1/embeddings/model   -> { "model": "BAAI/bge-small-en-v1.5" }

Limitations (first cut)

• The batch step loads the whole source column into the job-runner process before embedding. Fine for moderate columns; very large columns need a larger job-runner Job or a chunked rewrite.
• Brute-force search only — no ANN index (see Storage model).
• Re-embedding replaces the destination table wholesale; there is no incremental "embed only new rows" mode yet.