Autonomous Agents

An agent is a monitor. When it runs it:

1. Probes — runs a query (SQL or Python) against the lakehouse.
2. Decides — hands the probe result, plus its memory of recent runs, to an LLM that returns a structured judgment: should we act, and why.
3. Acts — if the LLM says act (and a cooldown allows it), it fires an action.

Canonical example: "sales up 20% week-over-week → notify / buy more inventory." The probe computes the number; the LLM judges whether that warrants action; the action posts to Slack, hits a webhook, runs SQL, or hands off to a Fi agent.

agent definition → trigger (cron / event / manual) → probe → LLM decision → (cooldown) → action

Agents are not automations: an automation runs an ordered list of steps blindly, with no branching. An agent is a single conditional — probe, judge, maybe act. The two share the job-runner poll loop and the cron helpers and nothing else.

Enabling agents

Agents are off by default. Enabling them lets the job-runner make outbound LLM calls, so air-gapped clusters should leave them off.

# config.yaml
agents:
  enabled: true

This maps to AGENTS_ENABLED on the API and job-runner. When disabled, agent reads still work but create / run are rejected with a 403.

Letting Fi write the agent

The fastest way to create an agent is to ask Fi. The definition is just JSON to POST /api/v1/agents, and definite run agent create is a command Fi has in its sandbox — so "Fi, watch my sales table and ping #ops if revenue jumps 20% week-over-week" is enough; Fi composes the probe, instructions, and action and creates the agent. The hand-written definition below is the manual path.

Agent definition

{
  "name": "Sales surge watcher",
  "probe": {
    "type": "sql",
    "sql": "SELECT this.t AS this_week, last.t AS last_week FROM (SELECT sum(amount) t FROM analytics.sales WHERE order_date >= current_date - 7) this, (SELECT sum(amount) t FROM analytics.sales WHERE order_date >= current_date - 14 AND order_date < current_date - 7) last"
  },
  "decision_instructions": "Act if this week's sales are more than 20% above last week's.",
  "mode": "simple",
  "action_config": { "action_type": "slack_webhook", "integration": "slack-ops" },
  "cron_schedule": "0 9 * * 1",
  "cron_timezone": "America/New_York",
  "cooldown_seconds": 604800
}

An agent may have a cron_schedule, an event_trigger, both, or neither (manual-only) — they are independent.

Probes

sql

{ "type": "sql", "sql": "SELECT sum(amount) AS total FROM analytics.sales" }

Runs against the lakehouse. May be an aggregate. The result set (capped at 200 rows) becomes probe_result. SQL probes are stateless.

python

{ "type": "python", "script_name": "sales_probe", "requirements": ["pandas"] }

A Python probe runs through the same machinery as an automation python step (uv run, ephemeral venv, requirements, integrations, baked definite_lakehouse SDK). Specify the body with exactly one of script (inline), script_id, or script_name.

A Python probe is stateful: it is handed the agent's durable memory and run history, and returns its result — and an optional new watermark — via the SDK:

from definite_lakehouse import agent_input, emit_probe, query

ctx = agent_input()                       # {state, history, agent_id, run_id}
last = ctx["state"].get("last_total", 0)

total = query("SELECT sum(amount) AS t FROM analytics.sales").to_pylist()[0]["t"]

emit_probe(
    result={"total": total, "previous_total": last},   # fed to the LLM
    state={"last_total": total},                        # watermark for next run
)

emit_probe() must be called exactly once; the result object is the probe_result, and state (when given) merges into the agent's durable memory.

The decision

The probe result, the durable memory, and the recent run history go to an LLM, which is forced to call a single submit_decision tool returning {should_act, reasoning, action_args, memo}. The response is always structured JSON — there is no prose to parse.

Which LLM? By default the agent uses the deployment LLM — the same llm: provider/key configured in config.yaml that Fi uses. Supported for the default path: anthropic and litellm (the in-cluster proxy). Set decision_integration (+ decision_model) to override per-agent, or when the deployment uses a provider the default path can't reach (bedrock, vertex, azureOpenai without litellm).

Memory

An agent has two layers of memory, both shown to the LLM each run:

• Episodic — the agent_runs log. One immutable row per run with the probe result, decision, action result, and any error. The last decision_history_limit rows (including failed runs) are fed to the LLM, so it can see "last run errored", "last run posted this to Slack", and judge trends.
• Durable — the agents.state scratchpad. A JSONB blob that persists across runs. A Python probe reads and rewrites it (watermarks); the decision LLM writes a memo into it — "a note to your future self" — which is shown back on the next run. This is how an agent remembers "I already alerted on the May spike" without re-deriving it.

Action modes

simple

One predefined action, with arguments the LLM supplies in action_args:

fi

Hands the situation to a Fi agent run that reasons and acts with its full toolset. Requires fi.enabled. action_config.prompt_template is the base prompt; the decision reasoning and probe result are appended.

Cooldown

cooldown_seconds is the hard guard against re-firing the same alert. After an action fires, last_acted_at is stamped; any action within cooldown_seconds of it is suppressed — the run still succeeds, decision.suppressed_by_cooldown is true, no action runs. Checked at execution time, so manual runs respect it too. The LLM also sees recent history and its own memo as a soft secondary guard.

Triggers

An agent fires on any of:

• Cron — cron_schedule (5-field, min interval once per minute) + cron_timezone. The job-runner scheduler tick enqueues overdue agents.
• Event — event_trigger, fired when an automation run succeeds:
  • {"type": "automation", "automation_id": "<pipeline id>"} — when that pipeline succeeds.
  • {"type": "table", "table": "main.orders"} — when any succeeded automation run had a pg_sync step that landed rows in that table. Use this for event-driven runs after new rows arrive.
• Manual — POST /api/v1/agents/{id}/runs or definite run agent run.

Event-triggered runs go through the same cooldown gate.

CLI

definite run agent create --name "Sales surge watcher" \
  --definition agent.json \
  --cron "0 9 * * 1" --cron-timezone America/New_York \
  --cooldown-seconds 604800

definite run agent run <agent_id>
definite run agent status <run_id>     # probe_result / decision / action_result
definite run agent list
definite run agent runs <agent_id>     # history

API

GET    /api/v1/agents
POST   /api/v1/agents
GET    /api/v1/agents/{agent_id}
PATCH  /api/v1/agents/{agent_id}
DELETE /api/v1/agents/{agent_id}
POST   /api/v1/agents/{agent_id}/runs
GET    /api/v1/agents/{agent_id}/runs
POST   /api/v1/agents/{agent_id}/trigger-schedule
GET    /api/v1/agent-runs/{run_id}

Same bearer-token auth as the rest of the API. Create / run require agents.enabled. trigger-schedule (advance next_fire_at by one tick for testing) requires admin.

Cost notes

• simple mode: one bounded LLM decision call per run (max_tokens ~1024, history capped by decision_history_limit). Cost scales with trigger frequency — pair a frequent cron/event trigger with a cooldown.
• fi mode: the decision call plus a full Fi agent run, but the Fi run only happens when should_act is true and the cooldown has passed.

Current limits

• One probe and one action per agent — no multi-step branching.
• The probe runs with full lakehouse privileges (operator-authored, like an automation sql / python step). Don't expose agent creation to untrusted users.
• No missed-run replay — an agent that falls more than an hour behind its cron skips the backlog.
• Durable state is persisted whenever the probe succeeds; a probe should be idempotent (it may re-run if a later stage fails).