LLM Setup

Fi, Pi, and autonomous agents all call an LLM. The provider is configured by the llm: block in config.yaml and resolved at deploy time — when it's misconfigured (no key, model name the provider doesn't host, IAM not yet approved) Fi surfaces an "LLM not reachable" banner in the thread and points you here.

This page covers the three supported providers (Anthropic direct, AWS Bedrock, Google Vertex), how to switch between them, the model IDs the deployment ships with, and the most common reasons Fi can't reach an LLM on a fresh install.

The llm: block

The llm: block in config.yaml is a tagged union by provider. On a fresh deploy the simplest working configuration is:

llm:
  provider: anthropic
  model: claude-sonnet-5
  api_key: { env: ANTHROPIC_API_KEY }

provider defaults to anthropic and model defaults to claude-sonnet-5 when omitted, so the only required field is api_key. api_key is a SecretRef — supply it as an env var ({ env: ANTHROPIC_API_KEY }), a file ({ file: /etc/secrets/... }), or a pre-created Kubernetes Secret.

Anthropic — the easiest path

The fastest way to get Fi working is an Anthropic API key.

1. Sign in at console.anthropic.com and create a key under Settings → API keys.

2. Set the key in your environment before running definite init / definite upgrade:

   export ANTHROPIC_API_KEY=sk-ant-...
   

3. Confirm the llm: block references it:

   llm:
     provider: anthropic
     model: claude-sonnet-5
     api_key: { env: ANTHROPIC_API_KEY }
   

4. Apply the change:

   definite upgrade
   

The new key reaches the definite-secrets Kubernetes Secret on the next helm apply. Fi and Pi pick it up on the next run — no pod restart needed.

You can also paste an Anthropic (or OpenAI) key directly from Settings → Models → Providers on a self-hosted deployment: it is stored encrypted and takes effect on the next Fi run, no definite upgrade needed. On hosted (managed-proxy / litellm) deployments those key fields do not apply — models are served through the in-cluster LiteLLM proxy, so pick your Fi model from the Settings → Models catalog instead.

Rotating the key

To rotate, update the env var to the new key and run definite upgrade. The chart re-renders the Secret in place; the next Fi/Pi run uses the new value.

AWS Bedrock

Bedrock is the right choice when the deployment lives inside an AWS account that already has IAM/IRSA wired up. There is no API key — auth flows through the chart's ServiceAccount + an IRSA-bound IAM role.

1. Request model access

Bedrock requires explicit model-access approval per region. Open the console and request access to the Claude models the deployment uses:

https://console.aws.amazon.com/bedrock/home#/modelaccess

Switch the region picker to the same region you'll set in llm.region (e.g. us-east-1). Request access to Anthropic Claude Sonnet and Anthropic Claude Haiku — approval is usually instant for paying accounts but can take hours on a new one. Until it's approved every InvokeModel call returns AccessDeniedException and Fi shows the banner.

2. IAM / IRSA

The chart's ServiceAccount must be annotated with eks.amazonaws.com/role-arn=<bedrock-invoker-role>. The role's trust policy must bind your EKS OIDC provider to system:serviceaccount:<namespace>:<sa>, and its permissions must allow bedrock:InvokeModel on the inference profile ARNs plus the underlying foundation models the profile dispatches to. (Cross-region inference profiles route between regional endpoints, so the policy needs both ARNs.)

3. config.yaml

llm:
  provider: bedrock
  region: us-east-1
  model: claude-sonnet-4-6
  inference_profile_prefix: us   # geo prefix; see "Non-US regions" below

Set llm.litellmUpstream: bedrock if you're using the in-cluster LiteLLM proxy (the default litellm provider path). Direct Bedrock uses a cross-region inference profile. The proxy accepts the friendly model name and maps it to the profile ID — see the model table below.

Non-US regions

Claude on Bedrock is only invocable through a cross-region inference profile whose geo prefix must match your region: us (United States), eu (Europe), apac/au (Asia Pacific / Australia). The default is us. For a non-US region, set the prefix in both Terraform and config.yaml (the API, Fi sandbox, and LiteLLM all re-base model IDs onto it; you can leave llm.model as a friendly name like claude-sonnet-4-6):

llm:
  provider: bedrock
  region: ap-southeast-2
  model: claude-sonnet-4-6
  inference_profile_prefix: au

# terraform.tfvars — the `au` profile dispatches compute to ap-southeast-4,
# so the IAM policy must allow both regions' foundation-model ARNs.
bedrock_inference_profile_prefix  = "au"
bedrock_inference_profile_regions = ["ap-southeast-2", "ap-southeast-4"]

Check the AWS docs for which regions your chosen profile fans out to, and make sure model access is granted in the Bedrock console for that region.

Bedrock model IDs shipped with the deployment

The LiteLLM template ships these mappings. Friendly names are kept stable across providers so changing upstream doesn't break LLM_MODEL.

Gemini (gemini-3-5-flash) is Vertex-only and is not offered on Bedrock hosted installs (central onprem.hosted_llm_model.upstreams filters the Settings picker by llm.litellmUpstream).

The us. prefix shown above is the default; it routes the call across the US regional Bedrock endpoints (cross-region inference profile). For non-US regions set llm.inference_profile_prefix (see Non-US regions) and the prefix is swapped accordingly. Bedrock has deprecated the legacy on-demand foundation-model IDs for these models — only the inference-profile ARN is invocable.

Google Vertex AI

Vertex is the right choice when the deployment lives inside a GCP project with Workload Identity wired up.

1. Enable the API and grant access

In your GCP project, enable the Vertex AI API. Grant the chart's GCP service account roles/aiplatform.user on the project. The chart's Kubernetes ServiceAccount must carry the iam.gke.io/gcp-service-account=<gcp-sa>@<project>.iam.gserviceaccount.com annotation so Workload Identity binds the two.

For Claude on Vertex you must also enable each model in the Model Garden of the region you'll use — the per-model "Enable" button issues the publisher binding the call needs.

2. config.yaml

llm:
  provider: vertex
  project: my-gcp-project
  region: global
  model: claude-sonnet-5

For Gemini and current Claude defaults, use region: global; some regional endpoints return 404 even with the right publisher ID if that model is not enabled there.

Vertex model IDs shipped with the deployment

Friendly model names use dashes (k8s/env-var friendly). The upstream Vertex path keeps the dot between version digits — e.g. gemini-3.5-flash, not gemini-3-5-flash. The dashes-only form 404s.

Azure OpenAI

Azure OpenAI is the right choice when the deployment lives in an Azure subscription with an Azure OpenAI resource. Auth is an API key (the resource's key), not IAM.

llm:
  provider: azure_openai
  endpoint: https://acme.openai.azure.com
  deployment: gpt-4o
  api_key: { env: AZURE_OPENAI_API_KEY }

Unlike the other providers, there is no separate model field. Fi uses the deployment name as the model id when it calls Azure (the request path is {endpoint}/openai/deployments/{deployment}/responses), so two rules apply:

• Name your Azure OpenAI deployment after a supported OpenAI model id, e.g. gpt-4o. A deployment named anything else fails with azure-openai-responses/<name>: unknown to pi-ai.
• Azure OpenAI serves only OpenAI models. Claude friendly names like claude-sonnet-4-6 do not work here; pick anthropic, bedrock, or vertex if you need Claude.

The chart wires AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY into the Fi sandbox from the llm: block, so no other setup is needed once the deployment is named correctly.

Autonomous agent decisions and automations that fall back to the deployment LLM (rather than a per-agent decision_integration) currently support anthropic, openai, and litellm only. Interactive Fi works on Azure OpenAI; agent-decision fallback on Azure is a known follow-on.

Switching providers or changing models

The llm.provider and llm.model fields in config.yaml are the ground-truth — change them and run definite upgrade. The provider switch restarts the LiteLLM proxy (which re-renders its model list); the model switch takes effect on the next Fi run.

On a fresh install the API reconciles the fi_model_config singleton to config.yaml's llm.provider / llm.model on startup, so config.yaml is authoritative out of the box (the 007 seed's anthropic / claude-sonnet-5 default doesn't leak onto a non-anthropic deployment). It only rewrites the row while it's still untouched (updated_by IS NULL), so an admin's later Settings → Models choice survives restarts.

For a per-deployment runtime model override (no definite upgrade needed), admins can change the default in Settings → Models. That writes to the fi_model_config row in the app DB; the next Fi run reads it and routes the call through the in-cluster LiteLLM proxy. The chart-level api_key / IAM remains the auth source — the UI just picks the model.

Capping daily LLM spend

llm.budget puts a hard ceiling on what this deployment can spend on model calls in a day. It is off by default:

llm:
  provider: litellm
  budget:
    enabled: true
    dailyUsd: 250

Run definite upgrade to apply it.

The cap is enforced by the in-cluster LiteLLM proxy, which every model call on the deployment goes through — Fi, automations, and agent decisions alike. Once the day's spend passes dailyUsd, the proxy rejects further calls with HTTP 429 before they reach the provider, and Fi shows "This deployment has used its daily LLM budget" instead of a provider error. The API logs one line when spend crosses 80% of the cap and one when it hits the cap.

Details worth knowing before you turn it on:

• It requires Postgres. The running total lives in a litellm schema on the database from postgres.url, which is what makes one cap hold across both proxy replicas and survive a restart. The proxy creates the schema on first boot; that boot is slower than usual.
• It fails open. If Postgres is unreachable, model calls keep working and spend is briefly unmetered. A database problem will not take Fi down.
• The window resets at 00:00 UTC, on a fixed daily grid that does not drift. The reset is applied by a periodic job rather than a clock trigger, so the counter can take a few minutes past midnight to zero.
• Cost is measured, not estimated. LiteLLM prices each response from its own cost map, which rates fresh input, output, cache writes, and cache reads separately per model.
• The cap can be overshot slightly. Spend is written every ~10s and read through a ~60s cache, so a burst can cross the line before the gate closes. That bounds a runaway to minutes rather than hours.

Raise dailyUsd (or set enabled: false) and run definite upgrade to lift a cap that is blocking legitimate work.

Troubleshooting

"Fi can't reach an LLM yet" banner

This banner appears in the Fi thread whenever the most recent run failed with an LLM error code. The most common causes on a fresh deploy:

• No ANTHROPIC_API_KEY set, or the key was rotated. Re-export the env var and run definite upgrade. Test with a short prompt in Fi.
• Bedrock model access pending. Check https://console.aws.amazon.com/bedrock/home#/modelaccess — the requested models must show Access granted in your llm.region.
• Vertex API not enabled, or roles/aiplatform.user missing. Confirm the GCP service account, the Workload Identity binding, and the per-model Model Garden enable.
• Wrong llm.model name. Must match one of the friendly names in the tables above. Anything else returns "configured LLM model is not known to Pi" from the sandbox.

Click Configure in Settings on the banner to jump to Settings → Models for a model swap, or Setup guide to come back to this page. The banner clears on the first successful run.

Other Fi failures

If the banner doesn't appear and Fi is still failing, the error is unlikely to be LLM-related. Open the run from the Fi sidebar, expand the activity strip, and read the error string — it often points at a missing integration secret, a SQL syntax error, or a sandbox networking issue (see fi.sandbox.egress).