Skip to main content

Budgets & FinOps

Every call through the gateway is priced and attributed, and agents can carry budgets that warn or hard-block before overspend.

Pricing

Cost is computed from a versioned pricebook shipped with KnoxCall (packs/ai-gateway-pricebook/), covering current Anthropic, OpenAI, and Gemini models with dated price rows. Two guarantees matter:
  • Never silently $0. If a model isn’t in the pricebook, the call is recorded with cost_source = 'unpriced' and surfaced through an unpriced_request_count — never booked as free. Provider inference falls back across providers rather than dropping to zero.
  • No remote config fetch. The pricebook is local — there is no run-time call to a third-party price feed. A weekly in-process check flags any observed-but-unpriced models via an alertable ai_gateway.unpriced_model audit event, so you update the manifest deliberately.
Each usage row records cost_source (pricebook / unpriced / cache_hit).

Embedding calls are metered too

If an agent runs the semantic cache (cache_mode: "semantic"), every cache lookup embeds the request — which is a real, billable provider call. Those calls are metered: they get their own usage rows, they appear in your exports, and they count against the agent’s budget and against a token’s signed daily caps. They were not, until 2026-08-28. If you turned semantic caching on before then, expect your row count and your metered spend to rise: the money was always being spent, and only the accounting changed. A budget can now trip on cache traffic. One deliberate asymmetry: under a token carrying a signed max_cost_per_day_usd, an embedding on a model that is not in the pricebook is not purchased at all. Semantic matching degrades to exact matching rather than making a spend the cap has no way to count. Since 2026-09-08 that degrade is no longer silent: the request’s usage row records it, and the Agent’s Budgets & cache tab counts it and names the reason.

When a reply’s usage cannot be read

Spend is read from the provider’s reply itself, from the usage report each provider sends:
  • usage for Anthropic, OpenAI and the OpenAI-compatible vendors (including a bare total_tokens).
  • usageMetadata for Gemini.
  • usage.inputTokens, or the x-amzn-bedrock-*-token-count headers, for Bedrock.
  • billed_units for Cohere.
A streamed reply is read from its own usage events:
  • Anthropic’s message_start and message_delta.
  • Gemini’s usageMetadata.
  • Groq’s x_groq.usage.
  • OpenAI’s final usage chunk. OpenAI sends that chunk only when the request asks for it, so when you stream from an OpenAI agent without stream_options.include_usage, KnoxCall asks for it on your behalf. Before the stream reaches you it removes the extra chunk and the "usage": null field OpenAI then adds to every other chunk, so you receive the stream you would have received without it. If you asked for usage yourself, you get it unchanged. Bedrock and Cohere do not stream through the gateway.
KnoxCall asks every provider for an uncompressed reply (Accept-Encoding: identity). It can still read a compressed reply, within limits: at most 25 MiB decoded and 2 seconds, two at a time per worker, and an 8 MiB window for zstd. If a paid reply’s usage still cannot be read, it is charged an estimate, never $0. That covers:
  • a reply that cannot be decoded, or that is over those limits;
  • a generation or embeddings reply that carries no usage report;
  • a stream that ended without one;
  • a streamed reply that was refused unread.
The estimate is:
  • Input: the request’s text at about four characters a token. Each inline image, document or audio clip counts as 1,600 tokens instead of its base64 length.
  • Output, when the reply was read: its text at about four characters a token.
  • Output, when the reply could not be read: the smaller of the request’s output ceiling (max_tokens and its equivalents) and the agent’s per-call cap. If neither is set, 4,096.
  • Embeddings: no output tokens.
How far off it is. For English prose, four characters a token is close to the provider’s own count. The estimate also counts every other string in the request, including the model name and role labels, so a short request overstates by a few tokens. It understates text in scripts that take more tokens per character, such as Chinese, Japanese and Korean (about one or two characters a token). An unread reply’s output ceiling usually overstates by far the most, because a reply rarely uses its whole ceiling. How you can tell. A buffered response carries X-Knox-AI-Usage-Estimated naming the reason. A streamed response cannot, because its headers have already been sent when the stream ends. The usage row does not yet say it is an estimate: it stores the estimated tokens and cost exactly as it stores a provider’s count, so exports and dashboards show it as real. A marker on the row is planned. When nothing is estimated. None of these are charged an estimate, and none of them is charged by this meter at all:
  • a call to an endpoint that answers with no usage report as normal (token counting, moderations, Gemini :embedContent), unless its reply could not even be decoded;
  • any request that is not a POST, including fetching a stored response;
  • a batch operation (a batch is charged when its results are fetched);
  • an OpenAI Responses call sent with background: true, or one that comes back queued or in_progress.

Budgets

Attach a budget to an agent with a daily and/or monthly USD cap and an overage action: A blocked call gets HTTP 429, not 402:
with X-Knox-AI-Budget-Pct alongside it. (Earlier revisions of this page said 402. 429 is what the gateway has always sent — treat a budget block as a back-off signal, and reach for the same retry logic you use for a rate limit.) X-Knox-AI-Budget-Pct carries the current utilization whenever there is something to say — over the cap on a warn agent, or from 95% of it on any agent. X-Knox-AI-Budget-Warning is set only by the warn action at the cap itself. A call comfortably inside its budget gets neither header, so treat their absence as “nothing to report”, not as “no budget configured”. Budgets are stored in a cross-worker store (Redis when enabled) so the cap holds across all workers, not per-process.
fallback is enforced. Earlier revisions of this page said it was not, and that was true until 2026-08-25: the handler branched only on block and warn, so an agent set to fallback proceeded on its expensive model at full price. It now serves the request from the agent named by fallback_agent_id — see Falling back to a cheaper agent for what is and is not swapped.
A budget cannot be enforced if its counter is unreadable. The spend counter lives in a cross-worker store; if that store is unavailable the gateway cannot know how much has been spent. It sets X-Knox-AI-Budget-Unmetered: 1, and then:
  • on a block agent it refuses with 503 budget_unenforceable rather than serve a request it cannot count;
  • on warn and fallback agents it serves the request, unmetered, and suppresses X-Knox-AI-Budget-Pct — a percentage computed from a frozen counter would read 0.0%, which is the lie in header form.
Treat X-Knox-AI-Budget-Unmetered as “this request was not counted against your cap”.

Failover chains (a different kind of fallback)

Not to be confused with the budget action above, and this one is implemented. An agent can carry an ordered list of fallback_route_ids. When the primary upstream answers 5xx (or the connection fails), the gateway tries each fallback route in order and serves the first non-5xx response. Each hop is a real, separately-authorised call, not a retry of the same one:
  • the fallback route resolves its own credential, so provider A’s key is never sent to provider B, and provider B’s linked-secret domain lock is checked;
  • a fallback the presenting token’s scope does not cover is skipped, so a 5xx cannot become a way to reach an upstream the token was never granted;
  • the fallback is loaded in the same environment as the primary, so a kp_test_ token cannot reach a Live route by way of a failure;
  • the usage row records the route that actually served, so cost lands against the provider that did the work.
A 4xx from the primary is returned as-is: failover is for a provider that is down, not for one that refused you.

Falling back to a cheaper agent

Set budget_overage_action: "fallback" and fallback_agent_id to another agent of yours. Past the cap, the request is served using that agent’s upstream route and default model — the two things that determine cost — and the response carries X-Knox-AI-Budget-Fallback: <slug> plus X-Knox-AI-Budget-Fallback-Model so your client can see which agent answered.
Your policies do not change. The firewall verdict, the PII stack and the tool allowlist of the agent you addressed stay in force; the fallback agent supplies only the route and the model. If it supplied its policies too, spending your own budget out would be a way to reach a laxer configuration — a control anyone could switch off with a credit card.The fallback route must be in the same data space (Live or Test) and, if the addressed agent is region-pinned, in the same region. A fallback that fails either check is not used.
If the fallback is configured but unusable — the agent was deleted, paused, has no primary route, or points at itself — the request is served on the original route and the response says so with X-Knox-AI-Budget-Fallback: unavailable. It is never silently served at full price. The calling token’s signed scope applies to the fallback as well. If the token’s egress_routes, models or providers do not cover the fallback agent’s route and model (every sub-request of a batch included), the request is refused with a 403: route_not_in_token_scope, model_not_in_token_scope or provider_not_in_token_scope. It is not served on the fallback, and it is not served at full price on the original route either. A token with a daily cost cap is refused token_scope_unpriced_model when the fallback’s model has no price. A hard agent or gateway cap gets budget_unpriced_model in the same case, because spend on an unpriced model would never move the counter. A call the fallback serves is priced at the fallback agent’s model and provider. Its usage row records that model and provider, so per-model views (/v1 usage by_model, the usage export with group_by=model, the hourly roll-up) show fallback traffic under the fallback’s model. If the fallback’s route fails, the request fails over to the fallback agent’s own fallback routes, never to the agent you addressed.

Caps carried by the token itself

An agent budget is set by you, the operator, on the agent. A capability scope can additionally carry max_cost_per_day_usd and max_tokens_per_day on one token, so a single credential can be confined below the agent’s own cap. These are HMAC-signed fields of the token — editing them requires the tenant master key — and they are enforced on every /v1/ai call before the upstream is dialled. Over a token cap the gateway answers HTTP 429 with:
Four related refusals exist so a cap can never be silently skipped rather than enforced:
There is no API today that mints a token with these fields set. Both mint endpoints — POST /v1/ai-gateway/agents/{agentId}/tokens and the admin twin — deliberately stamp an empty scope, because neither offers a parameter to set one and inventing a restriction the operator never asked for is worse than no restriction at all. The one path that produces a scoped token today is an OIDC binding, whose scope you author on the binding — and which now refuses a scope that narrows nothing. An explicit scope parameter on mint is planned and not shipped.

Attribution

Every call is attributed so you can build showback/chargeback:
  • User — pass X-KC-User on the request, set to the person’s key in your workspace directory:
    • for someone your identity provider provisions over SCIM, their SCIM externalId — or their userName, if your IdP sends no externalId;
    • for any other member, their KnoxCall sign-in email.
    The value is matched exactly as the directory holds it. An id with no directory entry does not block the call; the response carries X-Knox-AI-Attribution: user_unregistered and the usage row is left unattributed. The same happens once a person’s membership ends — deprovisioned by your IdP or removed from the team page — so spend is never billed to someone who has left. Their earlier usage keeps its attribution.
  • Team — the directory team for that user, unless the request names another with X-KC-Team.
  • Model and agent — recorded automatically.

The X-KC-Team override

X-KC-Team bills one call to a cost centre other than the user’s default team — a contractor working across projects, a batch job charged to whoever asked for it. It is a client header, so the gateway resolves it before it can reach a ledger:
  • it must be a UUID, and
  • it must name a group your workspace has declared.
A header that fails either check does not fail the request. The directory team stands, the response carries X-Knox-AI-Attribution: team_malformed or team_unknown, and the claimed value is logged. A request that sends X-KC-Team with no X-KC-User is attributed to the team alone — a service account or CI job has a cost centre and no employee behind it.
How people get into the directory. SCIM provisioning (Pro and Enterprise) adds each person the moment your IdP creates them — the directory entry and the workspace membership are written together, and a later rename or externalId change in the IdP moves the same entry rather than creating a second one. A member who is not provisioned over SCIM gets an entry the first time they connect their own account to an MCP server or approve a device, keyed on their KnoxCall email. If they later change that email, the same entry moves to the new address and the old address stops identifying them — at once in every workspace they belong to, and in a workspace they have left, at the moment they rejoin it. If the new address is already another person’s directory key in your workspace, their entry stops answering to either address — the old one comes back user_unregistered and the new one keeps naming the person who holds it — until that key changes and they next connect an MCP account or approve a device. Signing in with SAML SSO does not by itself create one. There is not yet a way to add directory entries for people who are not workspace members (your own application’s end users); until there is, attribute those calls by agent-per-team and group spend by agent. X-KC-Team resolves only against groups your workspace has declared, and there is not yet any way to declare one, so today it comes back team_unknown for every value.
The Usage tab on the gateway detail page reads the raw usage ledger (which carries the team attribution and cost_source that the daily rollup drops) and surfaces unpriced requests so a pricing gap is visible rather than hidden.

Showback / chargeback exports

Export aggregated spend for any period, grouped by the dimension your finance team bills on — user, team, agent, model, provider, or a custom tag (cost_center / project / …). Tag agents via their tags field, then group by tag:<key>.
Each row carries requests, input_tokens, output_tokens, cost_usd, and unpriced_requests (so a pricing gap is visible, never hidden). group_by is a strict whitelist and a tag key is bound as a parameter — no injection surface.

Provable budget enforcement

“Budgets” that only warn are theatre. KnoxCall’s enforcement is covered by a test suite that exercises each overage action against the budget store:
  • block — once the daily/monthly cap is reached, further calls are rejected before reaching the upstream (no spend past the cap).
  • warn — the call proceeds with X-Knox-AI-Budget-Warning; nothing is silently dropped.
  • per-call token ceiling — a request asking for more output than the agent’s budget_per_call_max_tokens is rejected up front, so one call can’t blow the budget in a single shot.
Enforcement reads a cross-worker store (Redis when enabled) so the cap holds across every worker, not per-process. Cache hits (response caching) cost 0, are recorded with cost_source = 'cache_hit', and never charge the budget.

Spend-spike detection

Budgets catch a hard ceiling. A spend spike is the softer failure — a 5x jump well under the cap, arriving on the invoice a month later. A daily job sweeps each agent’s rolled-up daily spend and raises an alertable ai_gateway.spend_spike audit event when the most recent day clears all four of: The baseline is a median + MAD (median absolute deviation), not a mean and standard deviation. The daily rollup gives at most ~30 points per agent, and a single earlier spike would poison a mean-based baseline into never firing again. The event carries the agent id, the day’s spend, the baseline and the multiple. Watch it with an alert rule on the ai_gateway_event type:
POST that to /admin/alerts with your tenant header. The same rule shape works for any AI-gateway audit action — ai_gateway.unpriced_model, ai_gateway.geo_velocity_anomaly, ai_gateway.canary_leak.
ai_gateway_event is not yet offered in the admin UI’s alert-type picker, so this rule has to be created through the API. The job that raises the event runs once a day on a single scheduler process — it needs at least a week of usage before it can say anything, and it reads the hourly usage rollup rather than raw requests.

Where to look

  • Usage tab — spend and request counts by period, with attribution + export.
  • Models tab — the effective pricebook and each model’s allowlist status.
  • OpenTelemetry — knoxcall.ai_gateway.cost_usd on every span for your own dashboards.