AI Gateway Quickstart

You adopt the AI Gateway by pointing your existing AI SDK at your agent’s gateway URL and using a phantom token as the API key. No new SDK, no code rewrite.

1. Create an agent

In the dashboard, go to AI Gateway → New agent. Pick your provider — Anthropic, OpenAI, Google Gemini, Cohere, Azure OpenAI or a self-hosted Ollama-compatible endpoint — paste that provider’s API key, and press Create agent. KnoxCall does the rest in one step: it encrypts the key into a normal KnoxCall secret, creates the upstream route that injects it, creates the gateway if you do not have one yet, creates the agent, and mints your first capability key. Azure OpenAI and self-hosted endpoints also ask for the base URL of your own resource, because that address is specific to you. When the agent is created you get two things:

an agent URL — https://<your-slug>.knoxcall.com/v1/ai/<agent-slug> (sandbox: https://sandbox-<your-slug>.knoxcall.com/...)
a phantom token — kp_live_a_… (shown once at mint time)

The final step also shows the environment variables, the curl command and the SDK snippet already filled in with that token, and a Send test request button that runs one prompt through the agent you just created.

The provider key you paste is envelope-encrypted with your tenant key before it is stored, and no endpoint ever returns it. It is injected at the upstream hop, so clients holding the capability key never see it.

Prefer to wire it up piece by piece? New Gateway still creates an empty gateway, and the agent form still lets you point at an existing secret instead of pasting a key.

The data plane is served on your tenant’s proxy subdomain (<slug>.knoxcall.com), not api.knoxcall.com. Always use the agent_url returned at agent creation as your base URL.

2. Swap your base URL

Anthropic SDK

from anthropic import Anthropic

client = Anthropic(
    base_url="https://acme.knoxcall.com/v1/ai/support-bot",  # your agent URL
    api_key="kp_live_a_...",                                  # your phantom token
)

msg = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Summarize this ticket…"}],
)
print(msg.content[0].text)

OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://acme.knoxcall.com/v1/ai/support-bot",
    api_key="kp_live_a_...",
)

resp = client.chat.completions.create(
    model="gpt-5",
    messages=[{"role": "user", "content": "Summarize this ticket…"}],
)
print(resp.choices[0].message.content)

Node / TypeScript

The provider’s own Node SDK works unchanged — same swap, same two lines:

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic({
  baseURL: 'https://acme.knoxcall.com/v1/ai/support-bot', // your agent URL
  apiKey: process.env.KNOXCALL_AI_TOKEN,                  // your kp_ phantom token
});

const msg = await client.messages.create({
  model: 'claude-sonnet-5',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'Summarize this ticket…' }],
});
console.log(msg.content[0].text);

For OpenAI, import OpenAI from 'openai' and pass the same baseURL / apiKey.

curl

curl https://acme.knoxcall.com/v1/ai/support-bot/v1/messages \
  -H "Authorization: Bearer kp_live_a_..." \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

What comes back

The provider’s own response shape, unchanged — the gateway does not wrap it:

{
  "id": "msg_01XyZ…",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-5",
  "content": [{ "type": "text", "text": "The customer is reporting…" }],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 27, "output_tokens": 112 }
}

What the gateway adds is in the headers, not the body:

X-Request-Id: 0d5b2a9e-1f3c-4a7d-8e2b-6c9a1f4d7e35
X-Knox-AI-Budget-Pct: 12.4

X-Request-Id is on every response and is the key into your audit trail for that exact call. The rest appear only when there is something to say — see the header reference below. A refusal replaces the body with an error envelope; nothing is ever silently degraded. That’s it. The call is now authenticated by the phantom token, screened by the prompt firewall, PII-redacted if a policy applies, counted against the agent’s budget, and recorded with per-call cost attribution.

3. Authentication schemes

The phantom token can be sent three ways — use whichever your SDK makes easiest:

Authorization: Bearer kp_live_a_...
x-api-key: kp_live_a_...
x-knox-ai-key: kp_live_a_...

For DPoP-bound tokens, also send a DPoP proof header on every request.

Everything you need is on the agent’s Connect panel

You do not have to keep the values from the create wizard. Open the agent (AI Gateway → your gateway → the agent) and the Connect panel at the top of the page has:

the agent URL, copyable — the base URL your SDK points at;
the environment variables for that agent, already filled in (ANTHROPIC_BASE_URL / OPENAI_BASE_URL and the key variable your provider’s SDK reads), so a working shell is one paste away;
a curl smoke test shaped for that agent’s provider;
the newly issued capability key, the one time it is shown, when you mint from the Capability keys list below it;
a Playground button, which runs a prompt through this agent’s real data plane and shows what the gateway did to it.

The panel derives the provider from the agent’s route, so the snippets are for the provider that agent actually calls — not a guess from the model name.

A capability key is displayed once, at mint time. If you lose it, revoke it and mint another; there is no endpoint that returns an existing token’s value.

4. Streaming

Streaming works unchanged — set stream: true (or Accept: text/event-stream). PII redaction runs inside the stream, so redactions appear as the tokens arrive rather than after the response completes:

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "…"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="")

Try it from the dashboard: the Playground

Every agent has a Playground (agent page → Playground). It sends the prompt you type through that agent’s own /v1/ai/<slug> data plane — the same URL your SDK uses — with a one-shot capability key that expires the moment it is used, and shows what the gateway did to the call beside the reply:

the redaction diff — your prompt with every detected value highlighted, next to the exact text the provider received, with the format-preserving surrogates that replaced them;
the firewall verdict for the prompt;
cache outcome, model, latency and cost for the request.

Those numbers are read back from the rows that request wrote (ai_gateway_usage, ai_gateway_firewall_events) and the PII token map for its conversation — not recomputed for display. Where a row is missing the panel says so rather than showing a reassuring default. The Playground spends the same provider tokens a real call does, and is subject to whatever budget the agent has: a quickstart agent starts with none, so set one on the agent before handing the Playground to a wider team.

From the command line

The KnoxCall CLI carries an ai command group over the same /v1 control plane, so gateways, agents, minting and spend are all reachable without opening the dashboard. It reuses the credential knoxcall login writes — no keys in scripts, and the tenant is resolved from the token, so no tenant header:

knoxcall ai gateways                       # id · slug · name
knoxcall ai agents <gatewayId>             # that gateway's agents
knoxcall ai mint <agentId> --kind agent \
                          --name "CI pipeline"   # shown once
knoxcall ai usage --period 30d             # cost + tokens by model
knoxcall ai usage --period 7d --agent <agentId>

The ai command group ships in the knoxcall-cli package, which is not published to a registry yet. The knoxcall binary that comes with the language SDKs today covers login, logout, whoami and init — not ai. Until knoxcall-cli is published, use the /v1 endpoints directly; the commands above map one-to-one:

Command	Endpoint
`ai gateways`	`GET /v1/ai-gateway/gateways`
`ai agents <gatewayId>`	`GET /v1/ai-gateway/gateways/{gatewayId}/agents`
`ai mint <agentId>`	`POST /v1/ai-gateway/agents/{agentId}/tokens`
`ai usage`	`GET /v1/ai-gateway/usage`

Useful request/response headers

Header	Direction	Meaning
`X-KC-User`	request	the person’s directory key for per-user cost attribution — SCIM `externalId` (else `userName`), or a member’s KnoxCall email (details)
`X-KC-Conversation-Id`	request	conversation id — scopes the reversible PII token map and the response cache across turns
`X-Request-Id`	response	UUID for this call (appears in audit logs)
`X-Knox-AI-Budget-Pct`	response	current budget utilization %
`X-Knox-AI-Tools-Stripped`	response	tools removed by the agent’s tool allowlist
`X-KC-Team`	request	cost-centre override — a team UUID your directory uses; see attribution
`X-Knox-AI-Cache`	response	present on a cache hit (`exact` or `semantic`)
`X-Knox-AI-Attribution`	response	why an attribution claim was not used: `user_unregistered`, `team_malformed`, `team_unknown` (comma-separated when more than one applies)
`X-Knox-AI-Output-Warning`	response	the agent’s output schema was violated and the action is `warn`
`X-Knox-AI-Retry`	response	`output_schema` — the answer you are reading is a schema-corrected retry
`X-Auth-Failure`	response	on a 401, the machine-readable refusal code (e.g. `step_up_required`, `oneshot_already_used`)

Response caching

Caching is off by default and is set per agent with cache_mode — in the dashboard on the Agent’s Budgets & cache tab, or over the /v1 Management API. An agent’s cache_mode is off unless you set it deliberately.

Mode	Behaviour
`off` (default)	No caching.
`exact`	A byte-identical request replays the saved response.
`semantic`	A request that means the same thing as a prior one replays it, matched by embedding cosine similarity above `cache_similarity_threshold` (default 0.95). Falls back to exact matching when it cannot embed.

A cache hit costs $0, is recorded with cost_source = 'cache_hit', does not charge the agent’s budget, and sets X-Knox-AI-Cache: exact or semantic.

What the hit rate actually is

The Agent’s Budgets & cache tab reports, over the last 30 days: hit rate, exact hits, near (semantic) hits, misses, bypassed requests, stored entries, and — for an agent in semantic mode — how many requests approximate matching actually ran for, its own hit rate, and how many degraded, with the reason. Two things about that report are worth knowing before you read a number off it. Bypassed is not a miss. A request that streamed, that hit a response-direction guardrail hook, that ran on an agent with a customer-managed key, or that the gateway could not attribute to a token, was never eligible for the cache. Those are counted beside the rate and never inside it. An agent whose traffic is all stream: true will show a large bypass count and no hit rate, which is the truthful answer rather than 0%. Before 2026-09-08 this number was wrong — for an agent in exact mode it read 100% from its first hit onward, because misses were never recorded. If you looked at it before that date, look again.

The number we have, and what it is worth. On our reference fixture (support-assistant-v1: 7 conversations, 30 requests, a mix of verbatim repeats and paraphrases) at the default 0.95 threshold, 13.3% of requests were served from cache — three exact, one approximate — with no false matches and no cross-caller hits. At 0.90 it is 30%; the exact-cache floor, which needs no embedding model at all, is 10%.That is a fixture measurement produced with a substitute embedder, because our test suite may never call a provider. It is not a forecast of your traffic, and we will not dress it up as one: how much a response cache saves depends almost entirely on how repetitive your prompts are. The number that matters is the one on your own Agent’s cache panel.

Isolation — what a hit can and cannot cross

Every cache entry is keyed on the phantom token that produced it and the conversation id, in addition to the agent and the request body. That means:

a cached response never crosses tenants;
it never crosses tokens, even inside one tenant — two users of the same agent do not serve each other’s answers, whatever conversation ids they send;
it never crosses conversations for one token;
a request the gateway cannot attribute to a token neither reads nor writes the cache at all.

The token is the boundary that carries the weight, because it is the identity KnoxCall verifies. X-KC-Conversation-Id is a client header — still worth making unguessable and unique per end-user conversation, since it partitions a single token’s own cache, but it is no longer the only thing standing between two callers.

Semantic matching — the requirements

Semantic matching runs only when all four of these hold (cache/index.ts, semanticEnabled). Each one is a boundary, not a limitation to work around:

cache_mode is semantic and cache_embedding_model is set.
The request carries an X-KC-Conversation-Id. Approximate matching needs an isolation boundary to be confined to; a stateless request has none, so it is exact-only.
The agent is not using reversible PII tokenization — that is, pii_request_mode is off and pii_response_mode is redact. Either half on its own is enough to keep semantic matching off. Conversation scoping already makes it safe, but approximate matching against a per-conversation token map is held apart until a dedicated review clears it. Tokenizing agents still get exact caching.
The upstream exposes an OpenAI-compatible /v1/embeddings on the same origin as the chat endpoint. Anthropic has no embeddings API, so an Anthropic-backed agent is exact-only.

When embedding is unavailable the request degrades cleanly to an exact-cache miss — never an error on the data plane.

Caching applies to buffered responses only. A streaming request is never served from cache and never stored, so an agent whose traffic is all stream: true will see no hits.

Next steps

Bring your own key

Why AI features use your own Anthropic key, and how to add it.

Tokens & DPoP

Mint scoped, sender-constrained tokens for agents and CI.

Model & output policy

Pin which models an agent may call, and the shape of what comes back.

Limits & availability

The ceilings the data plane enforces, and what we do and don’t promise.

​AI Gateway Quickstart

​1. Create an agent

​2. Swap your base URL

​Anthropic SDK

​OpenAI SDK

​Node / TypeScript

​curl

​What comes back

​3. Authentication schemes

​Everything you need is on the agent’s Connect panel

​4. Streaming

​Try it from the dashboard: the Playground

​From the command line

​Useful request/response headers

​Response caching

​What the hit rate actually is

​Isolation — what a hit can and cannot cross

​Semantic matching — the requirements

​Next steps

Bring your own key

Tokens & DPoP

Model & output policy

Limits & availability

AI Gateway Quickstart

1. Create an agent

2. Swap your base URL

Anthropic SDK

OpenAI SDK

Node / TypeScript

curl

What comes back

3. Authentication schemes

Everything you need is on the agent’s Connect panel

4. Streaming

Try it from the dashboard: the Playground

From the command line

Useful request/response headers

Response caching

What the hit rate actually is

Isolation — what a hit can and cannot cross

Semantic matching — the requirements

Next steps