Routing, Retries & Fail-over
Your application points at one agent URL. What happens behind it — how many providers, which one served this call, what happened when the first one said “slow down” — is the agent’s configuration, not your code.The default
Every agent starts withrouting_policy: {}, and that means:
- one attempt per route, no retries;
- fail over on 5xx to
fallback_route_ids, in order; - everything else — 429, 4xx, 2xx — returned to you unchanged.
Retrying a rate limit
The one you almost certainly want:Retry-After header — and if the route is still rate-limited after
max_attempts, the request moves to the next candidate.
Retry-After is honoured in one direction only: it can extend a wait, never
shorten one. A provider saying “retry after 1s” while the gateway is already
backing off 3.2s does not get hammered at 1s. Both
RFC 9110 forms are
understood — delta-seconds and an HTTP-date.
Jitter is on by default. Each delay is spread across 50–100% of its computed
value, so a fleet of workers that all hit the same rate limit at the same moment
does not retry in lockstep and re-trip it. Set
"jitter": false if you need a
deterministic schedule for a test.Timeouts
"retry_on": ["timeout"] covers a transport failure — connection reset, DNS
failure, the upstream hanging up. It is separate from 5xx because they call
for different appetites: a 502 usually means “this provider is having a bad
minute”, a connection reset often means “that one packet was unlucky”.
Weighted load-balancing
Two keys on the same provider, or two providers entirely, split by weight:- A route you do not name keeps weight 1. Adding a weight for one route does not silently drain the others.
- Weight 0 drains a route completely — no ordinary traffic and no fail-over traffic. That is what “park this provider” means; if you wanted “try it last”, give it weight 1 against a much larger one.
- Your token’s scope still wins. The capability scope on the phantom token is
checked against every candidate before it is dialled, so a weight can never
reach a route the token was not granted, and neither can a fail-over. Same for
Live/Test: a
kc_test_token resolves only sandbox routes at every hop.
Model aliases
Set once on the gateway; every agent under it honours them:"model": "fast" and you change what that means in
one place. Chains work (cheap → fast → a real id); a cycle resolves back to
the model you asked for rather than looping.
Aliases resolve before the agent’s model policy, which is the important part:
an alias target still has to pass that agent’s allowlist and denylist. If the
resolution ran the other way round, {"safe-model": "the-one-you-banned"} would
be a one-line way around your own denylist.
What it recorded
Every usage row now carries what actually happened:
Both are
NULL for a cache hit, because nothing was dispatched. attempts > 1
is the signal worth alerting on: it means a provider is making you pay for the
same answer twice.
The limits, and why they exist
Every field is clamped server-side:
These are not tuning suggestions. A request waiting to retry is holding an API
worker, so an agent configured with
max_attempts: 10000 and
backoff_ms: 600000 would be a self-inflicted outage — and a hostile upstream
answering 429 with Retry-After: 86400 would be someone else’s. The ceilings
are what make “configure your own retries” safe to offer at all.
max_attempts is also forced back to 1 whenever retry_on is empty, so a stored
policy never claims a retry that cannot happen. An unrecognised key inside
routing_policy is rejected with a 400 naming the field rather than stored and
ignored — a retryOn typo that reads as configured and does nothing is worse
than an error.