Skip to main content

​
Streaming Guarantees

Most gateways can only inspect a streamed AI response after it finishes, or log what they saw. KnoxCall redacts inside the live SSE stream. This page states the per-mode guarantee and publishes the latency cost — a number competitors who buffer or don’t enforce mid-stream can’t put on the table.

​
What in a stream is scanned

Redaction applies to the model’s text output, in every content block a response carries — not just the first: Each block gets its own hold-back window, so text from one completion is never spliced into another’s.
Not scanned: streamed tool-call arguments — delta.tool_calls[].function.arguments (OpenAI) and input_json_delta (Anthropic). They are JSON fragments arriving a few characters at a time, and splicing a replacement mid-fragment produces a document your client cannot parse, so redacting them needs a different mechanism than the hold-back window. If your agent passes sensitive values into tool arguments, use the MCP gateway — which does redact tool arguments recursively — or turn streaming off for that agent so the whole response is scanned as one document.
Extended thinking and signature. Redacting reasoning text means the signature Anthropic attaches to a thinking block no longer matches its contents, so replaying that block verbatim on a later turn will be rejected. This is not new behaviour on the streaming path — the buffered path has always scanned the whole response body — and the alternative is streaming your users’ data in the clear. Re-prompt rather than replaying thinking blocks through the gateway.

​
The three modes

Each agent picks a pii_streaming_mode (default holdback). All three record detections; only holdback and buffer alter the bytes the client sees.

​
Why holdback is the hard one (and the differentiator)

A sensitive token can arrive split across SSE frames — "...SSN 123-" in one delta, "45-6789..." in the next. A naïve mid-stream redactor emits the first half before it can tell it’s PII. KnoxCall’s hold-back FSM keeps the last N characters buffered until enough context has arrived to decide, then splices any redaction back into the SSE frame and re-serialises it — so the client never sees a partial entity, and framing is never corrupted. Bedrock punts (stream:false required); Cloudflare buffers; Portkey’s streaming guardrails are log-only. KnoxCall enforces.

​
Published benchmark

scripts/bench-ai-gateway-streaming.ts drives a representative Anthropic response (120 text deltas seeded with email/SSN/credit-card entities) through each mode and a raw passthrough baseline, and records p50/p95 TTFB (time-to-first-emitted-byte) and total drain time. The snapshot lives at scripts/bench-ai-gateway-streaming.results.json. This is a CPU-bound micro-benchmark of the redaction Transform itself (no network) — it isolates the overhead the gateway adds on top of the hundreds of milliseconds an LLM already takes. Representative numbers from one run on a developer machine: The headline: mid-stream redaction (holdback) reaches first byte in a small fraction of the time full-buffering does, and the absolute cost is far below the hundreds of milliseconds an LLM already takes. You get real, enforced, mid-stream PII protection essentially for free on the latency budget.

​
What CI actually enforces

Absolute milliseconds depend on the machine, so we do not assert them — a guard that fails because a runner was busy is a guard that gets deleted. What is enforced, on every commit, by tests/ai-gateway-streaming-latency-invariants.test.ts under plain npm test, driving the same harness this benchmark uses:
  • holdback emits to the client before the upstream stream has ended — the “redacts inside the live stream” claim reduced to an ordering of two events, with buffer as the negative control. No timing involved, so it cannot flake and cannot be satisfied by a fast full-buffer.
  • holdback’s TTFB is a small fraction of buffer-mode’s, and its overhead over raw passthrough stays under a generous ceiling.
  • monitor is indistinguishable from passthrough on first-byte latency and is byte-identical on the wire.
  • holdback and buffer actually redact the planted email, SSN and the credit-card number that is deliberately split across four SSE frames — checked on the reassembled text the end user reads, not on the wire bytes. Without this, a redactor that did nothing would post the best latency numbers in the file.
  • the committed snapshot satisfies the same relative invariants, so --write cannot commit numbers that disagree with the code.
Re-generate the snapshot with npx tsx scripts/bench-ai-gateway-streaming.ts --write. Absolute numbers will differ on your hardware; the orderings above will not.

​
The hold-back window

pii_streaming_holdback_chars (default 96) is the sliding tail. It must be at least as long as the entity you expect, or that entity can never sit inside the window at once. Values below the default are raised to 96 automatically rather than honoured: a shorter window silently stops detecting anything longer than itself while still reporting redaction as enabled, and the window costs nothing at first byte (see the benchmark above) — so there is no trade to make. Raise it (256 is a reasonable setting for name/address-style entities) rather than lowering it. Entities longer than the window are handled separately. Four detector patterns are open-ended — JWTs and the Anthropic/OpenAI, Stripe and GitHub key shapes — and none of them matches until the whole value has arrived, so the window alone cannot protect them: a real JWT is 200–800 characters. The FSM therefore also recognises a possible prefix of one of these anchored at the buffer’s end and holds from there, releasing when the value completes and is redacted. Holding is bounded (a few KB per stream); past that bound the gateway redacts what it has rather than hold more, so the identifying head of a credential is never emitted in the clear.

​
Configure the mode

Set it on the agent — via the PII tab on the gateway detail page, or over the API: