Skip to main content

​
Zero configuration

There is nothing to turn on and nothing to mark up. Send requests the way you already do — no special headers. When a request repeats context the API has seen before — a stable system prompt, documents pasted into the conversation, a growing message history — the repeated part is recognized and served from cache. There are also no cache lifetimes to manage and nothing to invalidate. If the repeated context is available in cache you get the discount; if it isn’t, the request simply counts as normal input. Correctness never depends on it. If you already send Anthropic-style cache_control markers, they are honored and passed through untouched — and when you place your own markers, the gateway uses your breakpoints instead of its own. On the OpenAI shape, a stable prompt_cache_key (or user) value keeps a conversation’s turns on the same cache.

​
How it bills

Every request takes your credits for the model that served it. Each kind of token counts at that model’s own rate: (PAYG mode is the one exception: there, cache writes count at the model’s cache-write rate.) Where a model doesn’t cache input, repeated context is still recognized and still returns faster, but it counts as ordinary input: we don’t discount what the model doesn’t. Usage reporting is protocol-standard on both dialects either way. On the OpenAI shape, prompt_tokens includes the cached subset, broken out under prompt_tokens_details.cached_tokens:
OpenAI dialect — usage on a cached turn
On the Anthropic shape, input_tokens counts fresh input only and cached input is reported as cache_read_input_tokens:
Anthropic dialect — usage on a cached turn

​
Getting the most from it

Caching keys off repetition, so the discount follows how you shape your prompts:
  • Stable content first. Put the system prompt, documents, and examples at the top; the part that changes each turn — the user’s latest message — last.
  • Keep the prefix byte-stable. A timestamp, a random id, or a reordered field near the top of the prompt makes every turn look new. Append to the conversation; don’t rewrite what came before.
  • Long sessions benefit most. Agent sessions and document Q&A resend a large stable prefix on every turn — exactly the shape the cache rewards.
  • Keep a conversation together. A cache is warm only where the previous turn ran. Send a stable prompt_cache_key (OpenAI shape) or place cache_control markers on the same blocks each turn (Anthropic shape) so your turns land together.
  • Churn benefits least. Tool-call-heavy loops whose context changes substantially between turns repeat less, so less of their input is served from cache.
The workloads that gain the most are text-heavy: long-document analysis, retrieval-augmented chat, and coding sessions over a large stable context.

​
Where it shows up

Two places. First, the usage block of every response, in the dialect-standard fields above — your existing token accounting keeps working unchanged. Second, your credits: on a model that caches input, a request whose input is mostly cached uses far fewer credits than its raw token count suggests. The dashboard’s usage page shows what each request used.
Two things cache less than you might expect. Heavy tool-loop turns with fast-changing context have little that repeats — the discount follows repetition and can’t manufacture it. And on a model that doesn’t cache input there is no discount to apply. Budget every workload as if nothing were cached and treat cache savings as upside.