Prompt Caching

If you send the same long prefix on every request — a system prompt, a tool list, a document you keep asking questions about — the provider can keep it warm and skip re-processing it. Cached input tokens are billed at roughly a tenth of the normal input rate.

The catch is the write. Putting something into the cache costs 1.25× the normal input rate, so a cached prefix that is never read again made that request 25% more expensive. Caching pays when the prefix is reused, and loses money when it is not.

Do you need to do anything?

Depends on the model.

The models that need marking are the ones where the provider will not guess for you. If you send nothing, you simply get no caching — not an error.

Automatic caching: read the result

Nothing to send. Check whether you got a hit:

from openai import OpenAI
 
client = OpenAI(api_key="sk-...", base_url="https://backend.sovereigneg.com/v1")
 
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[
        {"role": "system", "content": LONG_SYSTEM_PROMPT},
        {"role": "user", "content": "What does clause 7 say?"},
    ],
)
 
usage = response.usage
print(usage.prompt_tokens)                        # total input, cached included
print(usage.prompt_tokens_details.cached_tokens)  # how many were served warm

prompt_tokens is inclusive — it already contains cached_tokens. Do not add them together.

The first request of a conversation reports cached_tokens: 0. Send the same prefix again within the cache lifetime and the count jumps.

Claude: mark the prefix with cache_control

Put cache_control on the last content part you want cached. Everything before it is cached; everything after stays fresh. Order matters — put your stable content first.

response = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": LONG_CONTRACT_TEXT,
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        },
        {"role": "user", "content": "What does clause 7 say?"},
    ],
)

Note the content is a list of parts, not a plain string — a marker has to attach to something, and a bare string has nowhere to put it.

Markers work on text and image_url parts.

Rules

• At most 4 markers per request. A fifth is a 400.
• type must be ephemeral. Anything else is a 400.
• Send ttl: "5m" or omit ttl entirely. Do not send 1h — see The 1-hour cache below.
• Below the model's minimum, the marker does nothing. No error, no cache.

Depending on which provider serves the model, a rejection may come from us — naming the exact field, e.g. messages[].content[].cache_control.type — or be relayed from the upstream provider in its own wording. The rule is the same either way; only the error text differs.

Models that support cache control

These are the models where markers do something. The minimum prefix column is the floor below which the provider ignores your marker silently — no error, no cache, so the marker is simply wasted.

Claude — mark with cache_control:

OpenAI — mark with prompt_cache_breakpoint plus prompt_cache_options:

Any Claude model not listed above is treated as needing 4,096, and any unlisted GPT model as 2,048 — the worst case for that provider, so a marker is never sent where it would do nothing.

The floors are not ordered by generation. Haiku 4.5 needs 4,096 while the older Haiku 3.5 needs 2,048; Opus 5 needs 512 while Opus 4.5 needs 4,096. Any rule you derive from two models will be wrong for a third — read the table.

Every other model on the platform either caches automatically (send nothing) or does not cache at all. In both cases a marker is unnecessary; the only way to tell them apart is to send the same prefix twice and look at cached_tokens.

GPT-5.6: explicit mode

GPT-5.6 and newer support explicit control, which requires both a marker and the option that switches explicit mode on:

response = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": LONG_SYSTEM_PROMPT,
                    "prompt_cache_breakpoint": {"mode": "explicit"},
                }
            ],
        },
        {"role": "user", "content": "Summarise section 3."},
    ],
    extra_body={"prompt_cache_options": {"mode": "explicit", "ttl": "30m"}},
)

prompt_cache_breakpoint is an object, not a boolean. Sending prompt_cache_breakpoint: true is rejected with a 400 invalid_request_error; {"mode": "explicit"} is the required shape, and explicit is the only valid mode.

Sending mode: explicit without a breakpoint gives you no caching at all — it switches off the implicit behaviour you would otherwise have got for free. That is worse than sending nothing. Send both or neither.

Breakpoints are accepted on text, image_url, input_audio, file and refusal parts. Marking anything else returns a 400.

Unlike Claude's 5-minute window, GPT-5.6 cached prefixes have a minimum 30-minute TTL — that is the value to send in prompt_cache_options.ttl.

prompt_cache_key — routing affinity

Optional, and useful on every model including ones with no explicit caching:

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[...],
    extra_body={"prompt_cache_key": "support-bot-v3"},
)

It routes requests sharing a prefix toward the same machine, which raises your hit rate. Requests with different keys can land on different machines and each pay their own write.

Use one key per stable prefix — per assistant, per document, per tenant. Do not put a per-request value in it; a unique key on every call defeats the point.

Your key is namespaced to your organisation before it leaves us, so it can never collide with another customer's cache. The value you send is an input to that namespace, not what travels upstream. Two organisations sending the identical key get unrelated caches.

What it costs

Per million input tokens, relative to that model's normal input rate:

So a prefix read once after being written roughly breaks even. Read twice or more and you are ahead. Written and never read, you paid 25% extra for nothing.

A cache hit refreshes the clock for free, so a cache renews itself indefinitely for as long as a conversation stays active. The window is idleness, not total elapsed time — 5 minutes on Claude, 30 minutes on GPT-5.6.

Your exact per-component rates are on the model's page in the console — cache read and write are priced per model, not as a fixed ratio.

The 1-hour cache

Claude offers a 1-hour lifetime. It is not supported here — do not send cache_control.ttl: "1h".

A 1-hour write costs 2× the input rate instead of 1.25×, which needs two subsequent reads to break even rather than one. Since a 5-minute cache already renews itself on every hit, the longer tier only helps when your gaps genuinely exceed five minutes.

Be aware that "not supported" is enforced on some routes and not others: on models we serve directly from Anthropic, ttl: "1h" is rejected with a 400 naming messages[].content[].cache_control.ttl. On models reached through an aggregator, the same value may be accepted upstream — in which case you are charged the 2× write premium with none of the break-even maths done for you. Since which provider serves a given model can change, treat 1h as unavailable regardless of what a particular request appears to accept.

If your workload has long idle gaps and you want the 1-hour tier enabled properly, contact support rather than sending it speculatively.

Verifying it works

Send the same request twice and compare:

for i in range(2):
    r = client.chat.completions.create(model=MODEL, messages=MESSAGES)
    d = r.usage.prompt_tokens_details
    print(f"call {i}: prompt={r.usage.prompt_tokens} cached={d.cached_tokens}")
 
# call 0: prompt=8421 cached=0
# call 1: prompt=8421 cached=8192

If the second call still reports cached_tokens: 0, work through these in order:

1. Is the prefix long enough? Below the model's minimum, the marker is ignored silently. This is the most common cause.
2. Is the prefix byte-identical? Caching is prefix matching. A timestamp, a session id, or a reordered JSON key at the start of your prompt invalidates everything after it. Move volatile content to the end.
3. On Claude, did you send cache_control? There is no implicit caching.
4. On GPT-5.6, did you send both the breakpoint and prompt_cache_options? And is the breakpoint the object {"mode": "explicit"} rather than true? A boolean is a 400, not a silent miss.
5. Did the window lapse since the last hit — 5 minutes on Claude, 30 on GPT-5.6?

Errors

These are the errors we raise ourselves, with the offending field named in error.param. The same mistakes can instead surface as an upstream 400 in the provider's own wording, depending on which provider serves the model — so match on the status and your own request shape, not on the message text.

A request that is simply too short to cache is not an error — it returns 200 with cached_tokens: 0.