How to handle prompt caching with multimodal?

I was building a real-time assistant that fires a request to an LLM every few seconds — transcript turns going in as text, screenshots of the screen going in as images — and I wanted each request to see the full session so far rather than a sliding window of the last minute. Text is cheap, so carrying the whole transcript is easy to justify. The moment images enter the picture, though, the obvious design quietly falls apart, and the reason is entirely about prompt caching. This post is the chain of reasoning I wish someone had handed me before I started reconstructing multimodal history: what the cache actually rewards, how to shape the request so it keeps hitting, and the handful of mistakes that silently cost you every hit.

Caching is a prefix match, on both providers

The one fact everything else follows from is that prompt caching is a position-anchored longest-prefix match. The cache is keyed on the exact bytes of the rendered prompt, matched from the first token forward, and it stops at the first divergence. It cannot pick up a matching chunk from the middle. If turn two’s prompt differs from turn one’s by a single character anywhere in the prefix, the match fails at that character and everything after it is uncached, even if the rest is byte-identical.

The two providers expose this differently. With Anthropic you place the breakpoints yourself — you mark a content block with cache_control and the prefix up to that block becomes cacheable. With OpenAI the prefix caching is automatic; you don’t annotate anything, the server just recognizes a prefix it has seen before. The mechanics underneath are the same prefix match, which means the design problem is the same on both. The render order on Anthropic is tools, then system, then messages, so the stable content has to come first and the volatile content last — but that ordering discipline pays off identically on OpenAI even though you never write a single breakpoint.

Reconstruct the history as real turns

The first instinct is to flatten everything into one giant string — the whole transcript, with past hints inlined as (suggested: ...) markers — and send it as a single user message. It works, and it caches, but the model sees its own past advice narrated as text rather than as something it said, which weakens its sense of what it has already told you. The better shape is to rebuild an actual alternating conversation, where each past hint is a real assistant turn:

const messages = [
  { role: "user",      content: [{ type: "text", text: renderTurns(beforeHint1) }] },
  { role: "assistant", content: [{ type: "text", text: hint1 }] },
  { role: "user",      content: [{ type: "text", text: renderTurns(between1and2) }] },
  { role: "assistant", content: [{ type: "text", text: hint2 }] },
  // ... and finally the current turn, which is the one thing that grows
];

This shape is the thing that makes caching work, because every request shares the same prefix right up to the final user turn. Each turn the conversation extends by one user message and one assistant reply, and the cache extends with it. If you imagine the entries the server writes, they nest:

entry written on turn 1:  system + user1
entry written on turn 2:  system + user1 + hint1 + user2
entry written on turn 3:  system + user1 + hint1 + user2 + hint2 + user3

Turn three’s prompt is turn two’s prompt plus more, so it matches the entry turn two wrote and reads it back, then writes a slightly longer entry of its own. As long as you reconstruct that array deterministically — same system prompt, same turn ordering, the hint text pulled verbatim from storage — the prefix stays byte-identical and per-request cost stays flat even as the session runs for an hour. The thing that does not belong anywhere in that nested prefix is an image, and the rest of this post is about why.

The tempting design, and why it bites

If caching is a prefix match, the seductive move is to make the prefix stable by replaying everything, images included. Persist each turn — transcript text and the screenshot from that moment — and rebuild the exact same message array every request:

{
  role: "user",
  content: [
    { type: "text",  text: renderTurns(turn) },
    { type: "image", source: { type: "base64", media_type: "image/jpeg", data: frameForThisTurn } },
  ],
}

Because the bytes are identical turn over turn, the prefix keeps matching and the whole history — pictures and all — reads from cache at a fraction of the input price. Mechanically this is correct. Images are cacheable; you can put a breakpoint after an image block; entry three extends entry two extends entry one, and you get cache reads that include the images.

The thing this reasoning leaves out is the one variable that decides everything for a real-time app: prompt caching discounts processing, not transmission. The cache lives server-side and is keyed by content hash. There is no cache pointer you send instead of the bytes — you upload the full messages array on every single request, and the server recognizes the prefix and bills it cheaply. So persisting images and replaying them in place means re-uploading every historical image, as base64, on every request.

For a meeting assistant that is fatal. A downscaled screenshot is on the order of a hundred kilobytes to a megabyte of base64. An hour-long meeting with a frame every twenty-five seconds is around a hundred and fifty images. Replay them all and the end-of-meeting request body is tens of megabytes, pushed over the wire every twenty-five seconds. The token cost is cached and cheap, but the upload latency is paid in full every time — and on an assistant whose entire value is being fast, latency is the whole game. You also drag stale visual context forever: a slide from thirty minutes ago almost never informs the hint about what’s on screen right now, yet you pay to carry it, in upload bytes and in cached tokens and in the context window it occupies. The asymmetric premise — text is cheap, images are not — exists precisely to avoid this, and full replay maximizes exactly the cost you wanted to minimize.

Keep history text-only, attach images after the breakpoint

The design that actually works splits storage from what you send. Persist whatever you want for the history UI or for replay, but send the model only the recent frames — the last minute or two — and keep the conversation history text-only. The text transcript is the cheap, high-value context, and it stays fully cached as the session grows. The images become a small, bounded tail that rides on the final turn and is re-sent at full price each request, which is fine because there are only ever two or three of them.

The subtlety that took me a second to see is where the cache breakpoint goes. The textbook multi-turn pattern says to put the breakpoint on the last block of the most recent turn. If you follow that literally with images on that turn, the breakpoint lands after the image:

// Wrong: the breakpoint sits after the image.
{
  role: "user",
  content: [
    { type: "text",  text: tail },
    { type: "image", source: { type: "base64", media_type: "image/jpeg", data: frame },
      cache_control: { type: "ephemeral" } },
  ],
}

On the next request that turn becomes history, and if you strip its image — because frames are transient and you don’t want to carry them forward — you’ve changed bytes that were inside the cached region, and the cache misses from that point on. That is the failure people mean when they say attaching images breaks cache hits. It is real, but it comes entirely from this one wrong placement.

The fix is that the breakpoint never goes after an image. Order each turn’s content text first, put the breakpoint on the last text block, and let the image blocks sit strictly after it. Concretely, that means two breakpoints: one on the last block of the completed history, and one on the current turn’s text tail, with the images trailing past both:

const messages = [
  { role: "user",      content: [{ type: "text", text: renderTurns(beforeHint1) }] },
  { role: "assistant", content: [{ type: "text", text: hint1 }] },
  { role: "user",      content: [{ type: "text", text: renderTurns(between1and2) }] },
  {
    role: "assistant",
    // breakpoint A — end of the stable history; slides forward each turn
    content: [{ type: "text", text: hint2, cache_control: { type: "ephemeral" } }],
  },
  {
    role: "user",
    content: [
      // breakpoint B — current transcript tail, before the images
      { type: "text",  text: renderTurns(sinceHint2), cache_control: { type: "ephemeral" } },
      { type: "image", source: { type: "base64", media_type: "image/jpeg", data: frame1 } },
      { type: "image", source: { type: "base64", media_type: "image/jpeg", data: frame2 } },
    ],
  },
];

Now walk it forward. When the current turn recedes into history and you drop its images, you’ve only changed bytes that were already past every breakpoint, so they were never in any cache entry to begin with. There is nothing to invalidate. Breakpoint A sits before the whole current turn and is untouched; breakpoint B sat before the images, so stripping them changes nothing the cache had stored. The text you cached reads as a hit, and the new turn’s text caches fresh after it. This is just ordinary incremental multi-turn caching, and images on the final turn don’t break it at all.

What you give up is caching the images themselves — each request re-sends the recent frames at full token price. But that isn’t a loss you could have recovered, because recent frames change every request; there is nothing stable about them to cache. Anything you wrote to cache you’d never read back. Leaving them uncached is correct, not a regression. The one genuine exception is a screen that stays static across several requests, but if you already suppress duplicate work when nothing on screen has changed, you aren’t re-sending that image anyway.

It comes out the same on OpenAI

When I first worked this out on Anthropic I worried the automatic caching on OpenAI would behave differently — that omitting images from history there would cause a miss, since OpenAI might have auto-cached the previous request’s images and would now see them gone. It doesn’t, and the reason is the same property that makes the Anthropic design clean: in this layout the images were never in the prefix to begin with. History is text-only by construction; images only ever sit on the final user turn, which is the changing tail. So there is nothing being “omitted from history” — history never had images.

Walk the prefix match between consecutive requests and you can see exactly what gets reused. The system prompt matches, all the older text turns match, and the match diverges at the single most-recent turn — the one transitioning from its final form (text plus images) into history (text only). That one turn gets reprocessed once. Everything before it still hits. And that is identical to the Anthropic case, where the breakpoint stops the read right before that same transitioning turn. The image handling is symmetric across both providers: images live in the uncached tail, full-price per request, and the deterministic text construction earns the prefix hit for free.

Watch the prefix for silent invalidators

All of this assumes the prefix is byte-stable, and the easiest way to lose every hit is to let something volatile leak into it without noticing. The cache fails silently — no error, just cache_read_input_tokens quietly sitting at zero — so you have to go looking. The one that got me was the system prompt. I had a note that I only wanted to include when the request carried screenshots:

// Silent invalidator: the system prompt changes shape per request.
const system = ctx.screenshots?.length
  ? BASE_PROMPT + SCREEN_NOTE
  : BASE_PROMPT;

Screenshots come and go from request to request, so this flips the system prompt between two different byte strings — and the system prompt sits at the very front of the prefix, so every flip rebuilds the entire cache behind it. The fix is to gate that note on something session-stable, like the meeting type or a per-session “screen is shared” flag, so the system prompt is frozen for the life of the session. The same discipline rules out the usual suspects: no Date.now() or uuid() interpolated into the system prompt, deterministic JSON serialization so key order never shuffles, and identical image encoding every time you replay a frame. Anything that changes per request belongs after the last breakpoint or nowhere at all.

It helps to know that not every change is equally destructive. The cache has tiers — tools, then system, then messages — and a change only invalidates its own tier and everything after it. Adding or removing an image, or editing message content, invalidates only the messages tier; the tools and system caches survive. Since our images live past the breakpoint anyway this is moot in practice, but it’s reassuring confirmation that image churn on the final turn can never reach back and disturb the system prefix.

To actually verify any of this is working, read the usage back off the response:

const res = await client.messages.create({ /* ... */ });
console.log(res.usage.cache_read_input_tokens);     // served from cache (~0.1x)
console.log(res.usage.cache_creation_input_tokens); // written this request (~1.25x)
console.log(res.usage.input_tokens);                // full price, uncached remainder

If cache_read_input_tokens is zero across two consecutive requests in one session, something volatile has snuck into the prefix — diff the rendered bytes of two requests and you’ll find it.

A few operational details that bit me

There is a minimum cacheable prefix, and early in a session you’re below it. On Opus 4.8 the floor is 4096 tokens; on Sonnet 4.6 it’s 2048. For the first minute or two the transcript prefix is just short, so it silently won’t cache — cache_creation_input_tokens reads zero and that’s fine, it starts caching once the history is long enough to clear the bar. Don’t go hunting for a bug that’s just a short prompt.

Caching has a time-to-live, and that interacts nicely with a chatty real-time app. The default entry lives five minutes; a write costs about 1.25× a normal input token and a read costs about a tenth. Because hints fire every twenty-five seconds during an active meeting, each request keeps the prefix warm for the next one essentially for free. The one place it bites is a long silence — if more than five minutes pass with no request, the next one pays a cold write to rebuild. If gaps like that are common you can bump the TTL to an hour, which keeps entries alive across the lulls at the cost of a 2× write, but I’d start with the five-minute default and only reach for the hour if the logs show cold writes hurting.

Two smaller constraints. You get at most four breakpoints per request, which is exactly why the design uses them as a pair of sliding markers — one at the end of stable history, one before the images — rather than sprinkling them everywhere. And a breakpoint only walks back about twenty content blocks looking for a prior cache entry; if a single turn contributes many blocks (a transcript chunk plus several images), drop an intermediate breakpoint so the next turn can still find the cache it’s extending.

Storage is a separate question

The last thing worth untangling is that none of this dictates how you store images. It’s tempting to conflate “I want frames in the history” with “I’ll base64 them into a Postgres column so the prefix replays identically.” But storage and the request are orthogonal. What you send the model is only the recent frames, and your in-memory frame buffer already has those — you don’t even need the database on that path. What you persist, if you want frames for a history UI or for debugging, should go to object storage keyed by URL, not base64 in a row: base64 is about a third larger than the raw bytes and it bloats both the row and the write-ahead log for data the model never reads back. Persist all the frames if the product wants them; send only the recent ones to the model; keep the two decisions independent.

So the whole multimodal-caching question collapses to a placement rule. Reconstruct the history as deterministic, text-only turns; put the cache breakpoint on the last text block; keep the volatile content — the newest transcript tail and the recent images — strictly after it. Get that ordering right and the text history bills at cache-read rates for the life of the session, the payload stays flat no matter how long the meeting runs, and the only thing you pay full price for is the handful of frames that actually describe the present moment. Which is exactly the thing you wanted the model to see.