Avoid Paying Twice for the Same Work

The situation

Many AI systems repeat work because the application layer does not know that two requests are effectively equivalent or that the source data has not changed since the last processing run.

The naive solution

Optimise the prompt while still calling the model every time.

The hidden constraint

The cheapest model call is the one I do not need to make. If a deterministic result or previously computed artifact is still valid, another inference adds cost without adding value.

The decision

Introduce caching and incremental computation where correctness allows it, with explicit invalidation rules rather than blind reuse.

The implementation

Good candidates include embeddings for unchanged documents, deterministic classifications, expensive source analysis, static summaries, and generated artifacts keyed by stable inputs and prompt version. For data pipelines I would process only changed records. For back-office work I would batch when the provider or workload makes batching more efficient. Cache keys should include the parts of the prompt, model, parameters, and source version that materially affect the result.

The tradeoff

Caching introduces staleness and invalidation problems. I would never reuse output merely because it is cheaper; the product needs to know when a result is still valid.

The evidence I would look for

I would track cache hit rate, spend avoided, and invalidation rate. If the cache is complex but rarely hit, it is probably not the right optimisation target.

The reusable lesson

Do not optimise a repeated inference until you have asked why it is being repeated at all.

The consultant's questions

• Which expensive operations repeat against unchanged input?
• What can be cached safely and for how long?
• Are embeddings or analyses regenerated unnecessarily?
• Could a batch or incremental job replace thousands of independent calls?