Work backwards from the decision
Batch processing lets you submit a group of requests and collect the results later. It can suit a catalogue clean-up, a set of evaluations or a weekly classification report. Ask the person who uses the output for the latest useful delivery time. Then subtract the time they need to check exceptions and prepare the decision. That gives you the processing window you actually have.
Google describes a 24-hour target turnaround for its Gemini Batch API. Anthropic documents a 24-hour expiry for Message Batches and identifies requests that expire before processing. These belong to different services. Check the model, account limits, data handling and terms available to your organisation before picking one. A provider’s target is a poor basis for promising a customer a particular delivery time.
Illustrative example: a weekly complaints report
Imagine a UK distributor whose service manager reviews delivery complaints at noon on Friday. Staff already handle urgent complaints in the support desk as they arrive. The manager wants a separate count of recurring causes so the warehouse team can decide what to investigate. Each closed ticket can be classified on its own.
Suppose the team freezes the reporting list at noon on Wednesday. It reserves Friday morning for checking disputed categories and Thursday afternoon for recovering missing results. That leaves roughly a day for the first processing attempt. If the meeting moves to Thursday morning, the team must reconsider the plan. A faster result from last week gives them little protection against this week’s deadline.
Keep urgent cases in their existing queue. A customer reporting a missing delivery should still get a response while the background analysis runs. Label the report with its cut-off so staff can tell which late complaints will appear next week.
Make a list you can reconcile
Before submission, save one row for each item you intend to process. Give it an opaque request reference and record the source record, source version, reporting period and classification rules used. Keep the mapping in your own system. Send only the fields the model needs and check who can access the files holding requests and results.
When results arrive, join them to that list using the request reference. OpenAI’s Batch API documentation explains that output can appear in a different order from the input and provides a custom_id for matching. Joining on row position could give one customer’s complaint the category intended for another. Check for repeated, missing and unfamiliar references before importing anything.
Check the version again before applying a result. If a ticket was corrected after submission, the team can either keep the original snapshot for the dated report or reclassify the corrected record. Make that choice explicitly and record it so the figures can be explained later.
Decide what to do with an incomplete report
Give the person preparing the report a count of expected items, accepted classifications and unresolved items. Anthropic, for example, distinguishes successful, errored, cancelled and expired requests. An individual request may also succeed technically and return a category that your team cannot use. Put those records in the review queue.
Suppose 200 tickets were submitted, 174 classifications passed review, 16 were ambiguous and 10 had no usable response. Show all three counts. If most of the ambiguous tickets mention damaged orders, leaving them out could change which cause looks most common. The manager can review those tickets first, issue a report marked provisional or postpone the decision.
Retry only items that still need a result. Check job status before resubmitting after a connection failure. Make each accepted update safe to apply again and give someone authority to stop recovery at the cut-off. This avoids a late retry overwriting a category a member of staff has already corrected.
Compare the cost per usable result
Choose a previous reporting period and run a trial with approved, representative data. Use the same inputs, model and classification instructions for the batch and immediate routes where possible. Ask the reviewer to judge the outputs against the same criteria. Include awkward cases and measure the time until the report is ready for the meeting.
Add model charges, file handling, retries and staff time spent reconciling results. Divide by the number of classifications accepted before the deadline. A discounted model call can still produce an expensive report if recovery takes most of Friday morning. Check current provider prices for your intended model and record the assumptions alongside the comparison.
For the first live report, agree the latest submission time, a named reviewer and the minimum coverage needed to make the warehouse decision. Rehearse a partial return before the meeting. If the team can explain every missing item and still deliver a useful report on time, it has a basis for moving the next reporting period to batch.