← Blog
Lavender documents held by an orange binder clip in a shallow document tray.
Published on
2026-11-13

Batch inference: plan a reliable offline AI workload

Batch inference runs a trained model over a defined collection of inputs without requiring an immediate response for each one. It suits work such as overnight document classification, scheduled embedding generation, and image labeling. The useful question is whether the full job can finish correctly by its deadline, at an acceptable cost.

A fast model is only part of that answer. You also need to know which inputs were processed, which outputs passed validation, and what happens when a worker stops halfway through. This guide uses a fictional nightly document-labeling job to make those decisions concrete. It describes an architecture to evaluate, not a tested deployment or a GPU benchmark.

What does batch inference mean?

Training changes a model's parameters; inference uses a trained model to produce predictions or generated outputs. In a batch inference workflow, a job processes a finite dataset and produces results for later use. “Offline” usually describes the absence of an interactive response requirement. It does not mean the system has no network connection.

A managed service can provide that job interface. For example, Amazon SageMaker Batch Transform supports dataset inference without a persistent endpoint. You can also operate a worker yourself: vLLM documents offline inference through its LLM class. These are different operating arrangements, with different responsibilities for scheduling, recovery, and infrastructure.

TermDecision it describesExample
Offline batch inferenceWhen a defined job must finishLabel yesterday's documents before the morning report
Device batch sizeHow much work is processed together on a deviceGroup compatible inputs within available memory
Continuous batchingHow a serving engine schedules active requestsAdmit new sequences as others finish

An offline job can use an engine that performs continuous batching internally. Neither term guarantees a particular throughput or discount. Our continuous batching guide covers the serving behavior in more detail.

Start with a deadline and an acceptance rule

Suppose a team needs to classify 10,000 documents overnight. The morning report needs one approved category per valid document, while unreadable files must appear in an exception report. Before choosing an instance, define when the results are needed, which categories are allowed, and whether unresolved inference failures block delivery.

Use a representative sample to evaluate output quality and estimate the work. Include long documents, unusual languages, empty files, and records near input limits. An estimate based only on short, clean examples can hide the cases that dominate elapsed time or memory. Reserve time for provisioning, model loading, validation, and recovery, rather than allocating the whole window to generation.

Use an interactive service when people or applications need responses as requests arrive. For recurring finite jobs with flexible deadlines, evaluate a worker that starts for the job and stops afterward. Compare the complete operating cost and reliability of both approaches; infrequent use alone does not prove the batch design is cheaper.

Give every input a stable identity

Create an input manifest before starting the run. It should identify the intended dataset and make it possible to reproduce the job's scope, even if the source collection changes later. A practical manifest records:

  • A job identifier, creation time, deadline, and expected input count.
  • A stable record ID and immutable input reference or verified content hash for each document.
  • The model revision, tokenizer, prompt template, label definitions, and generation settings.
  • The preprocessing revision and output schema version.
  • The storage locations and retention rules for outputs and failure records.

Use immutable input copies or verify that each referenced object still matches the recorded content before processing it. Keep credentials out of the manifest. Restrict access to source data, generated text, and logs according to the dataset's sensitivity.

A stable record ID lets you associate results with inputs even when workers finish out of order. Keep attempt IDs separately. A useful logical output key combines the job definition or version with the record ID; changing the model or prompt should create a distinguishable run rather than silently replace the previous evidence.

Process bounded chunks and record progress

Split the manifest into manageable chunks. Chunk size controls scheduling and recovery scope; it does not have to equal the model's device batch size. Avoid making one enormous unit that must restart from the beginning after a failure.

For a language model, consider input length, expected output length, memory use, and engine limits when grouping records. Similar-length inputs may be easier to schedule efficiently, but preserve their IDs and test the actual workload. An out-of-memory error should trigger a controlled change or an exception, not an unlimited retry of the same failing batch.

CPU preprocessing can also become the bottleneck. Measure parsing and tokenization separately from inference before increasing GPU capacity. If you parallelize preparation, the Python multiprocessing guide explains the process-management concerns. More workers can increase memory pressure and contention, so set explicit concurrency limits.

Write attempt results and validation outcomes as work finishes. Only mark a chunk committed after its expected outputs and exception records have been verified. If files or objects form the result, publish a completion manifest or pointer after those writes succeed. Do not assume several object writes are one atomic transaction.

Make retries safe and visible

A worker may lose its connection after producing an output but before recording success. Retrying blindly can create duplicate results or duplicate downstream actions. The AWS Builders' Library discussion of idempotent APIs explains why an uncertain response does not necessarily mean the original operation failed.

For this job, keep one authoritative final state per logical output key and store attempts separately. Coordinate workers so that two successful attempts cannot both publish the same logical result. Enforce this through the chosen database or coordination mechanism; a naming convention alone is insufficient.

Retry transient failures with a limit and delay. Put invalid inputs and persistent failures into a separate review queue. On restart, reconcile existing committed results before resubmitting work. A rerun should target the unresolved records unless the job definition has changed.

Idempotent publication prevents duplicate logical results; it does not guarantee identical model outputs across attempts. Preserve the accepted output and its provenance. Keep irreversible actions, such as sending customer messages, outside the inference worker and give that later stage its own duplicate-prevention rules.

Audit completion before releasing results

A process exiting successfully is not proof that the dataset is complete. Compare unique record IDs against the input manifest, detect missing and unexpected IDs, and check for conflicting final states. Validate the output schema, allowed labels, and required fields, then sample results for task quality.

Final stateIllustrative countDelivery treatment
Accepted output9,600Eligible for the report after quality checks
Rejected input300Included in the documented exception report
Unresolved failure100Requires recovery or an explicit delivery decision
Total accounted for10,000Matches the manifest, but does not mean every record succeeded

These numbers are fictional. They illustrate reconciliation, not measured model performance. The 100 unresolved failures still block a delivery rule that requires every valid document to be classified. Define any permitted exceptions before the run; do not quietly weaken the rule to meet the deadline.

Measure cost per accepted output

Track elapsed time from job submission to validated delivery, alongside the time spent preparing inputs, loading the model, generating, retrying, and validating. Record accepted outputs, exceptions, and unresolved failures. Raw tokens per second cannot tell you whether the report arrived on time.

For a stated accounting scope, divide total job cost by the number of accepted outputs. Include compute during startup and idle periods, storage, transfers, retries, and any paid orchestration. Track engineering and review effort separately or include it consistently in both alternatives. A zero-accepted-output job has no meaningful cost-per-accepted-output ratio.

Compare configurations using the same dataset and acceptance criteria. A faster configuration that produces more unusable results may cost more per accepted output. Stop or release resources after committed delivery and apply the agreed retention policy to inputs, intermediate files, logs, and outputs. Keep enough audit evidence to investigate the run.

Plan a batch workload with Hivenet

Compute with Hivenet provides CPU and GPU infrastructure to evaluate for customer-operated jobs. Object Storage with Hivenet provides an S3-compatible storage option for datasets and results. Confirm capacity, access, compatibility, and operating responsibilities for the proposed workload.

This architecture does not assume Hivenet supplies a managed Batch API, automatic retry orchestration, or a fixed inference discount. Your application still needs the job controls described above. For a broader deployment checklist, see our production LLM inference guide.

To discuss a batch inference workload, bring the model and revision, dataset size, input and output length distribution, deadline, representative sample, acceptance criteria, and data-handling requirements. Those details make it possible to scope a trial that tests the full job rather than just whether the model loads.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.