Batch Processing

Warning

Batch API support is currently under development in AvalAI. Use this guide to prepare batch-shaped workloads and to run a supported rate-limit-safe client-side alternative today.

Batch processing is useful when you have many independent requests that do not need an immediate response: nightly evals, offline classification, embeddings generation, data cleanup, or content review. When the hosted Batch API becomes available, use /v1/batches. Until then, use controlled client-side concurrency with retries and rate-limit headers.

Adapted from the official OpenAI Batch API guide and OpenAI Cookbook patterns, with AvalAI endpoint, API key, and model changes.

Tip

OpenAI's hosted Batch API has OpenAI-specific benefits such as a 50% discount and a separate higher-limit pool. Do not model those as AvalAI guarantees yet. The supported pattern today is a client-side worker that consumes normal AvalAI route/model pricing and rate limits.

Choose the Right Pattern

Workload	Recommended path today
Immediate user response	Synchronous Chat Completions or Responses request
Many independent requests	Rate-limit-safe client-side processor
Prompt/model regression checks	Promptfoo Evals with AvalAI
Future hosted async batches	/v1/batches, when available

Hosted Batch Shape to Plan For

OpenAI's current Batch API flow is: prepare a .jsonl file, upload it with purpose="batch", create a batch with input_file_id, poll the batch object, then download output_file_id and inspect error_file_id for failed rows. Design AvalAI batch-shaped jobs around the same lifecycle so migration is mostly a transport change when hosted batches are enabled.

Current OpenAI-compatible batch targets include:

Endpoint	Good fit
/v1/responses	Offline reasoning, extraction, summarization, and tool-free generation
/v1/chat/completions	Existing chat workloads that should stay on Chat Completions
/v1/embeddings	Repository, catalog, or document embedding jobs
/v1/moderations	Large-scale text or image moderation queues
/v1/images/generations and /v1/images/edits	Offline image production or editing queues
/v1/videos	Offline video-render queues when JSON request bodies are supported

Endpoint-specific notes from the OpenAI Batch shape:

• Keep each input file scoped to one endpoint and one model.
• Do not set stream: true in batch rows; batch outputs are returned later through files, not live streams.
• /v1/embeddings batches are also limited to 50,000 embedding inputs across all requests.
• /v1/moderations rows must include input; use omni-moderation-latest for text plus image inputs and prefer image_url over base64 for large images.
• /v1/videos batch rows should use JSON bodies only. Upload assets ahead of time and reference them by supported file IDs or image URLs instead of multipart uploads.

Prepare JSONL Inputs

Keep one request per line. This shape is easy to process locally and can later be reused for hosted batches.

{"custom_id":"ticket-001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.5","messages":[{"role":"system","content":"Classify the ticket as Hardware, Software, Billing, Account, or Other. Return only the label."},{"role":"user","content":"My monitor does not turn on."}]}}
{"custom_id":"ticket-002","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.5","messages":[{"role":"system","content":"Classify the ticket as Hardware, Software, Billing, Account, or Other. Return only the label."},{"role":"user","content":"I was charged twice this month."}]}}

Use stable custom_id values so retries and results can be reconciled safely. Output order is not guaranteed to match input order, so treat custom_id as the join key.

Responses API Batch Rows

For new text-generation jobs, prefer a Responses-shaped row. It keeps instructions separate from user input and makes the eventual migration to /v1/responses direct.

{"custom_id":"ticket-001","method":"POST","url":"/v1/responses","body":{"model":"gpt-5.5","instructions":"Classify the ticket as Hardware, Software, Billing, Account, or Other. Return only the label.","input":"My monitor does not turn on."}}
{"custom_id":"ticket-002","method":"POST","url":"/v1/responses","body":{"model":"gpt-5.5","instructions":"Classify the ticket as Hardware, Software, Billing, Account, or Other. Return only the label.","input":"I was charged twice this month."}}

Run a Client-Side Batch Safely

import json
import os
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)


def run_request(item):
    if item.get("method", "POST") != "POST":
        raise ValueError(f"Unsupported method: {item.get('method')}")

    url = item["url"]
    body = item["body"]

    if url == "/v1/chat/completions":
        response = client.chat.completions.create(**body)
        return response.choices[0].message.content, response.model_dump()

    if url == "/v1/responses":
        response = client.responses.create(**body)
        return response.output_text, response.model_dump()

    if url == "/v1/embeddings":
        response = client.embeddings.create(**body)
        vector_lengths = [
            len(embedding_item.embedding) for embedding_item in response.data
        ]
        return vector_lengths, response.model_dump()

    raise ValueError(f"Unsupported batch row URL: {url}")


def run_one(item, max_retries=5):
    for attempt in range(max_retries):
        try:
            output, response_body = run_request(item)
            return {
                "custom_id": item["custom_id"],
                "url": item["url"],
                "output": output,
                "response": response_body,
            }
        except Exception as exc:
            message = str(exc)
            if "429" not in message and "rate" not in message.lower():
                raise
            wait = min(60, 2**attempt)
            time.sleep(wait)

    raise RuntimeError(f"Retries exhausted for {item['custom_id']}")


with open("requests.jsonl", "r", encoding="utf-8") as f:
    requests = [json.loads(line) for line in f if line.strip()]

results = []
with ThreadPoolExecutor(max_workers=4) as pool:
    futures = [pool.submit(run_one, item) for item in requests]
    for future in as_completed(futures):
        results.append(future.result())

with open("results.jsonl", "w", encoding="utf-8") as f:
    for row in results:
        f.write(json.dumps(row, ensure_ascii=False) + "\n")

The local worker routes rows by url, so one code path can process Chat Completions, Responses, and Embeddings rows. Add explicit handlers before using additional routes such as Images or Moderations.

Start with low concurrency and raise it only after watching your rate limit headers. For large jobs, split input files into chunks and checkpoint results after each chunk.

Add Status Polling and Your Own Callback

OpenAI's hosted async patterns split into two related ideas: Batch API for many independent rows, and background Responses for one long-running response that can be polled by ID. AvalAI Batch and background Responses are not generally available yet, so implement the same developer experience in your own worker:

1. Create a local job record with queued, in_progress, completed, failed, and cancelled states.
2. Store every input row by custom_id, prompt version, model, and retry count.
3. Run workers with bounded concurrency and write partial results as soon as each row finishes.
4. Expose GET /jobs/{id} so clients can poll without holding an HTTP connection open.
5. If a callback is useful, send your own webhook after the job reaches a terminal state.

If you mirror OpenAI Batch status names in your own worker, map them to local behavior explicitly:

OpenAI-style status	Local worker behavior
validating / failed	Validate JSONL shape, endpoint, model availability, and unique custom_id values before starting API calls.
in_progress	Run bounded workers, observe AvalAI rate-limit headers, and checkpoint each completed row.
finalizing / completed	Write output rows and error rows, persist counts, then mark the job terminal.
expired	Keep completed rows, record unfinished custom_id values, and retry only unfinished work in a new job.
cancelling / cancelled	Stop scheduling new rows, let in-flight calls settle when safe, and preserve partial results.

For webhook-style callbacks, borrow the operational rules from OpenAI's webhook guidance: respond quickly with a 2xx, move expensive processing to a background worker, retry failed deliveries with backoff, and deduplicate events with a stable event ID. Keep a signing secret for callbacks you emit or receive so downstream systems can verify the payload.

Need	AvalAI-safe pattern today
One long response that may exceed a client timeout	Queue one Responses or Chat Completions request in your worker and poll your own job record
Thousands of independent rows	Use the JSONL + client-side worker pattern in this guide
External notification on completion	Emit an application webhook from your worker after storing results
Later migration to hosted Batch	Preserve .jsonl, custom_id, output_file_id/error_file_id-style metadata in your job table

Operational Checklist

• Use idempotent custom_id values.
• Store failed rows separately so they can be retried without repeating the full job.
• Cap concurrency by both requests per minute and tokens per minute.
• Add exponential backoff for 429 responses.
• Keep prompts and model IDs versioned with the dataset.
• Persist job state so clients can safely poll and reconnect.
• Sign any application-level webhook callbacks you send to downstream systems.
• Keep JSONL upload files below 200 MB and split very large jobs into smaller shards.
• Store output_file_id and error_file_id with the job record when hosted batches become available.
• Expect completed output files to expire according to the hosted provider's retention policy; OpenAI's reference behavior deletes batch output files after 30 days.
• For expired batches, keep completed rows and retry only unfinished custom_id values from the error file.
• For eval workloads, use Promptfoo Evals with AvalAI so results are easier to compare in CI.

Related Resources

• Batch API Reference
• Rate Limits
• Rate-Limit-Safe Parallel Requests
• Response Headers
• Background Processing
• Webhooks
• OpenAI Batch API guide
• OpenAI Background mode guide
• OpenAI Webhooks guide