Batch API Reference

Warning

Feature Not Implemented!

This functionality is currently under development and not yet available in AvalAI. For batch-shaped workloads today, use the Batch Processing Guide and Rate-Limit-Safe Parallel Requests.

Create large batches of API requests for asynchronous processing. The OpenAI-compatible Batch API pattern returns results within a 24-hour window and is designed for offline work that can wait for asynchronous completion.

Tip

OpenAI's hosted Batch API advertises a 50% discount and a separate higher-limit pool for OpenAI accounts. Those commercial terms are OpenAI-hosted behavior, not an AvalAI guarantee while /v1/batches is under development. Until this endpoint is enabled, client-side batch workers use normal AvalAI route/model pricing and rate limits.

Related guide: Batch Processing Guide

Related examples:

• Rate-Limit-Safe Parallel Requests
• Promptfoo Evals with AvalAI

OpenAI-compatible workflow

When this endpoint is enabled in AvalAI, use the same lifecycle as the OpenAI Batch API:

1. Prepare a .jsonl file with one request per line.
2. Upload the file with purpose="batch" through the Files API.
3. Create a batch with input_file_id, endpoint, and completion_window.
4. Poll the batch object until it reaches completed, failed, expired, or cancelled.
5. Download output_file_id for successful rows and inspect error_file_id for failed or expired rows.

Use stable custom_id values because output row order may not match input row order.

Planning constraints

• Keep every JSONL input file scoped to one endpoint and one model.
• Use unique custom_id values; join output rows by custom_id, not by line order.
• Do not set stream: true; batch results are written to output and error files.
• OpenAI's reference limits are 50,000 requests per batch and 200 MB per input file; AvalAI limits may be narrower during rollout.
• Do not assume OpenAI Batch discounts, retention, or separate rate-limit pools apply to AvalAI until AvalAI documents them for /v1/batches.
• For /v1/moderations, include input in every row. Use image_url instead of large base64 payloads for multimodal moderation to keep JSONL files small.
• For /v1/videos, plan for JSON bodies only: upload assets first and reference them with supported file IDs or image URLs rather than multipart uploads.

Status and callback planning

AvalAI does not currently expose hosted Batch webhooks. When you need completion notifications before /v1/batches is enabled, run the job through your own worker and emit your own application webhook after results are stored. Use a stable event ID for idempotency, sign callback payloads with a secret, and make receivers acknowledge quickly with 2xx before doing expensive work. See Webhooks for a receiver pattern.

OpenAI background Responses are a separate async pattern for one long-running response, not a replacement for Batch rows. If AvalAI exposes background Responses in the future, expect to poll a Response object until it leaves queued or in_progress; until then, model this behavior in your own job table.

Batch status playbook

Use statuses as workflow states, not just display labels:

Status	Meaning	What your app should do
validating	The input file is being checked before execution starts.	Keep polling with backoff; surface validation progress only to operators.
failed	The input file failed validation.	Stop retries for the same file, inspect errors or error_file_id, fix JSONL rows, and submit a new batch.
in_progress	Rows are being executed.	Continue polling; do not assume output files are complete.
finalizing	Execution finished and result files are being prepared.	Keep polling; prepare storage for output and error file downloads.
completed	Result files are ready.	Download output_file_id, process rows by custom_id, and archive the batch metadata.
expired	The 24-hour window ended before all rows completed.	Keep completed rows, inspect expired rows in the error file, and resubmit only unfinished custom_ids.
cancelling	Cancellation is in progress and in-flight work may still finish.	Pause downstream consumers and wait for cancelled.
cancelled	Cancellation is complete.	Download any partial results, mark unfinished rows as cancelled, and avoid double-processing completed rows.

For your own worker-based fallback, mirror the same state machine. This makes it easier to migrate to hosted /v1/batches later without changing downstream reporting.

Create batch

POST https://api.avalai.ir/v1/batches

Creates and executes a batch from an uploaded file of requests.

Request Body

Parameter	Type	Required	Description
input_file_id	string	Yes	The ID of an uploaded file that contains requests for the new batch. Your input file must be formatted as a JSONL file, and must be uploaded with the purpose batch. The file can contain up to 50,000 requests, and can be up to 200 MB in size. See upload file for how to upload a file.
endpoint	string	Yes	The endpoint to be used for all requests in the batch. OpenAI-compatible targets are /v1/responses, /v1/chat/completions, /v1/embeddings, /v1/completions (legacy), /v1/moderations, /v1/images/generations, /v1/images/edits, and /v1/videos. AvalAI availability may be narrower during rollout. /v1/embeddings batches are also restricted to a maximum of 50,000 embedding inputs across all requests in the batch.
completion_window	string	Yes	The time frame within which the batch should be processed. Currently only 24h is supported.
metadata	map	No	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
output_expires_after	object	No	Optional expiration policy for generated output and error files when the selected route supports it. Use the same anchor: "created_at" and seconds shape as Files API expiration, and download important result files before they expire.

Example Request

Bash

curl https://api.avalai.ir/v1/batches \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input_file_id": "file-abc123",
  "endpoint": "/v1/chat/completions",
  "completion_window": "24h",
  "metadata": {
    "customer_id": "user_123456789",
    "batch_description": "Nightly eval job"
  }
}'

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

batch = client.batches.create(
    input_file_id="file-abc123",
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"customer_id": "user_123456789", "batch_description": "Nightly eval job"},
)
print(batch)

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

async function main() {
  const batch = await client.batches.create({
    input_file_id: "file-abc123",
    endpoint: "/v1/chat/completions",
    completion_window: "24h",
    metadata: {
      customer_id: "user_123456789",
      batch_description: "Nightly eval job",
    },
  });
  console.log(batch);
}
main();

Responses API version

To run a Responses-shaped batch, create the batch with endpoint: "/v1/responses" and prepare the input file with /v1/responses rows.

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

batch = client.batches.create(
    input_file_id="file-abc123",
    endpoint="/v1/responses",
    completion_window="24h",
    metadata={"batch_description": "Nightly Responses eval job"},
)

print(batch)

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const batch = await client.batches.create({
  input_file_id: "file-abc123",
  endpoint: "/v1/responses",
  completion_window: "24h",
  metadata: {
    batch_description: "Nightly Responses eval job",
  },
});

console.log(batch);

Bash

curl https://api.avalai.ir/v1/batches \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '
  {
    "input_file_id": "file-abc123",
    "endpoint": "/v1/responses",
    "completion_window": "24h",
    "metadata": {
      "batch_description": "Nightly Responses eval job"
    }
  }'

• Batch creation still uses /v1/batches; only the batch endpoint changes to /v1/responses.
• In each JSONL row, messages → input and system/developer guidance → instructions or a developer item.
• In output rows, read the generated text from the Responses body, typically body.output_text.

Returns

The created Batch object.

Retrieve batch

GET https://api.avalai.ir/v1/batches/{batch_id}

Retrieves a batch.

Path Parameters

Parameter	Type	Required	Description
batch_id	string	Yes	The ID of the batch to retrieve.

Example Request

curl https://api.avalai.ir/v1/batches/batch_abc123 \
  -H "Authorization: Bearer $AVALAI_API_KEY"

Returns

The Batch object matching the specified ID.

Cancel batch

POST https://api.avalai.ir/v1/batches/{batch_id}/cancel

Cancels an in-progress batch. The batch will be in status cancelling for up to 10 minutes, before changing to cancelled, where it will have partial results (if any) available in the output file.

Path Parameters

Parameter	Type	Required	Description
batch_id	string	Yes	The ID of the batch to cancel.

Example Request

curl https://api.avalai.ir/v1/batches/batch_abc123/cancel \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -X POST

Returns

The cancelled Batch object.

List batches

GET https://api.avalai.ir/v1/batches

List your organization's batches.

Query Parameters

Parameter	Type	Required	Description
after	string	No	A cursor for use in pagination. after is an object ID that defines your place in the list.
limit	integer	No	A limit on the number of objects to be returned. Limit can range between 1 and 100, and the default is 20.

Example Request

curl "https://api.avalai.ir/v1/batches?limit=2" \
  -H "Authorization: Bearer $AVALAI_API_KEY"

Returns

A list of paginated Batch objects.

The batch object

Parameter	Type	Description
id	string	The identifier, which can be referenced in API endpoints.
object	string	The object type, which is always batch.
endpoint	string	The AvalAI API endpoint used by the batch.
errors	object or null	Contains details about errors if any occurred during batch processing.
input_file_id	string	The ID of the input file for the batch.
completion_window	string	The time frame within which the batch should be processed.
status	string	The current status of the batch (e.g., validating, in_progress, completed, failed, cancelling, cancelled, expired).
output_file_id	string or null	The ID of the file containing the outputs of successfully executed requests.
error_file_id	string or null	The ID of the file containing the outputs of requests with errors.
created_at	integer	The Unix timestamp (in seconds) for when the batch was created.
in_progress_at	integer or null	The Unix timestamp (in seconds) for when the batch started processing.
expires_at	integer or null	The Unix timestamp (in seconds) for when the batch will expire.
finalizing_at	integer or null	The Unix timestamp (in seconds) for when the batch started finalizing.
completed_at	integer or null	The Unix timestamp (in seconds) for when the batch was completed.
failed_at	integer or null	The Unix timestamp (in seconds) for when the batch failed.
expired_at	integer or null	The Unix timestamp (in seconds) for when the batch expired.
cancelling_at	integer or null	The Unix timestamp (in seconds) for when the batch started cancelling.
cancelled_at	integer or null	The Unix timestamp (in seconds) for when the batch was cancelled.
request_counts	object	The request counts for different statuses within the batch (total, completed, failed).
metadata	map	Set of key-value pairs attached to the object.

Example Batch Object

{
  "id": "batch_abc123",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": "file-abc123",
  "completion_window": "24h",
  "status": "completed",
  "output_file_id": "file-cvaTdG",
  "error_file_id": "file-HOWS94",
  "created_at": 1711471533,
  "in_progress_at": 1711471538,
  "expires_at": 1711557933,
  "finalizing_at": 1711493133,
  "completed_at": 1711493163,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {
    "total": 100,
    "completed": 95,
    "failed": 5
  },
  "metadata": {
    "customer_id": "user_123456789",
    "batch_description": "Nightly eval job"
  }
}

Responses API version

When the batch targets /v1/responses, the batch object uses the same lifecycle fields; only endpoint and the eventual row body shape differ.

{
  "id": "batch_abc123",
  "object": "batch",
  "endpoint": "/v1/responses",
  "input_file_id": "file-abc123",
  "completion_window": "24h",
  "status": "completed",
  "output_file_id": "file-cvaTdG",
  "error_file_id": null
}

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Result files and expiration

When a batch reaches completed, use output_file_id with the Files API to download successful results. Rows that fail validation, expire, or fail during execution are written to error_file_id when available. If you set output_expires_after, copy durable results to your own storage before the output files expire.

Output order may differ from input order, so always join rows by custom_id. If a batch expires before all rows finish, completed rows remain available and unfinished rows are reported as errors.

Request input object

The structure of each line in the JSONL input file.

Parameter	Type	Required	Description
custom_id	string	Yes	A developer-provided per-request ID that will be used to match outputs to inputs. Must be unique for each request in a batch.
method	string	Yes	The HTTP method to be used for the request. Currently only POST is supported.
url	string	Yes	The AvalAI API relative URL to be used for the request (e.g., /v1/chat/completions or /v1/responses).
body	object	Yes	The request body for the API call (e.g., parameters for chat completions).

Keep all rows in one input file on the same endpoint and model. Do not set stream: true; batch results are delivered through output files after processing.

Example Input Line

{
  "custom_id": "request-1",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "gpt-5.4-mini",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is 2+2?"
      }
    ]
  }
}

Responses API version

Use this line shape when the selected model supports /v1/responses. messages moves to input, the system message moves to instructions, and the final text is read from response.output_text inside the response body.

{
  "custom_id": "request-1",
  "method": "POST",
  "url": "/v1/responses",
  "body": {
    "model": "gpt-5.4-mini",
    "input": "What is 2+2?",
    "instructions": "You are a helpful assistant."
  }
}

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Request output object

The structure of each line in the JSONL output file(s).

Parameter	Type	Description
id	string	The ID of the batch request.
custom_id	string	The developer-provided ID from the input file.
response	object or null	The response object from the API call if successful. Includes status_code, request_id, and the body.
error	object or null	Details about the error if the request failed.

Example Output Line (Success)

{
  "id": "batch_req_wnaDys",
  "custom_id": "request-2",
  "response": {
    "status_code": 200,
    "request_id": "req_c187b3",
    "body": {
      "id": "chatcmpl-9758Iw",
      "object": "chat.completion",
      "created": 1711475054,
      "model": "gpt-5.4-mini",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "2 + 2 equals 4."
          },
          "finish_reason": "stop"
        }
      ],
      "usage": {
        "prompt_tokens": 24,
        "completion_tokens": 15,
        "total_tokens": 39
      },
      "system_fingerprint": null
    }
  },
  "error": null
}

Example Output Line (Error)

{
  "id": "batch_req_abcxyz",
  "custom_id": "request-3",
  "response": null,
  "error": {
    "code": "invalid_request_error",
    "message": "Invalid model ID provided."
  }
}