Processing Audio in Chat Completions

This example shows how to use AvalAI's OpenAI-compatible /v1/chat/completions endpoint with audio-capable models. Keep Chat Completions when the model must accept input_audio or return message.audio directly. Use the Responses migration path when you want transcript-based reasoning, tools, structured output, or state handling before generating speech.

Related docs: Audio API, Audio processing guide, Responses vs Chat Completions

Available Audio Chat Models

Model	Best for
gpt-audio-1.5	Highest-quality audio conversations and longer context.
gpt-audio	Balanced audio input/output workflows.
gpt-audio-mini	Lower-cost development, support bots, and high-volume voice features.

Check model details before deploying because endpoint availability can vary by account tier and provider route.

Pattern 1: Generate Spoken Audio from Text

Use this pattern when the user sends text and you want the model to respond with both text and audio.

Bash

curl https://api.avalai.ir/v1/chat/completions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-audio-mini",
    "modalities": ["text", "audio"],
    "audio": { "voice": "alloy", "format": "wav" },
    "messages": [
      {
        "role": "system",
        "content": "You are a friendly voice assistant. Keep answers brief."
      },
      {
        "role": "user",
        "content": "Explain how AvalAI billing works in one paragraph."
      }
    ]
  }'

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

completion = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "wav"},
    messages=[
        {
            "role": "system",
            "content": "You are a friendly voice assistant. Keep answers brief.",
        },
        {
            "role": "user",
            "content": "Explain how AvalAI billing works in one paragraph.",
        },
    ],
)

message = completion.choices[0].message
print(message.content)

if message.audio:
    with open("answer.wav", "wb") as output:
        output.write(base64.b64decode(message.audio.data))

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const completion = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "alloy", format: "wav" },
  messages: [
    {
      role: "system",
      content: "You are a friendly voice assistant. Keep answers brief.",
    },
    {
      role: "user",
      content: "Explain how AvalAI billing works in one paragraph.",
    },
  ],
});

const message = completion.choices[0].message;
console.log(message.content);

if (message.audio?.data) {
  await fs.writeFile("answer.wav", Buffer.from(message.audio.data, "base64"));
}

Pattern 2: Send Audio Input to the Model

Use input_audio when the model should reason over audio directly. Keep the audio file short enough to fit the selected model and request limits.

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("customer_question.wav", "rb") as audio_file:
    audio_b64 = base64.b64encode(audio_file.read()).decode("utf-8")

completion = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "coral", "format": "wav"},
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is the customer asking? Answer briefly.",
                },
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio_b64, "format": "wav"},
                },
            ],
        }
    ],
)

print(completion.choices[0].message.content)

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const audioB64 = (await fs.readFile("customer_question.wav")).toString("base64");

const completion = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "coral", format: "wav" },
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What is the customer asking? Answer briefly." },
        {
          type: "input_audio",
          input_audio: { data: audioB64, format: "wav" },
        },
      ],
    },
  ],
});

console.log(completion.choices[0].message.content);

Pattern 3: Continue a Voice Conversation

For short conversations, store the text transcript in your application and include previous assistant/user turns in messages. Store audio bytes separately; only send audio again when the model needs to inspect it.

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

messages = [
    {"role": "system", "content": "You are a concise voice support assistant."},
    {"role": "user", "content": "Can I use OpenAI SDKs with AvalAI?"},
]

first = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "mp3"},
    messages=messages,
)

messages.append({"role": "assistant", "content": first.choices[0].message.content})
messages.append({"role": "user", "content": "Show me the base URL too."})

second = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "mp3"},
    messages=messages,
)

print(second.choices[0].message.content)

Responses API migration path

Direct input_audio and message.audio belong on Chat Completions for now. When the workflow benefits from Responses, split it into request-based audio steps:

1. Convert user audio to text with /v1/audio/transcriptions.
2. Send the transcript to /v1/responses.
3. Read response.output_text.
4. Create speech with /v1/audio/speech.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("customer_question.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",
        file=audio_file,
        response_format="text",
    )

response = client.responses.create(
    model="gpt-5.5",
    instructions="You are a concise support assistant. Answer for spoken playback.",
    input=transcript,
)

with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="coral",
    input=response.output_text,
) as speech:
    speech.stream_to_file(Path("response.mp3"))

JavaScript

import fs from "node:fs";
import fsp from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const transcript = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: fs.createReadStream("customer_question.wav"),
  response_format: "text",
});

const response = await client.responses.create({
  model: "gpt-5.5",
  instructions: "You are a concise support assistant. Answer for spoken playback.",
  input: transcript,
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "coral",
  input: response.output_text,
});

await fsp.writeFile("response.mp3", Buffer.from(await speech.arrayBuffer()));

Use this migration path when you need Responses features such as tool calls, structured outputs, previous_response_id, or manual Item replay. Keep Chat Completions when direct model audio output is the main feature.

Format and Latency Tips

• Use wav or pcm when low playback latency matters.
• Use mp3 for compact files and broad compatibility.
• Prefer gpt-audio-mini while developing; move to gpt-audio or gpt-audio-1.5 when quality or context length matters.
• Avoid repeatedly sending the same audio bytes in multi-turn chats; store transcripts and resend only the text context unless the model must inspect audio again.
• For long recordings, prefer /v1/audio/transcriptions and chunking over base64 audio inside Chat Completions.

Troubleshooting

Symptom	Fix
No audio in the response	Include "audio" in modalities and set the audio object with voice and format.
input_audio rejected	Confirm the selected model supports audio input and that the format matches the encoded bytes.
Payload too large	Use /v1/audio/transcriptions for files, compress audio, or split long recordings.
Model ignores previous audio	Store and resend the text transcript; do not assume raw audio remains available across requests.
Need tools or structured output	Use the Responses migration path, then synthesize the final text with /v1/audio/speech.