Processing Audio in Chat Completions

This example shows how to use AvalAI's OpenAI-compatible /v1/chat/completions endpoint with audio-capable models. Keep Chat Completions when the model must accept input_audio or return message.audio directly. Use the Responses migration path when you want transcript-based reasoning, tools, structured output, or state handling before generating speech.

Related docs: Audio API, Audio processing guide, Responses vs Chat Completions

Available Audio Chat Models

Model	Best for
gpt-audio-1.5	Highest-quality audio conversations and longer context.
gpt-audio	Balanced audio input/output workflows.
gpt-audio-mini	Lower-cost development, support bots, and high-volume voice features.

Check model details before deploying because endpoint availability can vary by account tier and provider route.

Pattern 1: Generate Spoken Audio from Text

Use this pattern when the user sends text and you want the model to respond with both text and audio.

Bash

curl https://api.avalai.ir/v1/chat/completions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-audio-mini",
    "modalities": ["text", "audio"],
    "audio": { "voice": "alloy", "format": "wav" },
    "messages": [
      {
        "role": "system",
        "content": "You are a friendly voice assistant. Keep answers brief."
      },
      {
        "role": "user",
        "content": "Explain how AvalAI billing works in one paragraph."
      }
    ]
  }'

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

completion = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "wav"},
    messages=[
        {
            "role": "system",
            "content": "You are a friendly voice assistant. Keep answers brief.",
        },
        {
            "role": "user",
            "content": "Explain how AvalAI billing works in one paragraph.",
        },
    ],
)

message = completion.choices[0].message
print(message.content)

if message.audio:
    with open("answer.wav", "wb") as output:
        output.write(base64.b64decode(message.audio.data))

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const completion = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "alloy", format: "wav" },
  messages: [
    {
      role: "system",
      content: "You are a friendly voice assistant. Keep answers brief.",
    },
    {
      role: "user",
      content: "Explain how AvalAI billing works in one paragraph.",
    },
  ],
});

const message = completion.choices[0].message;
console.log(message.content);

if (message.audio?.data) {
  await fs.writeFile("answer.wav", Buffer.from(message.audio.data, "base64"));
}

The audio returns as Base64 at choices[0].message.audio.data. To decode it into a playable file straight from the terminal (requires jq), request mp3 output and run a one-time command:

# macOS (zsh)
curl -sS "https://api.avalai.ir/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '{
    "model": "gpt-audio-mini",
    "modalities": ["text", "audio"],
    "audio": { "voice": "alloy", "format": "mp3" },
    "messages": [
      {
        "role": "system",
        "content": "You are a friendly voice assistant. Keep answers brief."
      },
      {
        "role": "user",
        "content": "Explain how AvalAI billing works in one paragraph."
      }
    ]
  }' \
  | jq -r '.choices[0].message.audio.data' \
  | base64 -D > answer.mp3

afplay answer.mp3

# Linux (bash/zsh)
curl -sS "https://api.avalai.ir/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '{
    "model": "gpt-audio-mini",
    "modalities": ["text", "audio"],
    "audio": { "voice": "alloy", "format": "mp3" },
    "messages": [
      {
        "role": "system",
        "content": "You are a friendly voice assistant. Keep answers brief."
      },
      {
        "role": "user",
        "content": "Explain how AvalAI billing works in one paragraph."
      }
    ]
  }' \
  | jq -r '.choices[0].message.audio.data' \
  | base64 --decode >answer.mp3

ffplay -nodisp -autoexit answer.mp3

# Windows (PowerShell)
$response = curl.exe -sS "https://api.avalai.ir/v1/chat/completions" `
  -H "Content-Type: application/json" `
  -H "Authorization: Bearer $env:AVALAI_API_KEY" `
  -d '{
    "model": "gpt-audio-mini",
    "modalities": ["text", "audio"],
    "audio": { "voice": "alloy", "format": "mp3" },
    "messages": [
      {
        "role": "system",
        "content": "You are a friendly voice assistant. Keep answers brief."
      },
      {
        "role": "user",
        "content": "Explain how AvalAI billing works in one paragraph."
      }
    ]
  }' | ConvertFrom-Json

[IO.File]::WriteAllBytes(
  (Join-Path $PWD "answer.mp3"),
  [Convert]::FromBase64String($response.choices[0].message.audio.data)
)

Start-Process .\answer.mp3

These are one-time terminal commands; nothing needs to be added to .zshrc, .bashrc, or the PowerShell profile.

Gemini 3.8 TTS

For new text-to-speech workloads, choose gemini-3.8-flash-tts for creative quality, expressive acting, regional accents, and stable long-form dialogue; choose gemini-3.8-flash-lite-tts for high throughput, low latency, and everyday read-aloud speech. Both accept text input and produce audio output. They are not transcription, audio-input conversation, Live API, or reasoning models.

AvalAI supports these TTS models only through native /v1beta/models methods, /v1/chat/completions, and /v1/audio/speech. Do not send them to /v1/responses, /v1/messages, or legacy /v1/text:synthesize. Use a separate conversational or reasoning model to write the answer before passing final text to TTS.

curl --fail-with-body -sS https://api.avalai.ir/v1/chat/completions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash-tts",
    "messages": [{"role": "user", "content": "Have a wonderful day!"}],
    "modalities": ["audio"],
    "audio": {"voice": "Zephyr", "format": "pcm16"}
  }' \
  --output chat-speech.json

python3 - <<'PYTHON'
import base64
import json
from pathlib import Path

response = json.loads(Path("chat-speech.json").read_text())
data = response["choices"][0]["message"]["audio"]["data"]
Path("speech.pcm").write_bytes(base64.b64decode(data, validate=True))
PYTHON

ffmpeg -f s16le -ar 24000 -ac 1 -i speech.pcm speech.wav

This request reads the supplied transcript; it does not answer questions or accept audio input. Chat returns Base64 at choices[0].message.audio.data, not message.content. The requested pcm16 is headerless signed 16-bit little-endian PCM at 24000 Hz, mono, so save it as raw PCM and convert with explicit input parameters. Do not use a .content fallback or strip DEPRECATED text from Base64. Native unary 3.8 output instead defaults to WAV: follow mimeType before deciding whether a WAV wrapper is needed. For native per-part speechMetadata and migration from gemini-3.1-flash-tts-preview / gemini-2.5-flash-tts / gemini-2.5-pro-tts, see Text to speech.

Pattern 2: Send Audio Input to the Model

Use input_audio when the model should reason over audio directly. Keep the audio file short enough to fit the selected model and request limits.

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("customer_question.wav", "rb") as audio_file:
    audio_b64 = base64.b64encode(audio_file.read()).decode("utf-8")

completion = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "coral", "format": "wav"},
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is the customer asking? Answer briefly.",
                },
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio_b64, "format": "wav"},
                },
            ],
        }
    ],
)

print(completion.choices[0].message.content)

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const audioB64 = (await fs.readFile("customer_question.wav")).toString("base64");

const completion = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "coral", format: "wav" },
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What is the customer asking? Answer briefly." },
        {
          type: "input_audio",
          input_audio: { data: audioB64, format: "wav" },
        },
      ],
    },
  ],
});

console.log(completion.choices[0].message.content);

Pattern 3: Continue a Voice Conversation

For short conversations, store the text transcript in your application and include previous assistant/user turns in messages. Store audio bytes separately; only send audio again when the model needs to inspect it.

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

messages = [
    {"role": "system", "content": "You are a concise voice support assistant."},
    {"role": "user", "content": "Can I use OpenAI SDKs with AvalAI?"},
]

first = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "mp3"},
    messages=messages,
)

messages.append({"role": "assistant", "content": first.choices[0].message.content})
messages.append({"role": "user", "content": "Show me the base URL too."})

second = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "mp3"},
    messages=messages,
)

print(second.choices[0].message.content)

Responses API migration path

Direct input_audio and message.audio belong on Chat Completions for now. When the workflow benefits from Responses, split it into request-based audio steps:

1. Convert user audio to text with /v1/audio/transcriptions.
2. Send the transcript to /v1/responses.
3. Read response.output_text.
4. Create speech with /v1/audio/speech.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("customer_question.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",
        file=audio_file,
        response_format="text",
    )

response = client.responses.create(
    model="gpt-5.6-luna",
    instructions="You are a concise support assistant. Answer for spoken playback.",
    input=transcript,
)

with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="coral",
    input=response.output_text,
) as speech:
    speech.stream_to_file(Path("response.mp3"))

JavaScript

import fs from "node:fs";
import fsp from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const transcript = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: fs.createReadStream("customer_question.wav"),
  response_format: "text",
});

const response = await client.responses.create({
  model: "gpt-5.6-luna",
  instructions: "You are a concise support assistant. Answer for spoken playback.",
  input: transcript,
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "coral",
  input: response.output_text,
});

await fsp.writeFile("response.mp3", Buffer.from(await speech.arrayBuffer()));

Use this migration path when you need Responses features such as tool calls, structured outputs, previous_response_id, or manual Item replay. Keep Chat Completions when direct model audio output is the main feature.

Format and Latency Tips

• Use wav or pcm16 in Chat Completions when low playback latency matters; /v1/audio/speech calls the raw format pcm.
• Use mp3 for compact files and broad compatibility.
• Prefer gpt-audio-mini while developing; move to gpt-audio or gpt-audio-1.5 when quality or context length matters.
• Avoid repeatedly sending the same audio bytes in multi-turn chats; store transcripts and resend only the text context unless the model must inspect audio again.
• For long recordings, prefer /v1/audio/transcriptions and chunking over base64 audio inside Chat Completions.

Troubleshooting

Symptom	Fix
No audio in the response	Include "audio" in modalities and set the audio object with voice and format.
input_audio rejected	Confirm the selected model supports audio input and that the format matches the encoded bytes.
Payload too large	Use /v1/audio/transcriptions for files, compress audio, or split long recordings.
Model ignores previous audio	Store and resend the text transcript; do not assume raw audio remains available across requests.
Need tools or structured output	Use the Responses migration path, then synthesize the final text with /v1/audio/speech.