Building Conversational Apps with Audio Models

Use AvalAI audio-capable chat models when one model call should accept audio, return spoken audio, or keep a short voice conversation moving. For deeper reasoning, tools, structured output, or state management, use a Responses-first pipeline: transcribe audio, reason with /v1/responses, then synthesize speech.

Related docs: Audio API, Processing audio in Chat Completions, Audio processing guide, Realtime and live audio, Responses vs Chat Completions

Choose the Architecture

Architecture	Use when	AvalAI route
Direct audio Chat Completions	You need input_audio, modalities, or message.audio in one call.	/v1/chat/completions
Responses-first voice assistant	You need tools, reasoning, structured output, or cleaner state handling.	/v1/audio/transcriptions → /v1/responses → /v1/audio/speech
Realtime voice app	You need live, low-latency browser or phone audio.	Start with Realtime and live audio for architecture planning; use AvalAI request-based routes unless Realtime is enabled for your account.

Audio Chat Models

Model	Use for
gpt-audio-1.5	Premium voice quality and longer voice conversations.
gpt-audio	Balanced audio input/output applications.
gpt-audio-mini	Development, support flows, and high-volume voice features.

Example 1: One-Shot Voice Reply

The simplest pattern sends text and asks the model for text plus audio. Store the text transcript for search and analytics; store audio bytes only when you need playback.

Bash

curl https://api.avalai.ir/v1/chat/completions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-audio-mini",
    "modalities": ["text", "audio"],
    "audio": { "voice": "alloy", "format": "mp3" },
    "messages": [
      {
        "role": "system",
        "content": "You are a concise voice assistant for a developer platform."
      },
      {
        "role": "user",
        "content": "Welcome a new developer to AvalAI and mention the OpenAI-compatible base URL."
      }
    ]
  }'

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

completion = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "mp3"},
    messages=[
        {
            "role": "system",
            "content": "You are a concise voice assistant for a developer platform.",
        },
        {
            "role": "user",
            "content": "Welcome a new developer to AvalAI and mention the OpenAI-compatible base URL.",
        },
    ],
)

message = completion.choices[0].message
print(message.content)

if message.audio:
    with open("welcome.mp3", "wb") as audio_file:
        audio_file.write(base64.b64decode(message.audio.data))

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const completion = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "alloy", format: "mp3" },
  messages: [
    {
      role: "system",
      content: "You are a concise voice assistant for a developer platform.",
    },
    {
      role: "user",
      content:
        "Welcome a new developer to AvalAI and mention the OpenAI-compatible base URL.",
    },
  ],
});

const message = completion.choices[0].message;
console.log(message.content);

if (message.audio?.data) {
  await fs.writeFile("welcome.mp3", Buffer.from(message.audio.data, "base64"));
}

Responses version: write the script first, then generate speech.

Use this when you want the response text to use Responses features and you do not need the same model call to return message.audio.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

script = client.responses.create(
    model="gpt-5.5",
    instructions="Write concise spoken copy for a developer platform.",
    input="Welcome a new developer to AvalAI and mention https://api.avalai.ir/v1.",
)

with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input=script.output_text,
) as speech:
    speech.stream_to_file(Path("welcome.mp3"))

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const script = await client.responses.create({
  model: "gpt-5.5",
  instructions: "Write concise spoken copy for a developer platform.",
  input: "Welcome a new developer to AvalAI and mention https://api.avalai.ir/v1.",
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "alloy",
  input: script.output_text,
});

await fs.writeFile("welcome.mp3", Buffer.from(await speech.arrayBuffer()));

Example 2: Multi-Turn Voice Conversation

For short sessions, keep a text transcript in your app and append assistant/user messages. Do not resend generated audio unless the model must analyze the audio itself.

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

messages = [
    {"role": "system", "content": "You are a short-spoken API support assistant."},
    {"role": "user", "content": "Can I use the OpenAI Python SDK with AvalAI?"},
]

first = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "coral", "format": "mp3"},
    messages=messages,
)

assistant_text = first.choices[0].message.content
messages.append({"role": "assistant", "content": assistant_text})
messages.append({"role": "user", "content": "Give me the minimal client setup."})

second = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "coral", "format": "mp3"},
    messages=messages,
)

reply = second.choices[0].message
print(reply.content)

if reply.audio:
    with open("followup.mp3", "wb") as audio_file:
        audio_file.write(base64.b64decode(reply.audio.data))

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const messages = [
  { role: "system", content: "You are a short-spoken API support assistant." },
  { role: "user", content: "Can I use the OpenAI Node SDK with AvalAI?" },
];

const first = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "coral", format: "mp3" },
  messages,
});

messages.push({ role: "assistant", content: first.choices[0].message.content });
messages.push({ role: "user", content: "Give me the minimal client setup." });

const second = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "coral", format: "mp3" },
  messages,
});

const reply = second.choices[0].message;
console.log(reply.content);

if (reply.audio?.data) {
  await fs.writeFile("followup.mp3", Buffer.from(reply.audio.data, "base64"));
}

Responses version: use `previous_response_id` for conversational state.

Responses can keep conversational state with previous_response_id. Send stable instructions on each turn, then synthesize the final text when you need playback.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

instructions = "You are a short-spoken API support assistant."

first = client.responses.create(
    model="gpt-5.5",
    instructions=instructions,
    input="Can I use the OpenAI Python SDK with AvalAI?",
    store=True,
)

second = client.responses.create(
    model="gpt-5.5",
    instructions=instructions,
    input="Give me the minimal client setup.",
    previous_response_id=first.id,
    store=True,
)

with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="coral",
    input=second.output_text,
) as speech:
    speech.stream_to_file(Path("followup.mp3"))

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const instructions = "You are a short-spoken API support assistant.";

const first = await client.responses.create({
  model: "gpt-5.5",
  instructions,
  input: "Can I use the OpenAI Node SDK with AvalAI?",
  store: true,
});

const second = await client.responses.create({
  model: "gpt-5.5",
  instructions,
  input: "Give me the minimal client setup.",
  previous_response_id: first.id,
  store: true,
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "coral",
  input: second.output_text,
});

await fs.writeFile("followup.mp3", Buffer.from(await speech.arrayBuffer()));

Example 3: Audio Input from the User

Use direct input_audio when the model should inspect the sound itself. For long recordings, noisy call audio, analytics, or tool-heavy workflows, transcribe first and use Responses.

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("question.wav", "rb") as audio_file:
    audio_b64 = base64.b64encode(audio_file.read()).decode("utf-8")

completion = client.chat.completions.create(
    model="gpt-audio-mini",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "wav"},
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Answer this spoken question briefly."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio_b64, "format": "wav"},
                },
            ],
        }
    ],
)

print(completion.choices[0].message.content)

JavaScript

import fs from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const audioB64 = (await fs.readFile("question.wav")).toString("base64");

const completion = await client.chat.completions.create({
  model: "gpt-audio-mini",
  modalities: ["text", "audio"],
  audio: { voice: "alloy", format: "wav" },
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Answer this spoken question briefly." },
        {
          type: "input_audio",
          input_audio: { data: audioB64, format: "wav" },
        },
      ],
    },
  ],
});

console.log(completion.choices[0].message.content);

Responses version: transcribe first, then reason over text.

This path is usually easier to debug because each stage has a clear artifact: transcript, model answer, and audio output.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("question.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",
        file=audio_file,
        response_format="text",
    )

answer = client.responses.create(
    model="gpt-5.5",
    instructions="Answer the user's spoken question briefly.",
    input=transcript,
)

with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input=answer.output_text,
) as speech:
    speech.stream_to_file(Path("answer.wav"))

JavaScript

import fs from "node:fs";
import fsp from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const transcript = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: fs.createReadStream("question.wav"),
  response_format: "text",
});

const answer = await client.responses.create({
  model: "gpt-5.5",
  instructions: "Answer the user's spoken question briefly.",
  input: transcript,
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "alloy",
  input: answer.output_text,
});

await fsp.writeFile("answer.wav", Buffer.from(await speech.arrayBuffer()));

Adding Tools

For tool-using voice assistants, the most reliable AvalAI pattern is to let /v1/responses choose tools and write the final answer, then pass response.output_text to /v1/audio/speech. Keep direct audio Chat Completions only when audio I/O is the main requirement and your selected model supports the needed tool behavior.

Production Tips

• Use gpt-audio-mini while prototyping; evaluate gpt-audio or gpt-audio-1.5 when quality or longer context matters.
• Prefer mp3 for saved files and wav or pcm for lower-latency playback.
• Store transcripts and metadata alongside audio files for search, moderation, analytics, and support review.
• Do not hard-code API keys; read AVALAI_API_KEY from the environment.
• Log model, modalities, audio format, latency, and token/audio usage for each turn.
• For files or recordings near upload limits, use /v1/audio/transcriptions with chunking instead of embedding base64 audio in chat messages.

Related Resources

• Audio API reference
• Realtime and live audio
• Chat Completions API reference
• Responses API reference
• Function calling guide
• Streaming responses guide