Audio Processing

AvalAI audio workflows cover text-to-speech, speech-to-text, audio-capable chat, and speech pipelines that combine transcription, reasoning, and generated speech. OpenAI's current audio docs split these into two broad architectures: request-based audio APIs for bounded files or generated speech, and Realtime sessions for live low-latency audio. In AvalAI today, use the request-based endpoints and audio-capable Chat Completions models unless Realtime support is explicitly announced.

Choose The Right Architecture

Goal	AvalAI path	Notes
Convert text to an audio file	/v1/audio/speech	Best for narration, accessibility, generated assistant speech, and cacheable audio.
Transcribe an uploaded audio file	/v1/audio/transcriptions	Best for captions, notes, search, analytics, and post-call processing.
Translate speech to English text	/v1/audio/translations	Use whisper-1 when you need English text output from non-English audio.
Add audio input/output to chat	/v1/chat/completions with an audio-capable model	Use modalities, audio, and input_audio; keep these examples on Chat Completions.
Use Responses for voice assistant reasoning	Transcribe → /v1/responses → TTS	Best when you need tools, state, structured outputs, or reasoning after speech becomes text.
Build live speech-to-speech UX	Realtime-style sessions when available	OpenAI uses Realtime sessions for live voice agents, translation, and live transcription; do not assume AvalAI support until announced.

Current Audio Model Families

AvalAI currently exposes audio models through the model catalog. Use /v1/models or provider pages for exact availability and pricing.

Family	Model examples	Use for
OpenAI TTS	gpt-4o-mini-tts, tts-1, tts-1-hd	General text-to-speech and low-latency generated speech.
Provider TTS	eleven_v3, eleven_multilingual_v2, eleven_flash_v2_5, gemini-2.5-flash-preview-tts, groq.playai-tts	Voice quality, multilingual speech, and provider-specific voices.
Transcription	gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize, whisper-1, scribe_v2	File transcription, diarization, captions, and analytics.
Audio chat	gpt-audio-1.5, gpt-audio-mini	Audio input/output inside Chat Completions.

Text To Speech

Text-to-speech takes three core inputs: model, input, and voice. Use instructions with compatible models to steer tone, pacing, emotion, or delivery style. OpenAI's TTS guidance also recommends clearly disclosing to users when a voice is AI-generated.

For OpenAI-compatible TTS, keep each request under 4,096 input characters, use marin or cedar as first candidates when quality matters, and treat tts-1 / tts-1-hd as compatibility models with fewer style-control options. If the selected route supports speed, start at 1.0 and adjust only after listening tests; the technical range can be wider than what users comfortably understand.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

speech_file = Path("speech.mp3")

audio = client.audio.speech.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input="Welcome to AvalAI. This voice was generated by AI.",
    instructions="Speak clearly in a warm, professional tone.",
)

audio.stream_to_file(speech_file)
print(f"Saved {speech_file}")

JavaScript

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const audio = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "alloy",
  input: "Welcome to AvalAI. This voice was generated by AI.",
  instructions: "Speak clearly in a warm, professional tone.",
});

const buffer = Buffer.from(await audio.arrayBuffer());
await fs.promises.writeFile("speech.mp3", buffer);

Bash

curl https://api.avalai.ir/v1/audio/speech \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini-tts",
    "voice": "alloy",
    "input": "Welcome to AvalAI. This voice was generated by AI.",
    "instructions": "Speak clearly in a warm, professional tone."
  }' \
  --output speech.mp3

Output Formats

Use mp3 for general playback, opus for internet streaming, aac for mobile/video ecosystems, flac for lossless archiving, and wav or pcm when latency and decoding overhead matter.

When you need progressive playback, prefer the SDK streaming helpers or the default binary audio stream. Use stream_format: "sse" only when your selected model and AvalAI route explicitly support event-style speech streaming.

Speech To Text

Use /v1/audio/transcriptions for bounded files. OpenAI's reference guide distinguishes file transcription from live Realtime transcription: file uploads are simpler, while live transcript deltas need a session-oriented Realtime architecture.

Use supported upload formats such as mp3, mp4, mpeg, mpga, m4a, wav, or webm, and keep OpenAI-compatible uploads at or below 25 MB. For longer recordings, split on sentence or turn boundaries so prompt context and downstream summaries remain coherent.

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("meeting.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",
        file=audio_file,
        response_format="text",
        prompt="This is a support meeting about AvalAI billing and API usage.",
    )

print(transcript)

JavaScript

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const transcript = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: fs.createReadStream("meeting.mp3"),
  response_format: "text",
  prompt: "This is a support meeting about AvalAI billing and API usage.",
});

console.log(transcript);

Bash

curl https://api.avalai.ir/v1/audio/transcriptions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -F file="@meeting.mp3" \
  -F model="gpt-4o-transcribe" \
  -F response_format="text" \
  -F prompt="This is a support meeting about AvalAI billing and API usage."

Diarization And Timestamps

• Use gpt-4o-transcribe-diarize when you need speaker-aware transcripts.
• Request diarized_json for speaker segments with start/end metadata.
• Use chunking_strategy="auto" for longer diarization inputs.
• Use whisper-1 with response_format="verbose_json" and timestamp_granularities[] when you need word-level timestamps.
• Use prompt to improve recognition of product names, acronyms, spelling style, or prior segment context. Prompt support is model dependent and is not supported by OpenAI's diarization model.
• Treat stream=true as completed-recording streaming for compatible non-Whisper models. whisper-1 does not support streamed transcription; live microphone or call transcription needs a Realtime route.

Translation

Use /v1/audio/translations when you want English text from audio in another language.

Python

with open("customer_call_fa.m4a", "rb") as audio_file:
    translation = client.audio.translations.create(
        model="whisper-1",
        file=audio_file,
    )

print(translation.text)

JavaScript

const translation = await client.audio.translations.create({
  model: "whisper-1",
  file: fs.createReadStream("customer_call_fa.m4a"),
});

console.log(translation.text);

Bash

curl https://api.avalai.ir/v1/audio/translations \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -F file="@customer_call_fa.m4a" \
  -F model="whisper-1"

Audio In Chat Completions

When a model must receive audio directly or return audio directly, keep the workflow on /v1/chat/completions. Do not replace this with a plain text-only Responses call.

Python

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("question.wav", "rb") as audio_file:
    encoded_audio = base64.b64encode(audio_file.read()).decode("utf-8")

completion = client.chat.completions.create(
    model="gpt-audio-1.5",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "wav"},
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Answer this voice question briefly."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": encoded_audio, "format": "wav"},
                },
            ],
        }
    ],
)

message = completion.choices[0].message
print(message.content)

JavaScript

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const encodedAudio = fs.readFileSync("question.wav").toString("base64");

const completion = await client.chat.completions.create({
  model: "gpt-audio-1.5",
  modalities: ["text", "audio"],
  audio: { voice: "alloy", format: "wav" },
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Answer this voice question briefly." },
        {
          type: "input_audio",
          input_audio: { data: encodedAudio, format: "wav" },
        },
      ],
    },
  ],
});

console.log(completion.choices[0].message.content);

Responses Migration Path

Use Responses after the audio has become text, or before TTS when the assistant needs richer reasoning. This keeps audio handling explicit while letting /v1/responses handle tools, state, structured outputs, and planning.

Python

with open("question.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",
        file=audio_file,
        response_format="text",
    )

answer = client.responses.create(
    model="gpt-5.5",
    instructions="Answer as a concise support assistant.",
    input=f"User said: {transcript}",
)

speech = client.audio.speech.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input=answer.output_text,
)

speech.stream_to_file("answer.mp3")

JavaScript

const transcript = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: fs.createReadStream("question.wav"),
  response_format: "text",
});

const answer = await client.responses.create({
  model: "gpt-5.5",
  instructions: "Answer as a concise support assistant.",
  input: `User said: ${transcript}`,
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "alloy",
  input: answer.output_text,
});

await fs.promises.writeFile("answer.mp3", Buffer.from(await speech.arrayBuffer()));

Realtime Planning Notes

OpenAI's Realtime docs separate live audio into voice-agent, translation, and transcription sessions. Treat these as architecture references for AvalAI until Realtime endpoints and models are listed as available. Do not document /v1/realtime, /v1/realtime/translations, SIP, or Realtime client-secret examples as runnable AvalAI flows unless AvalAI announces the exact route.

Realtime session type	Use when	AvalAI-compatible fallback today
Voice-agent session	The assistant should listen, speak, call tools, and manage turn state.	Transcribe → /v1/responses with tools/state → /v1/audio/speech.
Translation session	The app should continuously translate spoken input.	Use /v1/audio/translations for bounded files, or transcribe short chunks then translate text with /v1/responses.
Transcription session	You need live transcript deltas without assistant speech.	Use /v1/audio/transcriptions with stream=true for completed recordings, or send rolling file chunks from your server.

Choose the transport by where audio is captured: WebRTC for browser/mobile media, WebSockets for server-side media pipelines, and SIP for telephony. Until AvalAI exposes those transports, keep browsers and phones connected to your own backend and call AvalAI with server-held API keys.

For live systems, plan for:

• short-lived client credentials for browser/mobile clients,
• stable, privacy-preserving safety identifiers for abuse monitoring and user-level enforcement,
• explicit turn detection or voice activity detection,
• interruption handling and partial transcript events,
• latency budgets measured separately from transcript or translation quality,
• logs of transcript, generated audio, tool calls, and user feedback.

If you are migrating from an older OpenAI Realtime beta integration, keep the migration work separate from AvalAI adoption: remove beta-only headers, update event names and session shapes, and re-test the exact model, route, audio format, and failure-recovery path before moving traffic.

Production Checklist

• Disclose AI-generated voices to end users.
• Keep API keys on the server; never expose long-lived keys in browsers or mobile apps.
• Validate uploaded audio type, size, and duration before forwarding to the API.
• Preserve consent and privacy rules for call recording, diarization, and generated voices.
• Use domain prompts for names, product terms, and acronyms.
• Evaluate transcription quality with real accents, languages, background noise, and domain vocabulary.
• Evaluate voice output for pronunciation, tone, latency, and user comprehension.

Related Resources

• Audio API Reference
• Realtime and Live Audio
• Speech to Text
• Text to Speech
• Responses API Reference
• Chat Completions API Reference
• Conversational Audio Example
• Speaker-Aware Meeting Intelligence