Realtime and Live Audio

OpenAI's Realtime docs separate live audio sessions from request-based audio APIs. Use this guide to design live voice, translation, and streaming transcription systems while keeping your AvalAI implementation on supported routes unless Realtime is explicitly enabled for your account.

Adapted from the official OpenAI Realtime and audio overview, WebRTC, WebSocket, and Realtime transcription guides, with AvalAI endpoint, API key, model, and availability notes.

Warning

Live Realtime sessions are not currently listed as supported AvalAI routes in data/models.json. Treat OpenAI Realtime routes and model IDs as architecture references until AvalAI announces the route and the exact model appears in the supported model data.

Warning

Active AvalAI examples should use request-based audio endpoints and audio-capable Chat Completions models unless a Realtime route is announced for the selected account. Check /v1/models and the provider page before documenting or deploying live-session model IDs.

Choose the Right Audio Architecture

Goal	AvalAI path	Why
Generate spoken audio from final text	/v1/audio/speech	Simple, cacheable, and best for narration or assistant playback.
Transcribe uploaded files	/v1/audio/transcriptions	Best for meetings, captions, analytics, and post-call processing.
Reason over speech with tools	Transcribe → /v1/responses → TTS	Keeps Responses features such as tools, state, structured outputs, and reasoning.
One-call audio chat	/v1/chat/completions with gpt-audio-*	Good for short conversations with direct audio input/output.
Live browser or phone conversation	Realtime architecture, if enabled	Needed for barge-in, low first-audio latency, live events, and continuous turns.
Live transcription only	Realtime transcription, if enabled	Streams transcript deltas before the full utterance is complete.
Live speech translation	Realtime translation, if enabled	Uses a dedicated translation session that streams translated audio/text continuously.

Current AvalAI Audio Model Gate

Before publishing a runnable audio or Realtime example, verify support against data/models.json or the live /v1/models response. The current supported set is request-oriented: audio chat models such as gpt-audio, gpt-audio-1.5, and gpt-audio-mini; transcription models such as gpt-4o-transcribe, gpt-4o-mini-transcribe, and gpt-4o-transcribe-diarize; and speech models such as gpt-4o-mini-tts, tts-1, and tts-1-hd.

Use OpenAI Realtime docs to design future low-latency systems, but keep runnable AvalAI snippets on supported models unless all of these checks pass:

• The exact Realtime model ID is present in data/models.json or returned by /v1/models.
• The route is announced for AvalAI, including whether it uses WebRTC, WebSocket, SIP, or a server-minted client secret.
• The event names and session fields match the AvalAI route, not just an older OpenAI beta sample.
• A request-based fallback exists for transcription → Responses → TTS.

Supported Fallback: Responses-First Voice Pipeline

Use this production-friendly pattern when live Realtime sessions are not available. It is not as low-latency as a WebRTC session, but it is portable and lets you use Responses tools and state.

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with open("question.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe",
        file=audio_file,
        response_format="text",
    )

answer = client.responses.create(
    model="gpt-5.5",
    instructions="Answer as a concise voice assistant for AvalAI developers.",
    input=transcript,
)

speech_path = Path("answer.mp3")
speech = client.audio.speech.create(
    model="gpt-4o-mini-tts",
    voice="alloy",
    input=answer.output_text,
)
speech.stream_to_file(speech_path)

print(answer.output_text)
print(f"Saved {speech_path}")

JavaScript

import fs from "node:fs/promises";
import { createReadStream } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const transcript = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: createReadStream("question.mp3"),
  response_format: "text",
});

const answer = await client.responses.create({
  model: "gpt-5.5",
  instructions: "Answer as a concise voice assistant for AvalAI developers.",
  input: transcript,
});

const speech = await client.audio.speech.create({
  model: "gpt-4o-mini-tts",
  voice: "alloy",
  input: answer.output_text,
});

await fs.writeFile("answer.mp3", Buffer.from(await speech.arrayBuffer()));
console.log(answer.output_text);

Bash

curl https://api.avalai.ir/v1/audio/transcriptions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -F file="@question.mp3" \
  -F model="gpt-4o-transcribe" \
  -F response_format="text"

Realtime Session Concepts

When a live-session route is available, design around these OpenAI Realtime concepts:

• Session type: use a voice-agent session for assistant replies, a translation session for interpreter-style audio, or a transcription session for live text only.
• Connection method: use WebRTC for browser/mobile audio capture and playback; use WebSocket for trusted server-to-server media pipelines; use SIP only for telephony voice-agent designs when the route is explicitly available.
• Ephemeral credentials: browser and mobile clients should receive short-lived client secrets from your server, typically through a server-side POST /v1/realtime/client_secrets flow when the route is available; never expose a long-lived API key in client code.
• Events: live sessions exchange typed client and server events. Your app must handle audio deltas, transcript deltas, tool calls, interruptions, errors, and completion events.
• Safety identifiers: bind a stable, privacy-preserving end-user ID when creating or connecting a session so abuse monitoring can target the user, not the whole app. OpenAI uses the OpenAI-Safety-Identifier header for Realtime; confirm the equivalent AvalAI route behavior before launch.
• Dedicated translation flow: translation sessions are continuous and do not follow the normal assistant turn lifecycle. Do not call response.create for translation sessions; stream audio in and consume translated audio/transcript deltas out.
• Realtime transcription tuning: OpenAI's gpt-realtime-whisper path exposes latency/accuracy delay levels. Pick a target latency first, then test with real microphones, telephony audio, accents, noise, and domain vocabulary instead of clean synthetic clips.
• GA interface shape: newer OpenAI Realtime integrations remove the beta header, create ephemeral client secrets server-side, and use current event names such as response.output_audio.delta, response.output_text.delta, and response.output_audio_transcript.delta. Treat any beta-era sample as migration material, not a new implementation template.

Realtime Endpoint Map

Use these OpenAI GA shapes as architecture references, then verify AvalAI route support and /v1/models before publishing runnable examples:

• Voice-agent sessions: standard conversation sessions connect to /v1/realtime. Browser WebRTC flows can be initialized through /v1/realtime/calls, while browser/mobile ephemeral credential flows use /v1/realtime/client_secrets minted by your server.
• Translation sessions: dedicated translation uses /v1/realtime/translations and OpenAI documents gpt-realtime-translate for this path. Treat that model ID as planning context only until it appears in AvalAI model data and the route is announced.
• Transcription sessions: OpenAI documents gpt-realtime-whisper for streaming transcript deltas. In AvalAI, keep using request-based /v1/audio/transcriptions unless the realtime transcription route is enabled.
• Sideband server controls: WebRTC session creation can return a Location header with a call_id. A trusted server can use that call_id to open a sideband WebSocket to the same session, monitor events, send session.update, and answer tool calls without exposing business logic to the browser.
• Safety identifiers: set OpenAI-Safety-Identifier or the route's AvalAI equivalent from your trusted backend when creating the client secret or realtime call, not from browser code.

Realtime Transcription Events

For live captions without a spoken assistant response, design around a transcription-only session. In OpenAI's current shape, the session update sets session.type to transcription, streams audio with input_audio_buffer.append, and commits audio manually with input_audio_buffer.commit when turn detection is disabled or unavailable. Treat this as an architecture reference until AvalAI announces the matching route.

{
  "type": "session.update",
  "session": {
    "type": "transcription",
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "transcription": {
          "model": "gpt-realtime-whisper",
          "language": "en",
          "delay": "low"
        },
        "turn_detection": null
      }
    }
  }
}

Listen for conversation.item.input_audio_transcription.delta while partial text is arriving and conversation.item.input_audio_transcription.completed when the final transcript for that committed item is ready. Completion events from different user turns can arrive out of order, so reconcile deltas and final text by item_id instead of display order alone.

ws.on("message", (data) => {
  const event = JSON.parse(data);

  if (event.type === "conversation.item.input_audio_transcription.delta") {
    updateCaption(event.item_id, event.delta);
  }

  if (event.type === "conversation.item.input_audio_transcription.completed") {
    finalizeCaption(event.item_id, event.transcript);
  }
});

OpenAI's gpt-realtime-whisper docs expose audio.input.transcription.delay levels of minimal, low, medium, high, and xhigh. Lower values reduce caption latency; higher values give the model more audio context and can improve word error rate. Benchmark those settings with real microphones, telephony audio, accents, noise, code-switching, and domain vocabulary before choosing a production default.

WebRTC, WebSocket, and SIP

Transport	Use it when	Implementation notes
WebRTC	Browser or mobile app captures/plays audio directly	Better media performance; keep session creation on your server and use ephemeral credentials.
WebSocket	Your backend receives raw audio from a call system, worker, or media pipeline	Lowest-level path; you send and receive JSON events plus base64 audio chunks. Also use it as a sideband control channel when a WebRTC/SIP session exposes a call_id.
SIP	A phone number or SIP trunk should connect to a voice agent	Telephony-only architecture; OpenAI's SIP flow uses incoming-call webhooks, accept/reject/hangup call controls, and a follow-up WebSocket monitor. Treat it as architecture reference until AvalAI exposes the exact call routes.
Request-based pipeline	You do not need sub-second live audio	Simpler, easier to log, and supported by standard AvalAI endpoints.

Event Handling Notes

If you use WebSocket for server-to-server Realtime audio, output bytes arrive in incremental audio-delta events. Completion events such as response.output_audio.done and response.done confirm the turn is complete, but they do not carry the audio bytes themselves. Buffer or forward response.output_audio.delta chunks as they arrive, and configure audio format at the session level (session.audio.output.format) or per response (response.audio.output.format) where the route supports it.

For browser playback, prefer WebRTC over WebSocket whenever possible. It is more robust for client-device media under changing network conditions, while WebSocket is better for backend media pipelines that already own raw audio transport.

For dedicated translation sessions, keep appending source audio, including short silences, and consume both source and translated transcript deltas. When the source stream ends on a WebSocket translation session, send session.close and keep reading events until session.closed so final translated audio is not dropped.

When a Realtime route is enabled, attach an event_id to important client events and log it with any returned error event. Unlike normal HTTP responses, Realtime errors can arrive asynchronously, so the event_id is how your client or server links a failed event back to the action that sent it.

const event = {
  event_id: crypto.randomUUID(),
  type: "session.update",
  session: {
    type: "realtime",
    instructions: "Answer briefly and ask before taking account actions.",
  },
};

pendingEvents.set(event.event_id, "session.update");
ws.send(JSON.stringify(event));

ws.on("message", (data) => {
  const serverEvent = JSON.parse(data);

  if (serverEvent.type === "error") {
    const action = pendingEvents.get(serverEvent.event_id) || "unknown event";
    console.error(`Realtime ${action} failed`, serverEvent);
  }
});

Realtime Conversation Event Checklist

When a future AvalAI Realtime route is enabled, build a state machine around server events rather than assuming every turn is a single request/response:

Stage	Watch for	App action
Session opened	session.created, then session.updated after your session.update	Store the session ID, verify echoed settings, and log the model, voice, VAD mode, user ID, and tenant.
User speech starts	input_audio_buffer.speech_started	Stop or duck assistant playback and mark the current user turn as recording.
User speech ends	input_audio_buffer.speech_stopped, input_audio_buffer.committed	Finalize the audio buffer, create a pending transcript item, and decide whether to auto-create a response.
Transcript arrives	conversation.item.input_audio_transcription.delta and .completed	Reconcile partial and final text by item_id; do not rely on arrival order across turns.
Assistant streams	response.output_audio.delta, response.output_text.delta, response.output_audio_transcript.delta	Play or buffer audio deltas immediately, update captions, and keep partial text visually distinct from final text.
Tool call appears	response.function_call_arguments.delta, then response.done with a function_call item	Validate arguments, execute tools on the trusted server, send function_call_output, then trigger the follow-up response.
Turn completes	response.output_audio.done, response.output_text.done, response.done	Persist final transcript, usage, latency, selected tool outputs, and the last durable conversation item ID.

Turn Detection, Interruptions, and Cost Controls

OpenAI's Realtime docs expose several session-level controls that are easy to miss when copying short examples. Treat these as design inputs for future AvalAI Realtime routes:

• Turn detection: server_vad chunks audio based on silence, while semantic_vad waits until the model believes the user has finished an utterance. Tune threshold, prefix_padding_ms, and silence_duration_ms with real production audio. Higher thresholds can help in noisy rooms but may miss quiet speakers.
• Manual commits: transcription sessions using gpt-realtime-whisper should omit turn_detection or set it to null, then explicitly send audio-buffer commit events when an utterance is ready.
• Interruptions: when a user talks over generated speech, track how much of the assistant audio was actually played. Send a conversation.item.truncate event for the unplayed tail so the conversation state does not contain words the user never heard.
• Session updates: use session.update for instructions, audio formats, VAD, and truncation settings. Some properties, such as the selected output voice, may not be changeable after the model has already produced audio.
• Session duration: plan reconnection and state handoff; OpenAI's reference Realtime sessions are bounded rather than indefinitely long.
• Context truncation: long sessions eventually exceed context limits. OpenAI documents automatic oldest-item truncation, retention_ratio, token limits after instructions, and disabling truncation when you prefer an explicit error. Decide which behavior is safer for your product before live launch.
• Prompt caching and rate limits: keep session instructions, tool definitions, output voice, and audio config stable to preserve cacheability. Listen for rate_limits.updated events where available and surface graceful backoff in the UI.
• Noise reduction: if the route supports input audio noise reduction, test it by microphone type and environment. Noise reduction can improve VAD and transcript quality, but should be evaluated with your target language, accent, and domain vocabulary.

Production Checklist

• Start with request-based audio unless live turn-taking materially improves the product.
• Benchmark with real microphones, telephony audio, accents, background noise, and domain vocabulary.
• Configure VAD, interruption truncation, and reconnect behavior before exposing live speech to customers.
• Monitor Realtime rate-limit events and budget long sessions; prompt caching can drop when session configuration changes.
• Keep transcripts for audit/search; store generated audio only when needed for playback or compliance.
• For live transcription, tune latency versus accuracy and do not choose defaults from clean synthetic audio alone.
• For live translation, keep speaker tracks separate when possible and create one translation session per target language or direction.
• Add user-visible disclosure when voices are AI-generated.
• Use least-privilege tools if the voice workflow can take actions; require approval for purchases, account changes, email sends, and data deletion.
• Keep a fallback path from Realtime to transcription → Responses → TTS so the product remains usable if a live session fails.

Related

• Audio API
• Audio Processing
• Speech to Text
• Text to Speech
• Building Conversational Apps with Audio Models
• Responses API