Streaming API Responses

Learn how to stream model responses from the AvalAI API using server-sent events.

Adapted from OpenAI's official Streaming API responses, function calling, and Structured Outputs documentation, with AvalAI endpoint, API key, and route-availability notes.

Introduction

By default, when you make a request to the AvalAI API, the model generates its entire output before sending it back in a single HTTP response. When generating long outputs, waiting for the complete response can take time. Streaming responses lets you start receiving and processing the beginning of the model's output while it continues generating the full response. This is particularly useful for creating more interactive and responsive applications.

This guide focuses on HTTP streaming with stream=true over server-sent events (SSE). OpenAI also documents a persistent WebSocket transport for incremental inputs and previous_response_id state, but treat that as a separate pattern in AvalAI: use it only when AvalAI exposes a WebSocket-compatible route for your application. See Responses WebSocket Mode for the persistent-socket variant.

Enable Streaming

To start streaming responses, set stream=True (or the equivalent in your language's SDK) in your request to the relevant AvalAI API endpoint, such as the Chat Completions API or the Responses API.

Here's an example using the Responses API:

Bash

curl https://api.avalai.ir/v1/responses \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "gpt-5.5",
    "input": [
      {"role": "user", "content": "Say '\''double bubble bath'\'' ten times fast."}
    ],
    "stream": true
  }' \
  --no-buffer

Python

from openai import OpenAI  # Use the standard OpenAI Python library
import os

# Configure the client to use AvalAI endpoint and API key
client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

try:
    stream = client.responses.create(
        model="gpt-5.5",
        input=[
            {
                "role": "user",
                "content": "Say 'double bubble bath' ten times fast.",
            },
        ],
        stream=True,
    )

    print("Streaming response:")
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
        elif event.type == "response.refusal.delta":
            print(event.delta, end="", flush=True)
        elif event.type == "response.completed":
            print("\nStream complete.")
        elif event.type == "response.failed":
            raise RuntimeError(event.response.error)
        elif event.type == "error":
            raise RuntimeError(event)

except Exception as e:
    print(f"An API error occurred: {e}")

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

async function streamResponse() {
  try {
    const stream = await client.responses.create({
      model: "gpt-5.5",
      input: [
        {
          role: "user",
          content: "Say 'double bubble bath' ten times fast.",
        },
      ],
      stream: true,
    });

    console.log("Streaming response:");
    for await (const event of stream) {
      if (event.type === "response.output_text.delta") {
        process.stdout.write(event.delta);
      } else if (event.type === "response.refusal.delta") {
        process.stdout.write(event.delta);
      } else if (event.type === "response.completed") {
        process.stdout.write("\nStream complete.\n");
      } else if (event.type === "response.failed") {
        throw new Error(JSON.stringify(event.response.error));
      } else if (event.type === "error") {
        throw new Error(JSON.stringify(event));
      }
    }
  } catch (error) {
    console.error("An API error occurred:", error);
  }
}

streamResponse();

Go

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"os"
	"strings"
)

func main() {
	apiKey := os.Getenv("AVALAI_API_KEY")
	if apiKey == "" {
		fmt.Println("AVALAI_API_KEY is not set")
		return
	}

	body := `{
		"model": "gpt-5.5",
		"input": "Say 'double bubble bath' ten times fast.",
		"stream": true
	}`

	ctx := context.Background()
	req, err := http.NewRequestWithContext(
		ctx,
		http.MethodPost,
		"https://api.avalai.ir/v1/responses",
		strings.NewReader(body),
	)
	if err != nil {
		fmt.Printf("Request creation error: %v\n", err)
		return
	}
	req.Header.Set("Authorization", "Bearer "+apiKey)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Accept", "text/event-stream")

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		fmt.Printf("Stream request error: %v\n", err)
		return
	}
	defer resp.Body.Close()

	fmt.Println("Streaming response:")
	if _, err := io.Copy(os.Stdout, resp.Body); err != nil {
		fmt.Printf("\nStream read error: %v\n", err)
	}
}

PHP

<?php
require 'vendor/autoload.php';

$apiKey = getenv('AVALAI_API_KEY');

if (!$apiKey) {
    die("AvalAI API key not found. Please set the AVALAI_API_KEY environment variable.");
}

$customBaseUrl = 'https://api.avalai.ir/v1';

$client = OpenAI::factory()
    ->withApiKey($apiKey)
    ->withBaseUri($customBaseUrl)
    ->make();

try {
    $stream = $client->responses()->createStreamed([
        'model' => 'gpt-5.5',
        'input' => [
            ['role' => 'user', 'content' => "Say 'double bubble bath' ten times fast."],
        ],
        // 'stream' => true, // Often implied by createStreamed method
    ]);

    echo "Streaming response:\n";
    foreach ($stream as $event) {
        // Process each event as it arrives
        // The structure of $event depends on the specific PHP client library
        // Example: Accessing data if it's an object with a toArray method or public properties
        if (method_exists($event, 'toArray')) {
             print_r($event->toArray());
        } else {
             var_dump($event);
        }
        echo "\n---\n";
    }

} catch (Exception $e) {
    echo "An API error occurred: " . $e->getMessage() . "\n";
}

Responses vs. Chat Streaming

If you are migrating an existing /v1/chat/completions stream, update the consumer before swapping endpoints:

Chat Completions stream	Responses stream	Migration note
Each chunk has choices[0].delta.content	Text arrives as response.output_text.delta events	Append only the event.delta value to the visible text buffer.
Tool arguments may arrive inside chunk deltas	Tool arguments arrive as response.function_call_arguments.delta and finish with response.function_call_arguments.done	Buffer arguments and parse/execute only after the done event.
Completion metadata is spread across final chunks	response.completed carries the final response object	Read usage, status, and final output from the completed event.
Errors may arrive as transport errors or stream chunks	Responses can emit response.failed or error	Stop the stream and apply your normal retry/error policy.

For browser or proxy deployments, also disable response buffering where your stack allows it. Flush each text delta to the UI, but keep an internal buffer so you can reconcile with response.output_text.done or the final response.completed payload.

Read the Responses

The AvalAI API uses semantic server-sent events (SSE) for streaming. Each event is typed with a predefined schema, allowing you to listen for and handle specific event types relevant to your application.

You can identify individual events using the type property of the event object (or the class type if using an SDK like the Python or Node.js libraries).

Some key lifecycle events are emitted only once per stream, while others are emitted multiple times as the response is generated. For text streams, start with these event families:

Event	How to use it
response.created / response.in_progress	Initialize request state and show that generation has started.
response.output_item.added / response.output_item.done	Track typed output items such as messages, tool calls, or reasoning items.
response.content_part.added / response.content_part.done	Track content parts inside a message item.
response.output_text.delta	Append only this delta value to the user-visible text buffer.
response.output_text.done	Replace or verify the buffered text with the final text for that content part.
response.output_text.annotation.added	Capture citations or other annotations and render them after text offsets are stable.
response.refusal.delta / response.refusal.done	Stream refusal text separately from normal answer text and treat the final refusal as a safe terminal answer.
response.function_call_arguments.delta / response.function_call_arguments.done	Buffer tool arguments, then parse and execute only after the done event.
response.file_search_call.in_progress / response.file_search_call.searching / response.file_search_call.completed	Show retrieval status separately from final answer text when hosted file search is enabled for the route.
response.code_interpreter_call.in_progress / response.code_interpreter_call_code.delta / response.code_interpreter_call.completed	Display interpreter progress, stream generated code to an internal panel, and expose outputs only after the call completes.
response.completed	Read the final response object, usage, status, and complete output. If the final response status is incomplete, inspect incomplete_details before using partial output.
response.failed / error	Stop the stream for both cases. response.failed is tied to a response object; error can be a stream-level failure without a completed response.

For a full list of event types and their schemas, particularly when dealing with more complex scenarios like tool calls, refer to the Responses API Reference (Streaming section).

When building a stream consumer:

• Branch on event.type; do not assume every event carries text.
• Append only response.output_text.delta values to the user-visible buffer; use response.output_text.done to reconcile the final content part.
• Capture response.output_text.annotation.added events for citations, file references, or search annotations, then attach them after the related text has settled.
• Keep refusal output in a separate buffer from normal answer text so your UI can label it clearly and avoid trying to parse it as structured output.
• Treat response.completed as the final source for usage and completion metadata.
• Treat a final response with status: "incomplete" as a partial result; check incomplete_details.reason such as max_output_tokens or content_filter, then retry with a narrower prompt or larger output budget when appropriate.
• Stop or retry on response.failed or error instead of continuing to read a broken stream; log them differently because only response.failed is associated with a response object.
• For streamed tool calls, collect arguments until response.function_call_arguments.done before executing your function.
• For hosted tool events such as file search or code interpreter, update progress indicators without appending those payloads to the visible assistant text. Keep artifacts, retrieved snippets, and generated code in dedicated UI regions.
• Store the final response.id only if the selected AvalAI route supports follow-up state through previous_response_id; otherwise keep conversation state in your own application.

Reconnects, Resume, And Payload Size

For long responses, background jobs, or mobile clients, design your stream consumer to survive disconnects. Normal foreground streams are best treated as interactive, best-effort transports: if the connection drops, keep enough application state to retry the request or retrieve the completed response when the selected route supports retrieval.

OpenAI's background-streaming pattern requires creating the response with both background: true and stream: true, then storing the sequence_number emitted on each event as your cursor. Use the same pattern with AvalAI only when the selected /v1/responses route supports background responses and retrieval streaming:

{
  "model": "gpt-5.5",
  "input": "Write a long report and stream progress.",
  "background": true,
  "stream": true
}

If that support is enabled, persist the latest response ID and event sequence_number, then reconnect with GET /v1/responses/{response_id}?stream=true&starting_after=<sequence_number> so the server can continue after the last processed event. Background streams can have different time-to-first-token behavior than foreground streams, so measure latency for the model and provider you route through AvalAI.

If resume streaming is not available for the route, keep an application-side text buffer and request state so you can restart safely, suppress duplicate deltas, or fall back to retrieving the final response when it completes. Keep stream obfuscation enabled by default; only set include_obfuscation=false for trusted internal network paths where lower bandwidth is more important than payload-size normalization.

Here's a conceptual example of processing text deltas in Python:

# (Continuing from the 'Enable Streaming' Python example)

# ... client setup and stream initiation ...

try:
    stream = client.responses.create(
        model="gpt-5.5",
        input=[{"role": "user", "content": "Tell me a short story."}],
        stream=True,
    )

    full_text = ""
    annotations = []
    print("Streaming story:")
    for event in stream:
        if event.type == "response.output_text.delta":
            text_delta = event.delta
            print(text_delta, end="", flush=True)  # Print delta immediately
            full_text += text_delta
        elif event.type == "response.refusal.delta":
            print(event.delta, end="", flush=True)
        elif event.type == "response.output_text.annotation.added":
            annotations.append(event.annotation)
        elif event.type == "response.completed":
            print("\n--- Stream finished ---")
            # You can access completion reasons, usage stats, etc. from the final event
            # print(event.response)
            if event.response.status == "incomplete":
                print(f"\nIncomplete response: {event.response.incomplete_details}")
        elif event.type == "response.failed":
            print(f"\n--- Response failed: {event.response.error} ---")
            break
        elif event.type == "error":
            print(f"\n--- An error occurred: {event} ---")
            break  # Stop processing on error

    # The 'full_text' variable now holds the complete generated text
    # print("\n\nComplete Story:\n", full_text)

except Exception as e:
    print(f"\nAn API error occurred: {e}")

Advanced Use Cases

Streaming is also essential for more advanced interactions involving structured data or tool usage:

• Streaming Tool Calls: When using Function Calling, you can stream the model's decision to call a function and the arguments it intends to use. This allows your application to react faster. Events like response.output_item.added, response.function_call_arguments.delta, and response.function_call_arguments.done are used.
• Streaming Structured Outputs: If using Structured Outputs with text.format, streaming allows you to receive parts of the structured JSON response as they are generated.
• Hosted Tool Progress: When an AvalAI route exposes hosted tools, OpenAI-compatible streams may include file-search status events and code-interpreter code/progress events. Treat these as operational status, not final answer text, and keep route-dependent fallbacks such as manual RAG or application-side code execution ready.

Refer to the respective guides for detailed examples of streaming in these scenarios.

Moderation Risk

Note that streaming the model's output directly to end-users in a production application makes content moderation more challenging, as partial completions may be harder to evaluate against safety guidelines or policies. If your workflow requests moderation scores for generated content, those scores are available only after the full generated output is complete; they are not included with partial output deltas. For risky user-facing surfaces, consider buffering, post-stream review, or a hybrid approach that streams only after lightweight local checks. See the Safety Guidelines for more information.

Related Resources

• Responses API Reference
• Chat Completions API Reference
• Function Calling Guide
• Structured Outputs Guide
• Error Handling Guide
• Authentication Guide