Structured Outputs

Ensure model responses adhere to a specific JSON structure using text.format in /v1/responses or response_format in /v1/chat/completions.

Introduction

JSON is a standard format for data exchange. AvalAI allows you to enforce JSON output from compatible models, making it easier to integrate AI responses into your applications reliably.

There are two main ways to enforce JSON output:

1. Structured Outputs (json_schema): (Recommended) Ensures the output not only is valid JSON but also conforms precisely to a provided JSON Schema. This prevents issues like missing keys or invalid values.
2. JSON Mode (json_object): Ensures the output is a valid JSON object but doesn't validate against a specific schema. Requires careful prompting to guide the model towards the desired structure.

Benefits of Structured Outputs (json_schema):

• Type Safety: Guarantees schema adherence, reducing the need for validation and retries.
• Explicit Refusals: Safety-based refusals are programmatically detectable via a refusal field instead of potentially malformed JSON.
• Simpler Prompting: Less need for complex prompt instructions just to enforce formatting.

Quick Decision Guide

Need	Use	Why
A typed final answer for UI, storage, or routing	Responses text.format with type: "json_schema"	The model's direct answer is constrained to your schema.
Tool arguments for your application code	Function calling with strict: true	The schema describes a callable action, not the final user-facing answer.
Valid JSON only, schema not available	json_object mode	Useful as a fallback, but you still validate shape in your app.
A safety or policy fallback	A schema that includes nullable fields or an app-level error object	The model needs an allowed shape when user input cannot map cleanly to the task.

In AvalAI, prefer /v1/responses for new structured-output workflows and keep /v1/chat/completions examples for existing integrations that already depend on response_format.

Treat OpenAI's Structured Outputs docs as the API-shape reference, not a blanket guarantee across every provider route in AvalAI. Test the exact model, endpoint, and schema shape you plan to deploy; if json_schema is unavailable, fall back to JSON mode plus application-side validation.

Getting a Structured Response (json_schema)

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

event_schema = {
    "name": "calendar_event",
    "strict": True,
    "schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string", "description": "Name of the event"},
            "date": {"type": "string", "description": "Date of the event"},
            "participants": {
                "type": "array",
                "items": {"type": "string"},
                "description": "List of participants",
            },
        },
        "required": ["name", "date", "participants"],
        "additionalProperties": False,  # Important for strict schema adherence
    },
}

try:
    response = client.chat.completions.create(
        model="gpt-5.5",
        messages=[
            {
                "role": "system",
                "content": "Extract the event information into the specified JSON format.",
            },
            {
                "role": "user",
                "content": "Alice and Bob are going to the science fair on Friday.",
            },
        ],
        response_format={"type": "json_schema", "json_schema": event_schema},
    )

    message = response.choices[0].message
    if getattr(message, "refusal", None):
        print("Model refused:", message.refusal)
    else:
        print(json.loads(message.content))
except Exception as e:
    print(f"An API error occurred: {e}")

Responses API version

Use text.format with type: "json_schema" when the selected model supports /v1/responses. Read the final JSON string from response.output_text, and inspect response.output if you need to distinguish a normal output from a refusal.

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    instructions="Extract the event information into the specified JSON schema.",
    input="Alice and Bob are going to the science fair on Friday.",
    text={
        "format": {
            "type": "json_schema",
            "name": "calendar_event",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "date": {"type": "string"},
                    "participants": {
                        "type": "array",
                        "items": {"type": "string"},
                    },
                },
                "required": ["name", "date", "participants"],
                "additionalProperties": False,
            },
        }
    },
)

print(json.loads(response.output_text))

• Chat Completions response_format → Responses text.format
• messages → input, with durable system/developer guidance in instructions
• choices[0].message.content → response.output_text
• For safety refusals or mixed outputs, inspect response.output by item and content-part type.

(Note: For new structured-output flows, prefer /v1/responses with text.format. Keep Chat Completions examples for existing integrations that already use response_format.)

Parse Before You Trust

Treat the API schema as the first line of defense, not the only one. Before writing to a database, triggering a workflow, or rendering a privileged UI, check response.status, refusal content parts, and application-specific invariants. Then parse and validate the complete JSON object with your own runtime types.

For high-impact workflows, log the schema name/version, model, route, and validation errors. This makes regressions easier to diagnose when a prompt, schema, or provider route changes.

Keep Schemas and Types in Sync

OpenAI recommends using SDK helpers or schema libraries such as Pydantic and Zod so your runtime types do not drift away from the JSON Schema you send to the API. If you keep a hand-written schema, add a small CI check that fails when the schema or the corresponding application type changes without the other.

from pydantic import BaseModel, ConfigDict


class CalendarEvent(BaseModel):
    model_config = ConfigDict(extra="forbid")

    name: str
    date: str
    participants: list[str]


event_schema = {
    "name": "calendar_event",
    "strict": True,
    "schema": CalendarEvent.model_json_schema(),
}

Design Schemas for Model Reliability

A JSON Schema is both an API contract and part of the model prompt. Use clear, domain-specific names so the model can infer intent without extra prose:

• Prefer customer_refund_requested over vague keys such as flag.
• Add short description text for fields whose meaning, units, or allowed values could be ambiguous.
• Use enums for product states, priorities, routes, or categories that downstream code depends on.
• Keep unrelated decisions in separate schemas or subtasks; one giant schema can make mistakes harder to diagnose.
• Run evals against representative user inputs before changing field names, enum values, required fields, or nesting depth.

Schema contract checklist

Use this checklist before shipping a structured-output schema:

• Set strict: true when you need schema adherence, not just valid JSON.
• Add additionalProperties: false on every object.
• List every property in required; represent optional values with a union that includes null, such as {"type": ["string", "null"]}.
• Keep the root schema as an object instead of a top-level anyOf; use nested anyOf only where the selected route supports it.
• Put properties in the order you want humans and downstream logs to read them; structured outputs generally follow schema key order, but your parser should still use field names rather than positional assumptions.
• Use stable, versioned schema names such as support_ticket_v1; create a new version when the shape changes incompatibly so logs, cached schemas, and eval results stay comparable.
• Keep schemas small and stable. First requests for a new schema can have extra latency while the schema is processed and cached.
• Do not put tenant names, user identifiers, secrets, or sensitive business rules in schema names, descriptions, enum values, or $defs. Keep user-specific data in the prompt/input payload and keep schemas generic enough to be safely reused and cached.
• Prefer generated schemas from Pydantic or Zod when possible so application types and API schemas stay aligned.

For function tools, set strict: true explicitly rather than relying on endpoint defaults. Current OpenAI Responses behavior may try to normalize compatible tool schemas into strict mode, while Chat Completions remains non-strict by default. When a schema cannot be made strict, tool metadata may fall back to strict: false; detect that in testing and either simplify the schema or handle best-effort arguments defensively. Also treat schema caching as a performance implementation detail: schema definitions may be processed and cached by the provider, so verify route-specific retention requirements before using sensitive schemas.

Reuse and Recursive Schemas

Use $defs and $ref when the same object shape appears in multiple places, such as a nested UI tree, a workflow step list, or a recursive category hierarchy. OpenAI's Structured Outputs subset supports definitions and recursion, including root recursion with $ref: "#", but the schema must still obey the supported-subset rules: object fields are required, objects need additionalProperties: false, and size/depth limits still apply.

For AvalAI production routes:

• Keep recursive schemas shallow in real usage and enforce application-side limits for depth, array length, and string size.
• Use enums for node types, state values, and action names instead of free-form labels when downstream code depends on them.
• Validate the complete response with your own schema validator before rendering generated UI, executing workflow steps, or writing records.
• Test the exact provider route with a minimal recursive fixture before shipping; if it fails, flatten the schema or use IDs plus parent references.

Handle Refusals and Incomplete Outputs

Structured Outputs can make valid application JSON more reliable, but safety refusals and interrupted generations still need explicit handling:

• Refusals: Check message.refusal in Chat Completions or inspect response.output content parts in Responses before parsing output_text.
• Incomplete Responses: In Responses, check response.status == "incomplete" and response.incomplete_details.reason before parsing. In Chat Completions, inspect finish_reason. If output limits or content filters stop generation early, retry with clearer instructions, a smaller schema, or a higher output-token limit.
• User input mismatch: Tell the model what to do when the input cannot be mapped to the schema, such as returning null fields or an application-level error object.

response = client.responses.create(
    model="gpt-5.5",
    input="Extract a support ticket summary as JSON.",
    text={
        "format": {
            "type": "json_schema",
            "name": "support_ticket",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "summary": {"type": "string"},
                    "priority": {"type": "string", "enum": ["low", "medium", "high"]},
                },
                "required": ["summary", "priority"],
                "additionalProperties": False,
            },
        }
    },
)

if response.status == "incomplete":
    raise RuntimeError(f"Incomplete response: {response.incomplete_details.reason}")

for item in response.output:
    if item.type == "message":
        for content in item.content:
            if content.type == "refusal":
                raise RuntimeError(f"Model refused: {content.refusal}")

print(response.output_text)

Streaming Structured Outputs

Structured outputs can also be streamed through the Responses API. Use streaming when you want to update a UI as fields arrive, but treat each delta as partial text: parse and validate the complete JSON only after the final response is available.

import os
from typing import List

from openai import OpenAI
from pydantic import BaseModel, ConfigDict


class EntitiesModel(BaseModel):
    model_config = ConfigDict(extra="forbid")

    attributes: List[str]
    colors: List[str]
    animals: List[str]


client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

with client.responses.stream(
    model="gpt-5.5",
    instructions="Extract entities from the input text.",
    input="The quick brown fox jumps over the lazy dog with piercing blue eyes.",
    text_format=EntitiesModel,
) as stream:
    refusal_text = ""
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
        elif event.type == "response.refusal.delta":
            refusal_text += event.delta
            print(event.delta, end="", flush=True)
        elif event.type == "response.failed":
            raise RuntimeError(event.response.error)
        elif event.type == "error":
            raise RuntimeError(event.error)

    final_response = stream.get_final_response()
    if final_response.status == "incomplete":
        raise RuntimeError(
            f"Incomplete structured output: {final_response.incomplete_details.reason}"
        )
    if refusal_text:
        raise RuntimeError(f"Model refused: {refusal_text}")

    print("\n\nFinal JSON:")
    print(final_response.output_text)

• Prefer SDK stream helpers when available; they keep the schema, events, and final response in one flow.
• Keep refusal deltas separate from JSON deltas; do not parse a refusal as structured output.
• In raw event handling, response.output_text.delta carries partial text; Python helpers commonly signal completion with response.completed, while JavaScript streams may also expose response.output_text.done as an output-text boundary. Treat the final response object as the source of truth.
• After the stream finishes, inspect final_response.status before parsing so max-output or content-filter interruptions do not become partial JSON bugs.
• Do not trigger downstream actions from partial JSON fields unless your application can tolerate correction or rollback.
• For tool arguments, use the function-calling stream events described in the Function Calling Guide.

Supported Models

Structured Outputs (json_schema) is typically supported by newer models available through AvalAI, such as:

• gpt-5.5 and its snapshots
• gpt-5.4 and its snapshots
• Other advanced models may support structured outputs depending on provider and route. Check the provider page and test the exact /v1/responses or /v1/chat/completions path you plan to use.

Older models might only support the basic json_object mode.

Before adopting a model or provider route in production:

• Run one happy-path request, one incompatible-input request, and one safety-sensitive request so you can verify normal JSON, refusal handling, and incomplete-output handling.
• Confirm the route supports the schema shape you rely on (strict: true, nullable optional fields, nested anyOf, $defs, and enum sizes).
• Record the schema name/version with the response ID and model so future failures can be traced to a prompt, schema, or route change.

When to Use Structured Outputs vs. Function Calling

• Function Calling: Use when you want the model to output JSON specifically to call your application's functions/tools (e.g., query a database, call an external API). See the Function Calling Guide.
• Structured Outputs (text.format in Responses, response_format in Chat Completions): Use when you want the model's direct response to the user to be in a specific JSON format (e.g., for parsing and displaying in a UI, structured data extraction).

JSON Mode (json_object)

For models that do not support json_schema, you can use the simpler JSON mode by setting text.format={"type":"json_object"} in Responses or response_format={"type":"json_object"} in Chat Completions.

Important Considerations for json_object mode:

1. Explicit Prompting: You must instruct the model within your prompt (e.g., system message) to output JSON. Failure to do so might result in invalid output or infinite whitespace generation. The API may error if "JSON" isn't mentioned in the prompt context.
2. No Schema Guarantee: JSON mode only ensures the output is valid JSON; it does not guarantee it matches any specific structure (e.g., required keys, types). You'll need to implement your own validation.
3. Edge Cases: Handle potential incomplete JSON if max_tokens is reached or if content filtering stops the generation mid-object.

# Python Example using AvalAI with json_object mode
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

try:
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant designed to output JSON.",
            },
            {
                "role": "user",
                "content": "Extract the user's name and city: John Doe lives in London.",
            },
        ],
        response_format={"type": "json_object"},
    )
    # ... (add validation and error handling for the response content) ...
    print(response.choices[0].message.content)
except Exception as e:
    print(f"An error occurred: {e}")

Responses API version

Use JSON mode only when schema adherence is unavailable or unnecessary. With Responses, set text.format to json_object and explicitly instruct the model to output JSON.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.4-mini",
    instructions="You are a helpful assistant. Output only valid JSON.",
    input="Extract the user's name and city: John Doe lives in London.",
    text={"format": {"type": "json_object"}},
)

print(response.output_text)

• JSON mode guarantees valid JSON, not schema adherence.
• Prefer json_schema Structured Outputs when the model supports it.
• Detect incomplete JSON when output limits, refusals, or content filters stop generation early.

Best Practices & Tips

• Schema Design: Use clear, descriptive names and descriptions for keys in your JSON schema.
• Handling User Input: Instruct the model on how to respond if user input is irrelevant or cannot be mapped to the schema (e.g., return null values, specific error structure).
• Error Handling: Implement robust error handling for API errors, potential refusal responses (with json_schema), incomplete JSON (especially with json_object or low max_tokens), and validation failures.
• Iteration: Test and refine your prompts and schemas using evaluation data.

Troubleshooting Structured Output Failures

Symptom	Likely cause	What to change
API rejects the schema	Missing additionalProperties: false, missing required entries, unsupported constraints, or a root anyOf	Simplify the schema, list every property in required, use null unions for optional fields, and remove unsupported validation keywords
First request with a schema is slower	The schema is being processed and cached	Keep schemas stable, reuse names, warm important schemas during deploys, and avoid generating one-off schemas per user request
Schema contains sensitive values	Tenant/user details were embedded in schema names, descriptions, enums, or $defs	Move sensitive values into request input, keep schema names generic/versioned, and verify route retention requirements
The model refuses	The request triggered a safety refusal that may not match your schema	Check message.refusal or response.output content parts before parsing output_text, then show a safe fallback UI
Output is incomplete	Token limit, content filter, or transport interruption stopped generation	Check response.status / incomplete_details in Responses or finish_reason in Chat Completions before parsing
JSON mode hangs or returns whitespace	The prompt did not explicitly ask for JSON	Add a system/developer instruction that includes the word JSON, or prefer json_schema Structured Outputs
App types drift from schema	The JSON Schema and runtime model are maintained separately	Generate schemas from Pydantic/Zod or add CI checks that fail when one changes without the other

Related Examples

• Agentic Guardrails with Schema Workflow
• Promptfoo Evals with AvalAI

Supported JSON Schema Features (Subset)

Structured Outputs (json_schema) supports a significant subset of the JSON Schema specification, including:

• Types: string, number, integer, boolean, object, array
• Object Properties: properties, required, additionalProperties: false (required)
• Arrays: items (with a valid sub-schema)
• Enums: enum (limited total values/characters apply)
• Composition: anyOf (nested schemas must also be valid), $defs / $ref (for definitions and recursion)

Key Limitations:

• Root object cannot be anyOf.
• All defined properties within an object in the schema are treated as required. Use {"type": ["string", "null"]} to emulate optionality.
• additionalProperties: false is mandatory for objects.
• Size limits apply: up to 5,000 object properties total, up to 10 nesting levels, and up to 120,000 total characters across property names, definition names, enum values, and const values.
• Enum limits apply: up to 1,000 enum values across the schema; for a single string enum with more than 250 values, the combined enum string length must stay under 15,000 characters.
• Unsupported keywords include composition controls such as allOf, not, dependentRequired, dependentSchemas, if, then, and else. Fine-tuned model routes can have additional unsupported type-specific constraints such as minLength, pattern, minimum, patternProperties, and minItems.

If an unsupported schema feature is used with json_schema, the API will return an error.