Ask AI

Vision (Image Input)

Learn how to use the vision capabilities of models available through AvalAI to understand images. Vision allows you to provide images as input to a model and generate text responses based on the visual content.

To generate images, refer to the Image Generation guide.

API Endpoints
Image Input Methods
Vision Task Design Checklist
Providing Image File IDs
Providing Image URLs
- Using Chat Completions API
- Using Responses API
Providing Base64 Encoded Images
- Using Chat Completions API
- Using Responses API
Multiple Images
Recommended Vision Models
Gemini-Specific Image Capabilities
Gemini Robotics-ER: Vision for Physical Robotics
Calculating Costs

API Endpoints

AvalAI supports vision capabilities through two API endpoints:

Responses API (v1/responses) - Recommended for new multimodal workflows that need state, tools, structured outputs, or a migration path from text-only Responses calls.
Chat Completions API (v1/chat/completions) - Keep this for existing chat integrations and provider routes that expose vision through Chat Completions only.

Both endpoints can carry image input, but exact support depends on the selected model, provider route, and account configuration. Test the endpoint/model pair you plan to ship.

Recommended Vision Models

Use the latest model families for new multimodal applications:

OpenAI: gpt-5.5 for flagship image understanding, visual reasoning, tool use, and long-context multimodal workflows; gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano for lower-cost tiers.
Google/Gemini: gemini-3.5-flash for current flagship Flash multimodal reasoning, gemini-3.1-pro-preview for advanced Pro-class reasoning, gemini-3.1-flash-lite for high-throughput vision tasks, and gemini-2.5-flash for fast legacy workloads.
Anthropic: claude-opus-4-7, claude-opus-4-6, and claude-sonnet-4-6 for strong image analysis and document/screenshot understanding.
Z.AI: glm-5v-turbo for efficient vision-language tasks, with glm-5.1 for text-first reasoning workflows.
Alibaba/Qwen: Qwen VL families remain useful for OCR, visual question answering, and multilingual image understanding; use current qwen3.7 or qwen3.6 text models when the task does not require image input.

Image Input Methods

You can provide images as input in three ways:

By providing a fully qualified URL to an image file.
By providing an image as a Base64-encoded data URL.
By uploading the image to the Files API and passing its file_id to the Responses API as an input_image.

Use PDF file inputs for document pages, slide decks, forms, and chart-heavy files. PDF inputs are processed as both extracted text and page images by vision-capable models; non-PDF document images and embedded charts are not preserved unless you convert the file to PDF first.

Vision Task Design Checklist

OpenAI's vision guidance treats images as typed model inputs, not a generic file attachment. For AvalAI production workflows, make the task contract explicit before sending the image:

Choose the carrier deliberately: use a public URL for temporary web images, a Base64 data URL for local images, a file_id when file storage and reuse are enabled, and input_file/PDF inputs for page-based documents.
Pin detail when cost or fidelity matters: use low for fast classification or captions, high for small text, charts, UI screenshots, and object details, and original only on routes that support dense spatial analysis.
Label multi-image inputs: tell the model what each image represents, preserve order, and ask comparisons against explicit labels such as "Image A" and "Image B".
Ask for evidence and uncertainty: request visible text, observed objects, or a short confidence note; do not ask the model to infer hidden metadata, exact measurements, or identity unless your application provides that evidence.
Use schemas for downstream work: combine vision with Structured Outputs when a UI, database, or automation expects exact fields.
Redact before upload: remove secrets, faces, account numbers, location metadata, or private screenshots that are not required for the task.

Providing Image File IDs

Use a file ID when file storage is enabled and you want to upload an image once, reference it in multiple requests, or avoid repeatedly sending a large Base64 string. For image inputs, upload with purpose: "vision" and then reference the returned file ID from an input_image item in /v1/responses. If file storage is not enabled for your account or endpoint, use an image URL or Base64 data URL instead.

PythonJavaScriptBash

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

file = client.files.create(
    file=open("screenshot.png", "rb"),
    purpose="vision",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Describe the UI issue in this screenshot.",
                },
                {"type": "input_image", "file_id": file.id},
            ],
        }
    ],
)

print(response.output_text)

javascript

import fs from "fs";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const file = await client.files.create({
  file: fs.createReadStream("screenshot.png"),
  purpose: "vision",
});

const response = await client.responses.create({
  model: "gpt-5.5",
  input: [
    {
      role: "user",
      content: [
        { type: "input_text", text: "Describe the UI issue in this screenshot." },
        { type: "input_image", file_id: file.id },
      ],
    },
  ],
});

console.log(response.output_text);

bash

curl https://api.avalai.ir/v1/files \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -F purpose="vision" \
  -F file="@screenshot.png"

curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '{
    "model": "gpt-5.5",
    "input": [
      {
        "role": "user",
        "content": [
          {"type": "input_text", "text": "Describe the UI issue in this screenshot."},
          {"type": "input_image", "file_id": "file_abc123"}
        ]
      }
    ]
  }'

Providing Image URLs

Using Chat Completions API

Analyze the content of an image using its URL with the chat completions endpoint:

python

# Python Example using Chat Completions with Image URL
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",  # AvalAI API endpoint
)

response = client.chat.completions.create(
    model="gpt-5.5",  # Or another vision-capable model via AvalAI
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
                        # Optional: specify detail level
                        # "detail": "high" # or "low" or "auto" (default)
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

javascript

// JavaScript Example using Chat Completions with Image URL
import { OpenAI } from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY, // Ensure AVALAI_API_KEY is set
  baseURL: "https://api.avalai.ir/v1", // Use AvalAI base URL
});

async function main() {
  const response = await client.chat.completions.create({
    model: "gpt-5.5", // Or another vision-capable model via AvalAI
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "what's in this image?" },
          {
            type: "image_url",
            image_url: {
              url: "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
              // Optional: specify detail level
              // detail: "high" // or "low" or "auto" (default)
            },
          },
        ],
      },
    ],
  });

  console.log(response.choices[0].message.content);
}
main();

bash

# cURL Example using Chat Completions with Image URL
curl https://api.avalai.ir/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
          }
        }
      ]
    }
  ]
}'

go

// Go Example using Chat Completions with Image URL
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/openai/openai-go"
	"github.com/openai/openai-go/option"
)

func main() {
	client := openai.NewClient(
		option.WithAPIKey(os.Getenv("AVALAI_API_KEY")),
		option.WithBaseURL("https://api.avalai.ir/v1"),
	)

	imageURL := "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

	resp, err := client.Chat.Completions.New(
		context.Background(),
		openai.ChatCompletionNewParams{
			Model: openai.F("gpt-5.5"),
			Messages: openai.F([]openai.ChatCompletionMessageParamUnion{
				openai.UserMessage(
					openai.F([]openai.ChatCompletionContentPartUnionParam{
						openai.TextPart("What's in this image?"),
						openai.ImagePart(imageURL),
					}),
				),
			}),
		},
	)

	if err != nil {
		fmt.Printf("Error creating completion: %v\n", err)
		return
	}

	fmt.Println(resp.Choices[0].Message.Content)
}

php

<?php
// PHP Example using Chat Completions with Image URL
require 'vendor/autoload.php';

$apiKey = getenv('AVALAI_API_KEY');
$client = OpenAI::client($apiKey, [
  'base_url' => 'https://api.avalai.ir/v1',
]);

$response = $client->chat()->create([
  'model' => 'gpt-5.5',
  'messages' => [
    [
      'role' => 'user',
      'content' => [
        [
          'type' => 'text',
          'text' => 'What\'s in this image?'
        ],
        [
          'type' => 'image_url',
          'image_url' => [
            'url' => 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg',
            // 'detail' => 'high' // Optional: specify detail level
          ]
        ]
      ]
    ]
  ]
]);

echo $response->choices[0]->message->content;
?>

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

PythonJavaScriptBash

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image."},
                {"type": "input_image", "image_url": "https://example.com/image.png"},
            ],
        }
    ],
)

print(response.output_text)

javascript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const response = await client.responses.create({
  model: "gpt-5.5",
  input: [
    {
      role: "user",
      content: [
        { type: "input_text", text: "Describe this image." },
        { type: "input_image", image_url: "https://example.com/image.png" },
      ],
    },
  ],
});

console.log(response.output_text);

bash

curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '
  {
    "model": "gpt-5.5",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "input_text",
            "text": "Describe this image."
          },
          {
            "type": "input_image",
            "image_url": "https://example.com/image.png"
          }
        ]
      }
    ]
  }'

messages → input
system message → instructions or a developer item
choices[0].message.content → response.output_text
for tools and multimodal output, inspect response.output by item type.

Using Responses API

Analyze the content of an image using its URL with the responses endpoint:

python

# Python Example using AvalAI with Image URL
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",  # AvalAI API endpoint
)

response = client.responses.create(
    model="gpt-5.5",  # Or another vision-capable model via AvalAI
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "What's in this image?"},
                {
                    "type": "input_image",
                    "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
                    # Optional: specify detail level
                    # "detail": "high" # or "low" or "auto" (default)
                },
            ],
        }
    ],
)

# Access the generated text (may vary based on SDK version)
if hasattr(response, "output_text"):
    print(response.output_text)

else:  # Manual parsing if output_text is not available
    text_output = ""
    if response.output and isinstance(response.output, list):
        for item in response.output:
            if (
                item.type == "message"
                and item.content
                and isinstance(item.content, list)
            ):
                for content_part in item.content:
                    if content_part.type == "output_text":
                        text_output += content_part.text + "\n"
    print(text_output.strip())

javascript

// JavaScript Example using AvalAI with Image URL
import { OpenAI } from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY, // Ensure AVALAI_API_KEY is set
  baseURL: "https://api.avalai.ir/v1", // Use AvalAI base URL
});

async function main() {
  const response = await client.responses.create({
    model: "gpt-5.5", // Or another vision-capable model via AvalAI
    input: [
      {
        role: "user",
        content: [
          { type: "input_text", text: "what's in this image?" },
          {
            type: "input_image",
            image_url:
              "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
            // Optional: specify detail level
            // detail: "high" // or "low" or "auto" (default)
          },
        ],
      },
    ],
  });

  // Access the generated text (may vary based on SDK version)
  if (response.output_text) {
    console.log(response.output_text);
  } else {
    // Manual parsing if output_text is not available
    let textOutput = "";
    if (response.output && Array.isArray(response.output)) {
      response.output.forEach((item) => {
        if (
          item.type === "message" &&
          item.content &&
          Array.isArray(item.content)
        ) {
          item.content.forEach((contentPart) => {
            if (contentPart.type === "output_text") {
              textOutput += contentPart.text + "\n";
            }
          });
        }
      });
    }
    console.log(textOutput.trim());
  }
}
main();

bash

# cURL Example using AvalAI with Image URL
curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '{
  "model": "gpt-5.5",
  "input": [
  {
    "role": "user",
    "content": [
    {"type": "input_text", "text": "What is in this image?"},
    {
      "type": "input_image",
      "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
    }
    ]
  }
  ]
}'

go

// Go Example using AvalAI with Image URL
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/openai/openai-go"
)

func main() {
	// Set up the client with AvalAI base URL
	config := openai.DefaultConfig(os.Getenv("AVALAI_API_KEY"))
	config.BaseURL = "https://api.avalai.ir/v1"
	client := openai.NewClientWithConfig(config)

	// Create the response request with image
	imageURL := "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

	resp, err := client.CreateResponse(
		context.Background(),
		openai.ResponseRequest{
			Model: "gpt-5.5",
			Input: []openai.ResponseMessage{
				{
					Role: "user",
					Content: []openai.ResponseContent{
						{
							Type: "input_text",
							Text: "What's in this image?",
						},
						{
							Type: "input_image",
							ImageURL: &openai.ImageURL{
								URL: imageURL,
								// Detail: "high", // Optional: specify detail level
							},
						},
					},
				},
			},
		},
	)

	if err != nil {
		fmt.Printf("Error creating response: %v\n", err)
		return
	}

	// Extract text from the response
	var textOutput string
	for _, item := range resp.Output {
		if item.Type == "message" {
			for _, contentPart := range item.Content {
				if contentPart.Type == "output_text" {
					textOutput += contentPart.Text + "\n"
				}
			}
		}
	}

	fmt.Println(textOutput)
}

php

<?php
// PHP Example using AvalAI with Image URL
require 'vendor/autoload.php';

$apiKey = getenv('AVALAI_API_KEY');
$client = OpenAI::client($apiKey, [
'base_url' => 'https://api.avalai.ir/v1',
]);

$response = $client->responses()->create([
'model' => 'gpt-5.5',
'input' => [
[
'role' => 'user',
'content' => [
[
'type' => 'input_text',
'text' => 'What\'s in this image?'
],
[
'type' => 'input_image',
'image_url' => 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg',
// 'detail' => 'high' // Optional: specify detail level
]
]
]
]
]);

// Access the generated text (may vary based on SDK version)
if (isset($response->output_text)) {
  echo $response->output_text;
} else {
  // Manual parsing if output_text is not available
  $textOutput = '';
  foreach ($response->output as $item) {
    if ($item->type === 'message' && isset($item->content)) {
      foreach ($item->content as $contentPart) {
        if ($contentPart->type === 'output_text') {
          $textOutput .= $contentPart->text . "\n";
        }
      }
    }
  }
  echo trim($textOutput);
}
?>

Providing Base64 Encoded Images

Using Chat Completions API

Analyze the content of a local image by encoding it in Base64 with the chat completions endpoint:

PythonJavaScript

python

# Python Example using Chat Completions with Base64 Image
import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",  # Use AvalAI base URL
)


# Function to encode the image
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")


# Path to your image
image_path = "path/to/your/image.jpg"  # Update this path

# Getting the Base64 string
base64_image = encode_image(image_path)

response = client.chat.completions.create(
    model="gpt-5.5",  # Or another vision-capable model via AvalAI
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}",  # Adjust mime type if needed
                        # Optional: specify detail level
                        # "detail": "high"
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

javascript

// JavaScript Example using Chat Completions with Base64 Image
import fs from "fs";
import path from "path"; // Recommended for handling paths
import { OpenAI } from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY, // Ensure AVALAI_API_KEY is set
  baseURL: "https://api.avalai.ir/v1", // Use AvalAI base URL
});

async function main() {
  const imagePath = "path/to/your/image.jpg"; // Update this path
  const base64Image = fs.readFileSync(imagePath, "base64");
  const mimeType = "image/jpeg"; // Adjust if using PNG, GIF, etc.

  const response = await client.chat.completions.create({
    model: "gpt-5.5", // Or another vision-capable model via AvalAI
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "What's in this image?" },
          {
            type: "image_url",
            image_url: {
              url: `data:${mimeType};base64,${base64Image}`,
              // Optional: specify detail level
              // detail: "high"
            },
          },
        ],
      },
    ],
  });

  console.log(response.choices[0].message.content);
}
main();

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

PythonJavaScriptBash

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image."},
                {"type": "input_image", "image_url": "https://example.com/image.png"},
            ],
        }
    ],
)

print(response.output_text)

javascript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const response = await client.responses.create({
  model: "gpt-5.5",
  input: [
    {
      role: "user",
      content: [
        { type: "input_text", text: "Describe this image." },
        { type: "input_image", image_url: "https://example.com/image.png" },
      ],
    },
  ],
});

console.log(response.output_text);

bash

curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '
  {
    "model": "gpt-5.5",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "input_text",
            "text": "Describe this image."
          },
          {
            "type": "input_image",
            "image_url": "https://example.com/image.png"
          }
        ]
      }
    ]
  }'

messages → input
system message → instructions or a developer item
choices[0].message.content → response.output_text
for tools and multimodal output, inspect response.output by item type.

Using Responses API

Analyze the content of a local image by encoding it in Base64 with the responses endpoint:

PythonJavaScript

python

# Python Example using AvalAI with Base64 Image
import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",  # Use AvalAI base URL
)


# Function to encode the image
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")


# Path to your image
image_path = "path/to/your/image.jpg"  # Update this path

# Getting the Base64 string
base64_image = encode_image(image_path)

response = client.responses.create(
    model="gpt-5.5",  # Or another vision-capable model via AvalAI
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "What's in this image?"},
                {
                    "type": "input_image",
                    "image_url": f"data:image/jpeg;base64,{base64_image}",  # Adjust mime type if needed (e.g., image/png)
                    # Optional: specify detail level
                    # "detail": "high"
                },
            ],
        }
    ],
)

# Access the generated text (may vary based on SDK version)
if hasattr(response, "output_text"):
    print(response.output_text)
else:
    # Manual parsing logic...
    pass

javascript

// JavaScript Example using AvalAI with Base64 Image
import fs from "fs";
import path from "path"; // Recommended for handling paths
import { OpenAI } from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY, // Ensure AVALAI_API_KEY is set
  baseURL: "https://api.avalai.ir/v1", // Use AvalAI base URL
});

async function main() {
  const imagePath = "path/to/your/image.jpg"; // Update this path
  const base64Image = fs.readFileSync(imagePath, "base64");
  const mimeType = "image/jpeg"; // Adjust if using PNG, GIF, etc.

  const response = await client.responses.create({
    model: "gpt-5.5", // Or another vision-capable model via AvalAI
    input: [
      {
        role: "user",
        content: [
          { type: "input_text", text: "What's in this image?" },
          {
            type: "input_image",
            image_url: `data:${mimeType};base64,${base64Image}`,
            // Optional: specify detail level
            // detail: "high"
          },
        ],
      },
    ],
  });

  // Access the generated text (may vary based on SDK version)
  if (response.output_text) {
    console.log(response.output_text);
  } else {
    // Manual parsing logic...
  }
}
main();

Image Input Requirements

Input images must meet the following requirements:

File Types: PNG (.png), JPEG (.jpeg, .jpg), WEBP (.webp), non-animated GIF (.gif)
AvalAI inline size: Up to 20MB per inline image unless the selected provider route documents a different limit. Prefer file_id or a hosted URL for repeated or larger assets.
OpenAI route reference limits: OpenAI's vision reference allows up to 512MB total request payload and up to 1500 image inputs per request. AvalAI/provider routing can apply stricter account, endpoint, or gateway limits, so treat the 20MB inline guidance as the safer default unless your route has been verified.
Resolution and detail: Providers may resize images before tokenization. Use low for quick understanding, high for finer visual detail, original where supported for dense screenshots or spatial tasks, and auto only when variable cost/fidelity is acceptable.
Content Restrictions: No watermarks, logos, or NSFW content. The image must be clear enough for a human to understand; crop or enlarge small text instead of sending noisy full-resolution screenshots.

Specifying Image Detail Level

Use the detail parameter within the input_image object to control processing detail:

"detail": "low": Processes a reduced-resolution view. In OpenAI's reference behavior this is a 512px-style representation, useful for fast, low-cost classification, captioning, or rough scene understanding.
"detail": "high": Uses higher-fidelity image understanding. Use it when text, layout, UI details, charts, or small objects matter.
"detail": "original": Preserves more spatial detail on models/routes that support it, including recent OpenAI gpt-5.4/gpt-5.5 families. Use it for dense screenshots, localization, and computer-use-style analysis.
"detail": "auto": Lets the model/provider choose. On OpenAI gpt-5.5, auto and an omitted detail value behave like original; on some older families, auto may behave closer to high. Pin a detail level when cost or fidelity must be predictable.

For OpenAI routes, the newest reference behavior is:

Detail level	Practical effect
`low`	Most cost-efficient; uses a reduced 512px-style representation for quick scene understanding.
`high`	Standard high-fidelity mode; good default when text, UI, charts, or small objects matter.
`original`	Preserves the most spatial detail on supported models; best for dense screenshots, localization, and computer-use-style analysis.
`auto`	Delegates choice to the model/provider; on `gpt-5.5`, it behaves like `original`.

json

{
  "type": "input_image",
  "image_url": "...",
  "detail": "high"
}

Model Sizing and Tokenization

OpenAI documents two major sizing behaviors: newer GPT-5.5/GPT-5.4-style routes can use patch-based processing with 32px patches and model-specific patch budgets, while GPT-4o/GPT-4.1/o-series-style routes use tile-based processing with 512px tiles for high-detail analysis. AvalAI may route vision requests through OpenAI, Gemini, Anthropic, or other providers, so do not copy OpenAI token math blindly across providers. Treat detail as the portable cost/fidelity control, then verify actual usage in the response metadata, Model Details, and Pricing.

Multiple Image Inputs

You can provide multiple images in a single user turn. In Chat Completions, include several image_url parts in messages[].content; in Responses, include several input_image parts in input[].content. Each image adds token cost and latency, so send only the views needed for the task.

Using Chat Completions API

python

# Example with multiple images using Chat Completions
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What are in these images? Is there any difference?",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "URL_TO_IMAGE_1", "detail": "low"},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "URL_TO_IMAGE_2", "detail": "low"},
                },
            ],
        }
    ],
)
print(response.choices[0].message.content)

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "What are in these images? Is there any difference?",
                },
                {"type": "input_image", "image_url": "URL_TO_IMAGE_1", "detail": "low"},
                {"type": "input_image", "image_url": "URL_TO_IMAGE_2", "detail": "low"},
            ],
        }
    ],
)

print(response.output_text)

messages → input
system message → instructions or a developer item
choices[0].message.content → response.output_text
for tools and multimodal output, inspect response.output by item type.

Using Responses API

python

# Example with multiple images using Responses
response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "What are in these images? Is there any difference?",
                },
                {"type": "input_image", "image_url": "URL_TO_IMAGE_1", "detail": "low"},
                {"type": "input_image", "image_url": "URL_TO_IMAGE_2", "detail": "low"},
            ],
        }
    ],
)
# print(response.output_text)

Limitations

Be aware of the following limitations when using vision capabilities:

Medical Imagery: Not suitable for interpreting specialized medical images (CT, MRI) or for medical diagnosis.
Non-Latin Alphabets: Performance may degrade on images containing text in scripts like Japanese, Korean, etc.
Small Text: Enlarge text for better readability, but avoid cropping crucial context.
Detail level: Use detail: "low" for fast, low-cost understanding; use high, auto, or original where supported when small text, spatial layout, or screenshot fidelity matters.
Rotation: Rotated/upside-down text and images might be misinterpreted.
Visual Elements: Difficulty understanding complex graphs or styling variations (e.g., dashed vs. dotted lines).
Spatial Reasoning: Limited accuracy for tasks requiring precise spatial localization (e.g., chess positions).
Accuracy: May occasionally generate incorrect descriptions or captions.
Image Shape: Struggles with panoramic and fisheye images.
Metadata: Does not process original filenames or EXIF metadata.
Resizing: Images may be resized before analysis based on the selected detail level, potentially losing original dimension information.
Counting: Object counts may be approximate.
CAPTCHAs: Blocked for safety reasons.

Gemini-Specific Image Capabilities

Google's current Gemini models, including gemini-3.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite, gemini-2.5-pro, and gemini-2.5-flash, provide several advanced image understanding capabilities through AvalAI:

Object Detection with Bounding Boxes

Gemini models can detect objects in images and provide their bounding box coordinates. The coordinates are returned relative to the image dimensions, scaled to [0, 1000]. You need to descale these coordinates based on your original image size.

To get bounding boxes, include a clear instruction in your prompt:

Using Chat Completions API:

python

response = client.chat.completions.create(
    model="gemini-3.1-pro-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Detect all prominent items in this image. Return bounding boxes in [ymin, xmin, ymax, xmax] format normalized to 0-1000.",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                    },
                },
            ],
        }
    ],
)
print(response.choices[0].message.content)

Responses API version This version uses `gpt-5.5` because `gemini-3.1-pro-preview` may not be enabled for `/v1/responses` in the current AvalAI model data.

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image."},
                {"type": "input_image", "image_url": "https://example.com/image.png"},
            ],
        }
    ],
)

print(response.output_text)

messages → input
system message → instructions or a developer item
choices[0].message.content → response.output_text
for tools and multimodal output, inspect response.output by item type.

Using Responses API:

python

response = client.responses.create(
    model="gemini-3.1-pro-preview",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Detect all prominent items in this image. Return bounding boxes in [ymin, xmin, ymax, xmax] format normalized to 0-1000.",
                },
                {"type": "input_image", "image_url": "data:image/jpeg;base64,..."},
            ],
        }
    ],
)

To convert the normalized coordinates to pixel coordinates:

Divide each output coordinate by 1000
Multiply x-coordinates by the original image width
Multiply y-coordinates by the original image height

Image Segmentation

Starting with Gemini 2.5 and continuing through Gemini 3.1 models, Gemini can also segment objects and provide masks of their contours. Request segmentation masks with a clear instruction:

Using Chat Completions API:

python

response = client.chat.completions.create(
    model="gemini-3.1-pro-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Give segmentation masks for the wooden items. Output a JSON list where each entry contains the bounding box, mask, and label.",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "data:image/jpeg;base64,..."},
                },
            ],
        }
    ],
)

Responses API version This version uses `gpt-5.5` because `gemini-3.1-pro-preview` may not be enabled for `/v1/responses` in the current AvalAI model data.

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image."},
                {"type": "input_image", "image_url": "https://example.com/image.png"},
            ],
        }
    ],
)

print(response.output_text)

messages → input
system message → instructions or a developer item
choices[0].message.content → response.output_text
for tools and multimodal output, inspect response.output by item type.

Using Responses API:

python

response = client.responses.create(
    model="gemini-3.1-pro-preview",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Give segmentation masks for the wooden items. Output a JSON list where each entry contains the bounding box, mask, and label.",
                },
                {"type": "input_image", "image_url": "data:image/jpeg;base64,..."},
            ],
        }
    ],
)

Important Notes for Gemini Models

Base64 Only: When using Gemini models through AvalAI, images must be provided as base64-encoded strings. URL-based image inputs are not supported for Gemini models.
File Limits: Gemini 3.1, Gemini 2.5 Pro, 2.0 Flash, 1.5 Pro, and 1.5 Flash support a maximum of 3,600 image files per request.
Supported Formats: PNG, JPEG, WEBP, HEIC, and HEIF formats are supported.

Token Calculation for Gemini Models

Token calculation varies by Gemini model:

Gemini 3.1 / 2.5 Flash: 258 tokens if both dimensions ≤ 384 pixels. Larger images are tiled into 768x768 pixel tiles, each costing 258 tokens.

Gemini Robotics-ER: Vision for Physical Robotics

The gemini-robotics-er-1.5-preview model is Google's first vision-language model specifically designed for robotics applications. It excels at understanding physical scenes, spatial relationships, and generating actionable robot commands from visual input.

Key Capabilities

Object Detection with Coordinates: Returns precise 2D points [y, x] and bounding boxes [ymin, xmin, ymax, xmax] in normalized coordinates (0-1000)
Spatial Reasoning: Understands object relationships and scene context
Trajectory Planning: Generates waypoint paths for robot movement
Task Orchestration: Breaks down natural language commands into executable subtasks

Quick Example

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.chat.completions.create(
    model="gemini-robotics-er-1.5-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Point to the red cup"},
                {
                    "type": "image_url",
                    "image_url": {"url": "data:image/jpeg;base64,..."},
                },
            ],
        }
    ],
)

# Response includes normalized coordinates:
# "The red cup is at point [450, 620]"

Responses API version This version uses `gpt-5.5` because `gemini-robotics-er-1.5-preview` may not be enabled for `/v1/responses` in the current AvalAI model data.

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image."},
                {"type": "input_image", "image_url": "https://example.com/image.png"},
            ],
        }
    ],
)

print(response.output_text)

messages → input
system message → instructions or a developer item
choices[0].message.content → response.output_text
for tools and multimodal output, inspect response.output by item type.

When to Use Robotics-ER

Physical robot control requiring precise spatial understanding
Object localization with normalized coordinate outputs
Multi-step task planning for robotic manipulators
Scene understanding for navigation and safety monitoring

For a comprehensive guide on using this model including trajectory planning, spatial reasoning, and advanced robotics applications, see the full tutorial: AI in Robotics with Gemini Robotics-ER.

Calculating Costs

Image inputs are metered as tokens, and cost depends on model family, image dimensions, number of images, and detail.

low usually reduces cost and latency by sending a smaller representation.
high and original preserve more visual information but can consume more input tokens.
Multiple images add together; remove duplicate angles, thumbnails, or screenshots that do not affect the answer.
PDF inputs can be more expensive than plain text because vision-capable routes may process both extracted text and page images.
Provider routing matters: OpenAI, Gemini, Anthropic, and other providers can tokenize images differently.
Patch vs. tile accounting differs by model family. OpenAI documents 32px patch budgets for newer GPT-5-style image tokenization and 512px tile accounting for GPT-4o/GPT-4.1/o-series-style high-detail analysis; AvalAI users should still rely on route-specific usage and pricing.

Before production rollout, test representative images, log input token usage, and compare the result against Model Details, Pricing, and Rate Limits.

Vision (Image Input) ​

Table of Contents ​

API Endpoints ​

Recommended Vision Models ​

Image Input Methods ​

Vision Task Design Checklist ​

Providing Image File IDs ​

Providing Image URLs ​

Using Chat Completions API ​

Using Responses API ​

Providing Base64 Encoded Images ​

Using Chat Completions API ​

Using Responses API ​

Image Input Requirements ​

Specifying Image Detail Level ​

Model Sizing and Tokenization ​

Multiple Image Inputs ​

Using Chat Completions API ​

Using Responses API ​

Limitations ​

Gemini-Specific Image Capabilities ​

Object Detection with Bounding Boxes ​

Image Segmentation ​

Important Notes for Gemini Models ​

Token Calculation for Gemini Models ​

Gemini Robotics-ER: Vision for Physical Robotics ​

Key Capabilities ​

Quick Example ​

When to Use Robotics-ER ​

Calculating Costs ​

Vision (Image Input)

Table of Contents

API Endpoints

Recommended Vision Models

Image Input Methods

Vision Task Design Checklist

Providing Image File IDs

Providing Image URLs

Using Chat Completions API

Using Responses API

Providing Base64 Encoded Images

Using Chat Completions API

Using Responses API

Image Input Requirements

Specifying Image Detail Level

Model Sizing and Tokenization

Multiple Image Inputs

Using Chat Completions API

Using Responses API

Limitations

Gemini-Specific Image Capabilities

Object Detection with Bounding Boxes

Image Segmentation

Important Notes for Gemini Models

Token Calculation for Gemini Models

Gemini Robotics-ER: Vision for Physical Robotics

Key Capabilities

Quick Example

When to Use Robotics-ER

Calculating Costs