Images & vision

Vision models take images inside an ordinary chat message: the content becomes an array of parts mixing text and images. Send the image as a base64 data URL — that is the form every vision endpoint on the marketplace accepts, because the bytes travel inside the request instead of requiring the provider to fetch a remote URL.

Finding a vision model#

A model that accepts images lists image in its input modalities:

"architecture": { "modality": "text+image->text", "input_modalities": ["text", "image"], ... }

Filter the model list by input, or check architecture.input_modalities in the catalogue.

Sending an image#

Encode the file, build a data: URL with its media type, and put it in an image_url part:

import base64, os
from openai import OpenAI

client = OpenAI(base_url=BASE_URL, api_key=os.environ["HEIMDALL_API_KEY"])

with open("chart.png", "rb") as f:
    data_url = "data:image/png;base64," + base64.b64encode(f.read()).decode()

completion = client.chat.completions.create(
    model="huggingfacetb/smolvlm2-2.2b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What does this chart show?"},
                {"type": "image_url", "image_url": {"url": data_url}},
            ],
        }
    ],
)
print(completion.choices[0].message.content)

The same request as raw JSON:

{
  "model": "huggingfacetb/smolvlm2-2.2b",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What does this chart show?" },
        {
          "type": "image_url",
          "image_url": { "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUg...", "detail": "auto" }
        }
      ]
    }
  ]
}

A message may carry several images, interleaved with text parts, up to the model's context and the 8 MB request body limit — which is the practical ceiling on image size, since base64 grows the bytes by about a third.

Why data URLs#

Heimdallm forwards image_url.url to the serving endpoint verbatim and never fetches images on a provider's behalf. Marketplace vision endpoints do not fetch remote https: URLs either — a remote URL reaches the model as a string of characters, not an image. Embedding the bytes as a data URL is what guarantees the model actually sees the picture, whichever endpoint serves the request.

Pricing#

Image inputs are tokenized by the serving model and billed as prompt tokens at the endpoint's prompt price, and nothing else. There is no per-input-image price on this marketplace: the only per-picture meter is image_megapixel, which is charged on the pixels an endpoint RENDERS. As everywhere, an undeclared unit is never silently billed (see Pricing & billing).