Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What are multimodal models? Vision language models explained

Multimodal models read images, audio or video next to text. How vision language models turn pixels into tokens, what an image costs, with a Python demo.

What are multimodal models? Vision language models explained

A multimodal model is a model that takes in, and sometimes produces, more than one kind of data: text plus images, audio or video. The most common kind today is the vision language model (VLM), which cuts an image into small patches, turns each patch into a vector the language model can read like a token, and then answers in text. Once the image is tokens, everything you know about tokens and context windows applies to it, including the bill.

How do multimodal models see images?

The trick that made this work came from the Vision Transformer (ViT) paper, “An Image is Worth 16x16 Words” (Dosovitskiy et al., October 2020). It showed that “a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.” A 224 by 224 image cut into 16 by 16 pixel patches gives a 14 by 14 grid, which is 196 patches. Each patch is flattened, multiplied by a learned matrix into a vector, given a position, and fed to a transformer exactly like a sequence of word embeddings.

A vision encoder on its own only produces vectors. To make those vectors mean something in language, OpenAI’s CLIP (February 2021) trained an image encoder and a text encoder together on 400 million image and caption pairs from the internet, with one task: predict which caption goes with which image. The result is an image encoder whose output already lines up with text. CLIP matched the original ResNet-50’s ImageNet accuracy zero-shot, without training on any ImageNet examples.

The last step is connecting that encoder to a language model. LLaVA (Liu et al., April 2023) is the clearest published example of the recipe most open VLMs still follow:

  1. Take a pretrained CLIP encoder (ViT-L/14) and a pretrained language model (Vicuna).
  2. Add one trainable projection matrix W that maps each image feature into the language model’s word-embedding space: Hv = W * Zv.
  3. Stage 1: freeze both the encoder and the LLM, and train only W on 595K image and caption pairs. The paper describes this as training “a compatible visual tokenizer for the frozen LLM.”
  4. Stage 2: keep the encoder frozen and fine-tune W plus the LLM on 158K instruction-following examples that text-only GPT-4 wrote from image captions and boxes.

Both stages ran on 8 A100s: 4 hours for pretraining, 10 hours for fine-tuning. The image becomes a few hundred extra tokens at the front of the prompt, and the LLM learns to treat them as words.

Diagram: an image is cut into a grid of patches, each patch goes through a vision encoder and a projection into the same embedding space as text tokens, and the combined sequence of image tokens and text tokens flows into a single language model that writes text.

How do images turn into tokens and cost?

Hosted APIs bill images as input tokens, and some document the formula. Anthropic’s vision docs say Claude processes images in 28 by 28 pixel patches, one visual token each, so an image costs ceil(width / 28) × ceil(height / 28) tokens. Images above a model’s limit are downscaled first: 1,568 px on the long edge and 1,568 visual tokens on the standard tier, 2,576 px and 4,784 tokens on the high-resolution tier (Claude 4.7 and later, checked 2026-10-31).

This script does the patch arithmetic for ViT and for Claude, and checks it against Anthropic’s published table.

# vision_tokens.py: how an image becomes tokens. Python 3, stdlib only.
from math import ceil

def patches(w, h, p):
    """Number of p x p patches covering a w x h image (partial patches count)."""
    return ceil(w / p) * ceil(h / p)

# 1. ViT (Dosovitskiy et al., 2020): 224 x 224 image, 16 x 16 patches.
print("ViT-B/16, 224x224:", patches(224, 224, 16), "patch tokens")

# 2. Claude's documented rule: one visual token per 28 x 28 block.
#    Sizes from Anthropic's table that are not downscaled on either tier.
for w, h in [(200, 200), (1000, 1000), (1092, 1092)]:
    print(f"Claude, {w}x{h}: {patches(w, h, 28):>5} visual tokens")

# 3. A tall screenshot. Under the high-resolution tier limits (2,576 px, 4,784 tokens).
w, h = 1179, 2556
n = patches(w, h, 28)
print(f"Screenshot {w}x{h}: {n} visual tokens, high-resolution tier, no downscale")

# 4. What 1,000 of the 1000x1000 images cost as input, at $1 per 1M tokens.
print(f"1,000 images x 1,296 tokens at $1/M: ${1000 * patches(1000, 1000, 28) / 1e6:.2f}")

Output (Python 3.13.11):

ViT-B/16, 224x224: 196 patch tokens
Claude, 200x200:    64 visual tokens
Claude, 1000x1000:  1296 visual tokens
Claude, 1092x1092:  1521 visual tokens
Screenshot 1179x2556: 3956 visual tokens, high-resolution tier, no downscale
1,000 images x 1,296 tokens at $1/M: $1.30

The three Claude rows match Anthropic’s table exactly (64, 1,296 and 1,521), and the last line matches their worked example: at Claude Haiku 4.5’s $1 per million input tokens, a 1000 by 1000 image costs about $1.30 per thousand images. The screenshot row is the one that surprises people. A single tall phone screenshot is close to 4,000 tokens on a high-resolution model, more than many whole text prompts. If an agent takes a screenshot every step, as computer use does, images dominate the input bill.

Is a multimodal model one model or several?

Both designs exist, and the difference shows up in latency and in what the model can perceive.

Pipelines and adapters. LLaVA is an adapter design: a separate pretrained encoder bolted onto a language model through a projection. A voice assistant built as a pipeline goes further and chains three models: speech to text, an LLM, then text to speech. Each hop loses information. A transcript has no tone of voice, and a caption has no layout.

Natively multimodal. For GPT-4o (May 2024), OpenAI wrote that it “trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.” The same post gives the old Voice Mode’s average latency as 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. Gemini 1.5 took video and audio into a context of up to a million tokens. Natively multimodal models can keep information a pipeline drops, at the price of a much harder training run.

For a builder, the distinction matters less than the interface: in every major API today you put an image block in the message and the model answers in text, and some models can also output images or audio.

Why do multimodal models matter to a builder?

Most enterprise data is not clean text. It is scanned PDFs, invoices, forms, charts in slide decks, whiteboard photos and screenshots of dashboards. Before VLMs, each of those needed an OCR pipeline plus layout rules that broke whenever a vendor changed a template. A vision model reads the page as a page: tables, checkboxes, stamps and handwriting in one call.

From a data-platform seat, the line I would circle is document extraction. A VLM with structured output turns a folder of PDFs into rows, and it replaces a stack of brittle per-template parsers. Screenshots are the other big use: agents that operate a browser or a desktop work from pixels.

Where do multimodal models break?

Small text and dense layouts. Downscaling is silent. A 4K screenshot sent to a standard-tier model is resized to fit 1,568 px, and small text becomes unreadable. Crop to the region you care about rather than sending the whole screen.

Counting and spatial precision. Anthropic’s own limitations list says Claude’s counts are approximate, “especially with large numbers of small objects”, and that coordinate and localization outputs are approximate and should be verified. Images under 200 pixels, rotated or low quality are where it says hallucinations are more likely.

Hallucinated details. A VLM will describe a plausible chart value it cannot actually read. For extraction, make the model quote what it sees and validate numbers against totals or a second pass.

Prompt injection through pixels. Text inside an image is still text the model reads. A screenshot of a web page can carry instructions, which is the same failure I described in AI browsers and prompt injection.

Cost. As the script shows, a high-resolution image can cost more than the whole text prompt. Resize before you send.

What I would do

For document work, I would resize every image to the smallest size where the text stays legible, crop to the region that matters, and log visual tokens per request so the bill is visible. I would ask for structured output with the source text quoted next to each extracted field, then check totals and dates in code. For agents driving a screen, I would budget screenshots like any other token cost and reuse a cropped region when only part of the screen changed. And I would treat any text the model reads from an image as untrusted input, the same as a web page.

FAQ

What is the difference between a multimodal model and an LLM?

An LLM in the narrow sense reads and writes text tokens. A multimodal model also takes images, audio or video, usually by converting them into vectors the language model reads like tokens. Most frontier LLMs today are multimodal.

What is a vision language model?

A model that takes images and text in and writes text out. The usual design is a vision encoder (often a ViT trained CLIP-style), a projection into the language model’s embedding space, and a language model that reads image and text tokens together.

How many tokens is an image?

It depends on the model and the resolution. On Claude it is about one token per 28 by 28 pixels, so a 1000 by 1000 image is 1,296 tokens. Other providers use different patch sizes and tiling rules, so check their docs or count with their token-counting endpoint.

Can multimodal models read PDFs?

Yes. Anthropic’s PDF support docs, for example, say each page is converted into an image and the page’s extracted text is provided alongside it, so the model reads tables and charts that plain text extraction loses. You pay for both the text and the image tokens.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.