Multimodal AI

Working with images, documents, audio, and generated media

Introduction

Everything so far has been text in, text out. Real products rarely stay that clean: users upload receipts, scan contracts, record calls, and expect diagrams back. Modern models are multimodal, they accept images and documents alongside text, and dedicated models handle audio and image generation. This lesson shows what a single model can do natively (vision, document understanding), what has to be routed to a specialized service (speech, image generation), and how to wire them into one pipeline.

1. What Is Multimodal?

Modalities and their token cost

2. Vision

Sending images to the model

3. Documents

Native PDF understanding

4. Audio

Speech-to-text and back

5. Image Generation

Routing to an image model

6. Combined Pipeline

Chaining modalities together

7. Cost & Pitfalls

Tokens, latency, and PII

1. What Is Multimodal AI?

A multimodal model accepts more than one kind of input in the same request. Claude and Gemini, for example, take interleaved text and images (and PDFs) in a single message. Crucially, not every modality flows through one model: today Claude reads images and documents but does not hear audio or draw pictures, so a complete product often combines a reasoning model with specialized services.

Multimodal Inputs to One Model

Figure 1: Text, images, and documents can share a single request; the model reasons over all of them together

Images cost tokens. An image is billed as tokens based on its dimensions, a large photo can cost well over a thousand tokens. Resize and downsample before sending: most extraction tasks work fine at ~1000-1500px on the long edge, at a fraction of the cost of a full-resolution upload.

2. Vision: Sending Images

You pass an image as a content block with a base64 (or URL) source, alongside a text block that says what to do with it. The model can then describe, classify, compare, extract text from, or reason about the image.

import anthropic
import base64
import httpx

client = anthropic.Anthropic()

# Load an image and base64-encode it
url = "https://example.com/receipt.jpg"
image_data = base64.standard_b64encode(httpx.get(url).content).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/jpeg",
                    "data": image_data,
                },
            },
            {"type": "text", "text": "Describe what this image shows in one sentence."},
        ],
    }],
)
print(message.content[0].text)

# You can also pass a hosted image without downloading it yourself:
#   {"type": "image", "source": {"type": "url", "url": url}}

Describe & classify

Alt text, content moderation, product tagging

Extract text

Read handwriting, signs, screenshots, labels

UI to code

Turn a screenshot or mockup into HTML/JSX

Visual QA

Answer questions about charts, diagrams, photos

3. Document Understanding

Documents are the highest-value multimodal use case in business software: invoices, contracts, forms, statements. Claude accepts a PDF as a document content block and reads both its text layer and its visual layout, so tables and even scanned pages work without a separate OCR step. Pair it with a strict schema and you get reliable structured extraction.

import anthropic
import base64

client = anthropic.Anthropic()

with open("invoice.pdf", "rb") as f:
    pdf_data = base64.standard_b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "document",
                "source": {
                    "type": "base64",
                    "media_type": "application/pdf",
                    "data": pdf_data,
                },
            },
            {"type": "text", "text": (
                "Extract the invoice as JSON with keys: "
                "invoice_number, date, vendor, total, and line_items "
                "(each with description, qty, unit_price). "
                "Return only JSON."
            )},
        ],
    }],
)
print(message.content[0].text)

# Claude reads the PDF natively, both its text layer AND its layout,
# so tables, columns, and scanned pages all work without a separate
# OCR step. Validate the JSON with Pydantic before trusting it.

Structured Document Extraction

Figure 2: Extraction is only trustworthy once the output is validated against a schema, retry on failure

4. Audio: Transcribe, Then Reason

Claude has no native audio input or output. The standard pattern is a two-step pipeline: transcribe speech to text with a dedicated speech-to-text (STT) service, then let Claude do what it is best at, reasoning over the text. For the reverse direction, a text-to-speech (TTS) service turns Claude's output back into audio.

# Claude does not process audio directly. Transcribe first with a
# dedicated speech-to-text service, then reason over the text with Claude.

from openai import OpenAI          # Whisper is a convenient STT option
import anthropic

stt = OpenAI()
claude = anthropic.Anthropic()

# 1. Speech -> text
with open("support_call.mp3", "rb") as audio:
    transcript = stt.audio.transcriptions.create(
        model="whisper-1",
        file=audio,
    ).text

# 2. Text -> analysis (this is where Claude excels)
summary = claude.messages.create(
    model="claude-sonnet-5",
    max_tokens=512,
    messages=[{"role": "user", "content": (
        "Summarize this support call and flag any commitments we made:"
        f"\n\n{transcript}"
    )}],
).content[0].text
print(summary)

# For the reverse direction (text -> speech), use a TTS service such as
# ElevenLabs or OpenAI's audio API; Claude has no native audio output.

Service options: for STT, Whisper, Deepgram, and Google Speech-to-Text are common; for TTS, ElevenLabs and the OpenAI audio API. The reasoning step stays with Claude, the audio models are just adapters on either end.

5. Image Generation

Generating images is another job to route out: Claude cannot draw, so you dispatch that step to an image model (OpenAI's image API, Google Imagen, Stability, and others). The most effective pattern is a division of labor, Claude writes a precise image prompt, the image model renders it, and Claude's vision then reviews the result.

# Claude cannot generate images. When a workflow needs one, route that
# step to an image model, exactly the hybrid routing pattern from the
# "Building AI Applications" lesson.

from openai import OpenAI
import base64

image_client = OpenAI()

result = image_client.images.generate(
    model="gpt-image-1",           # or Google Imagen, Stability, etc.
    prompt="A clean, flat-style icon of a shipping box, green on white",
    size="1024x1024",
)

# gpt-image-1 returns base64 (not a URL); decode and save it
image_bytes = base64.b64decode(result.data[0].b64_json)
with open("box.png", "wb") as f:
    f.write(image_bytes)

# Common pattern: Claude writes the image prompt (it is good at that),
# then hands it to the image model:
#   1. Claude turns a vague request into a detailed image prompt
#   2. The image model renders it
#   3. Claude (vision) reviews the result and requests a revision

6. A Combined Multimodal Pipeline

Real features chain modalities. Consider an expense app: a user photographs a receipt, and the system extracts the data, checks it against policy, and returns a tidy summary. One vision call feeds a reasoning call feeds a rendered output, each step using the model best suited to it.

Receipt-to-Report Pipeline

Figure 3: Each stage uses the right model, vision to read the receipt, reasoning to apply policy, generation for the summary

7. Cost, Latency & Pitfalls

Do

Resize/downsample images before sending, resolution drives token cost
Always validate extracted data against a schema before using it
Cache results, the same receipt should not be re-processed
Ask for structured JSON, not prose, when the output feeds code

Watch out for

Blurry or low-res images produce confidently wrong text
Images often contain PII (IDs, faces), handle and store them accordingly
Vision + document calls are slower; stream or show progress
Generated images can carry watermarks or licensing terms, check them

Key Takeaways

Multimodal means mixed input in one request - Claude and Gemini take text, images, and PDFs together and reason over all of them.
Vision is a content block - pass an image (base64 or URL) plus a text instruction; use it to describe, extract, classify, or convert UI to code.
Document understanding is native - Claude reads a PDF's text and layout, no separate OCR; pair it with a schema for reliable extraction.
Audio is routed out - transcribe with an STT service, reason with Claude, and use TTS for spoken output.
Image generation is routed out - Claude writes the prompt, an image model renders it, Claude's vision reviews it.
Chain modalities per stage - use the right model at each step of a pipeline (extract, analyze, generate).
Mind cost, latency, and PII - resize images, validate outputs, cache, and treat uploaded media as sensitive data.

Back to "Agent Design Patterns"

Software Engineering in AI EraLesson 14

Next: "Securing AI Applications"