Multimodal AI

What Multimodal AI is

A multimodal model understands more than one medium — not just text, but also images, documents, audio and even video. In 2026 the leading models (GPT-5.6, Claude Opus 4.8, Gemini 3) are multimodal from the ground up, which unlocks powerful business uses: reading a photographed invoice, analyzing a screenshot, transcribing a support call and summarizing it.

This turns manual, tedious processes (typing data from documents, sorting images) into automatic ones — which is why it's one of the highest-ROI areas.

Images (Vision)

You send an image to the model and ask about it in natural language. Uses: analyzing screenshots, quality control, content recognition, accessibility. Example with OpenAI:

from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What is shown in the image? Return a short description."},
        {"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}
        # or base64: {"url": "data:image/jpeg;base64,...."}
    ]}],
)
print(resp.choices[0].message.content)

• Combine with structured output to get JSON (e.g. a list of detected items) instead of free text.
• High resolution costs more tokens — match the size to the need.

Documents & PDF — Document AI

One of the most in-demand uses: extracting structured data from documents (invoices, receipts, forms, contracts). Instead of traditional OCR + brittle rules, you send the document to a multimodal model and ask for JSON.

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024,
    messages=[{"role":"user","content":[
        {"type":"text","text":"Extract from the invoice: vendor, total, VAT, date. Return valid JSON only."},
        {"type":"image","source":{"type":"base64","media_type":"image/jpeg","data": img_b64}}
    ]}],
)
# -> {"vendor":"...","total":1234,"vat":210,"date":"2026-08-01"}

• Require a schema and add validation — a blurry document can cause errors.
• For long documents — split into pages or use the provider's native PDF capability.
• Ready-made template: see Invoice & Finance Bot on the templates page.

Audio & transcription

For audio you use a transcription model (like Whisper) that converts speech to text, then feed the text to an LLM for summary/analysis. A common chain: recording → transcription → summary + action items.

# Step 1: transcription
audio = open("call.mp3", "rb")
transcript = client.audio.transcriptions.create(model="whisper-1", file=audio)

# Step 2: analysis with an LLM
summary = client.chat.completions.create(
    model="gpt-5.6",
    messages=[{"role":"user","content": f"Summarize the call and give action items:\n{transcript.text}"}],
)

Uses: meeting summaries (see the AI Meeting Summary template), support-call analysis, generating transcripts for accessibility.

Tips & cost

• Images cost tokens. High resolution = more expensive. Downscale images that don't need full resolution.
• Always validate. Extraction from a document can get a field wrong — verify (e.g. subtotal+vat=total) before feeding the system.
• Non-English documents. Models read most languages reasonably, but handwriting or a poor scan make it hard — test on a real sample.
• Combine with hallucination prevention: instruct the model to say "unreadable" instead of guessing a field.
• Gemini is especially strong at multimodal and long context — consider it for long documents/video (guide).