Multimodal AI: Models That See, Hear and Speak

Until recently, AI models were specialists: one for text, one for images, one for audio. But you don't experience the world in separate channels — you see, hear and read all at once. Multimodal AI brings that same versatility to machines. Models like GPT-4V and Gemini can look at a photo and describe it, read a chart, transcribe speech, or answer questions about a video. This lesson explains the surprisingly elegant idea that makes it possible — and it builds directly on the embeddings you met in Lesson 22 and CLIP from Lesson 24.

What is multimodal AI?

A modality is a type of data — text, images, audio, video. A multimodal model can take in and/or produce more than one of them. Instead of you copy-pasting between a vision tool and a chat tool, one model handles a photo and your question about it together, reasoning across both.

The key idea: a shared embedding space

Here's the trick. Throughout this series, we turned things into vectors — words (Lesson 20), documents (Lesson 22), images (Lesson 17). Multimodal models take one more step: they map all modalities into a single shared vector space, so a photo of a dog and the word "dog" land in nearly the same place. Once everything speaks the same "vector language," a model can compare and combine them freely.

See the shared-space idea in code (runnable)

Here's a tiny illustration: imagine text and images already encoded into the same 3-number space. The text "a cat" should sit closer to the cat image's vector than the dog's. (Real models learn these alignments from millions of image–caption pairs; we hard-code them to show the concept.)

import numpy as np

# Pretend these already live in ONE shared space:
text_cat  = np.array([0.9, 0.1, 0.0])   # the words "a cat"
img_cat   = np.array([0.8, 0.2, 0.1])   # a cat photo
img_dog   = np.array([0.1, 0.9, 0.2])   # a dog photo

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print("cat-text vs cat-img:", round(cosine(text_cat, img_cat), 2))  # high
print("cat-text vs dog-img:", round(cosine(text_cat, img_dog), 2))  # low
# The word 'cat' matches the cat IMAGE — across modalities. That's the magic.

That cross-modal match — text finding the right image — is exactly how you search photos by description, or how a model "knows" a picture shows a cat.

Vision-language models

The most common multimodal models today are vision-language models (VLMs) — they take an image plus text and reason about both. They can describe a photo, answer questions about it ("what's wrong in this diagram?"), read text in an image (OCR), interpret charts, and even help debug a screenshot. It's a Transformer (Lesson 19) that accepts image tokens alongside word tokens.

Beyond images: audio and video

• Speech-to-text transcribes audio (e.g. Whisper-style models); text-to-speech generates natural voices.
• Audio understanding can identify sounds, music or tone.
• Video combines vision over time with audio — describing clips, finding moments, answering questions about footage.

How to use a multimodal model

Using one is a lot like a normal LLM call — you just include an image (or audio) alongside your text (conceptual sketch; real SDKs vary):

# Conceptual shape of a multimodal (vision) API call
response = client.chat.create(
    model="a-multimodal-model",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text",  "text": "What's in this image, and is it safe?"},
            {"type": "image", "url":  "https://example.com/photo.jpg"},
        ],
    }],
)
print(response.output_text)

Where multimodal AI shines

The limits

Multimodal models still hallucinate — they can confidently misread an image or miss details. They can struggle with fine text, precise counting, or spatial reasoning, and they inherit the same bias and privacy concerns (now extended to faces and voices). Impressive, but still to be verified.

The unifying idea: turn everything — words, pixels, sound — into vectors in one shared space, and a single model can reason across all of them at once.

Watching a model look at a screenshot of my own code and correctly explain the bug was the moment multimodal AI stopped feeling like a demo and started changing how I work. Realising it was "just" the same embedding trick from earlier lessons — applied to pixels instead of words — is what made it click for me rather than feel like magic. If you have access to a vision model, try handing it a messy photo or a chart; seeing it reason across text and image together is the fastest way to get the idea.

Key takeaways

• Multimodal AI works with more than one type of data — text, images, audio, video — in a single model.
• The core trick is a shared embedding space: all modalities become vectors in one "language."
• Vision-language models describe images, answer questions about them, read charts and text, and more.
• Using them is like a normal LLM call, just with an image or audio included alongside your text.
• They're powerful but still hallucinate and inherit bias/privacy concerns — verify important outputs.

Continue the series: ← Lesson 33: LLM Security  ·  Next: Lesson 35 — Multi-Agent Systems: AI Teams That Work Together →