Modality just means a way something is done — the channel information travels through, like text, image, or sound.

Have you ever asked a friend what their favorite learning modality is: text, image, or audio? Most people can name a few they use every day without thinking about it. A computer program, though, usually only gets one.

Optical Character Recognition (OCR) is a good example. It only ever works with one modality: text. That’s what makes it unimodal — it has exactly one channel to reason from, and nothing else. You, by contrast, are multimodal. You take in sight, sound, language, all of it at once, cross-referencing between them constantly without even noticing. Go you!

So how does OCR actually work, stuck with just that one channel?

When you take a photo of text and copy that text from the image on your iPhone, the program is detecting the shapes of the characters on the page. “Is this a circle? Does this look like a square?” Then it asks, “What does this look like?” and matches that shape against the alphabet and numbers. Simple pattern matching. No understanding of what a word or letter means. So if the handwriting’s messy, a “q” could read as a “9.”

In contrast, a multimodal model asks “what does this mean, given everything else on the page?” not “what shape is this?” It breaks the image into small pieces, converts them into numbers that represent meaning, and plots every letter, word, and concept it’s ever learned onto one giant map. A “9” isn’t a number it found a match for — it’s a point that has to land somewhere on that map, and where it lands depends on everything around it.

Which is where it gets fun. Say you hand it a doctor’s note with some sloppy handwriting. The model recognizes the context — this is a doctor’s note, doctor’s notes usually list medications — and uses that to make an informed guess about the handwriting. It’s not matching letters. It’s reading a mark that’s very likely “amoxicillin,” because of everything else on the page.

And that understanding piece is where multimodal models outperform OCR: seeing the big picture, not just the line in front of them