// AI that can see, hear, speak, and read — not just text
💡 The core idea: Early AI only worked with text. Multimodal AI can handle text + images + audio + video + code — all in one model. You can send a photo and ask "what's wrong with this circuit?" or share a graph and ask "explain this trend." Different input types all get converted to the same "number language" the AI understands.
What multimodal AI can do
Input types → what AI can do with them:
Image + Text → Text
📷 [photo of a skin rash] + "What does this look like?"
→
This appears to be contact dermatitis — a reaction from touching an irritant. The oval shape and sharp borders suggest direct contact rather than systemic cause. A dermatologist should confirm.
Chart Image → Text Analysis
📊 [line chart of stock price] + "What trend do you see?"
→
The chart shows an upward trend from Jan to Mar (+32%), followed by a sharp drop in April (−18%). This V-shaped recovery with a new lower high suggests bearish momentum setting in.
Audio → Text (Transcription + Translation)
🎤 [audio file in Spanish] + "Transcribe and translate"
→
Transcription (Spanish): "Hola, ¿cómo estás hoy?" — Translation (English): "Hello, how are you today?"
Text → Image Generation
💬 "A serene Japanese garden with cherry blossoms at sunset, photorealistic"
→
🖼️ [generated image of cherry blossom garden at golden hour, with lanterns reflected in a koi pond]
Code Screenshot → Working Code
📸 [screenshot of handwritten pseudocode on whiteboard]
→
def binary_search(arr, target): left, right = 0, len(arr)-1 while left <= right: mid = (left + right) // 2 ...
0 / 5 steps
How does an AI "see" an image?
An image is split into small patches (e.g., 16×16 pixel squares). Each patch is converted to a vector (a list of numbers) using a vision encoder (like CLIP or ViT). These patches are then treated exactly like text tokens — fed into the transformer's attention mechanism. The AI "reads" image patches like words.
IMAGES BECOME TOKENSSAME TRANSFORMER PROCESSES ALL