Multi-Modal AI Basics Cheat Sheet

Combine text, image, and audio understanding in one model using vision-language models, CLIP-style encoders, and multimodal prompting.

2 PagesBeginnerMar 20, 2026

Prompt a Vision-Language Model

Send an image alongside text to a multimodal chat model and get a grounded answer.

python

import anthropicimport base64client = anthropic.Anthropic()with open("chart.png", "rb") as f:    image_b64 = base64.standard_b64encode(f.read()).decode("utf-8")response = client.messages.create(    model="claude-sonnet-4-5",    max_tokens=500,    messages=[{        "role": "user",        "content": [            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": image_b64}},            {"type": "text", "text": "What trend does this chart show?"},        ],    }],)print(response.content[0].text)

Text-Image Similarity with CLIP

Embed text and images into a shared space and rank images by relevance to a caption.

python

import torchfrom transformers import CLIPModel, CLIPProcessormodel = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")inputs = processor(    text=["a dog on a beach", "a cat on a sofa"],    images=[image1, image2],    return_tensors="pt",    padding=True,)with torch.no_grad():    outputs = model(**inputs)logits_per_image = outputs.logits_per_imageprobs = logits_per_image.softmax(dim=1)

Audio Transcription + LLM Pipeline

Transcribe speech to text, then feed the transcript into a text model for downstream reasoning.

python

import whisperwhisper_model = whisper.load_model("turbo")result = whisper_model.transcribe("meeting.mp3")transcript = result["text"]# now hand the transcript to an LLM for summarizationsummary_prompt = f"Summarize the key decisions in this meeting:\n\n{transcript}"

Types of Multi-Modal Models

Common architecture families and what task each is built for.

Vision-language model (VLM)- e.g. Claude, GPT-4V-style — accepts images + text, generates text
CLIP-style dual encoder- separate image/text encoders trained to align in a shared embedding space
Text-to-image diffusion- e.g. Stable Diffusion — generates images conditioned on a text prompt
Speech-to-text (ASR)- e.g. Whisper — transcribes audio into text for downstream text pipelines
Any-to-any / omni models- single model handling text, image, and audio input and output natively

Pro Tip

When sending images to a VLM, resize to the model's documented optimal resolution before base64-encoding — oversized images burn tokens and cost without improving accuracy, since the model downsamples internally anyway.

Was this cheat sheet helpful?

Model Explainability (SHAP/LIME) Cheat Sheet

Synthetic Data Generation Cheat Sheet