Multi-Modal AI Basics Cheat Sheet
Combine text, image, and audio understanding in one model using vision-language models, CLIP-style encoders, and multimodal prompting.
2 PagesBeginnerMar 20, 2026
Prompt a Vision-Language Model
Send an image alongside text to a multimodal chat model and get a grounded answer.
python
import anthropicimport base64client = anthropic.Anthropic()with open("chart.png", "rb") as f: image_b64 = base64.standard_b64encode(f.read()).decode("utf-8")response = client.messages.create( model="claude-sonnet-4-5", max_tokens=500, messages=[{ "role": "user", "content": [ {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": image_b64}}, {"type": "text", "text": "What trend does this chart show?"}, ], }],)print(response.content[0].text)
Text-Image Similarity with CLIP
Embed text and images into a shared space and rank images by relevance to a caption.
python
import torchfrom transformers import CLIPModel, CLIPProcessormodel = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")inputs = processor( text=["a dog on a beach", "a cat on a sofa"], images=[image1, image2], return_tensors="pt", padding=True,)with torch.no_grad(): outputs = model(**inputs)logits_per_image = outputs.logits_per_imageprobs = logits_per_image.softmax(dim=1)
Audio Transcription + LLM Pipeline
Transcribe speech to text, then feed the transcript into a text model for downstream reasoning.
python
import whisperwhisper_model = whisper.load_model("turbo")result = whisper_model.transcribe("meeting.mp3")transcript = result["text"]# now hand the transcript to an LLM for summarizationsummary_prompt = f"Summarize the key decisions in this meeting:\n\n{transcript}"
Types of Multi-Modal Models
Common architecture families and what task each is built for.
- Vision-language model (VLM)- e.g. Claude, GPT-4V-style — accepts images + text, generates text
- CLIP-style dual encoder- separate image/text encoders trained to align in a shared embedding space
- Text-to-image diffusion- e.g. Stable Diffusion — generates images conditioned on a text prompt
- Speech-to-text (ASR)- e.g. Whisper — transcribes audio into text for downstream text pipelines
- Any-to-any / omni models- single model handling text, image, and audio input and output natively
Multi-Image Comparison Prompting
Send several images in one request and ask the model to reason across them rather than describe each independently.
python
import anthropic, base64client = anthropic.Anthropic()def to_b64(path): with open(path, "rb") as f: return base64.standard_b64encode(f.read()).decode("utf-8")content = []for i, path in enumerate(["before.png", "after.png"]): content.append({"type": "text", "text": f"Image {i+1}:"}) content.append({ "type": "image", "source": {"type": "base64", "media_type": "image/png", "data": to_b64(path)}, })content.append({"type": "text", "text": "List every visual difference between Image 1 and Image 2."})response = client.messages.create( model="claude-sonnet-4-5", max_tokens=800, messages=[{"role": "user", "content": content}],)print(response.content[0].text)
CLIP-Style Contrastive Loss (from scratch)
The symmetric InfoNCE objective that trains dual encoders to align matching text-image pairs and repel mismatched ones.
python
import torchimport torch.nn.functional as Fdef clip_loss(image_embeds, text_embeds, temperature=0.07): image_embeds = F.normalize(image_embeds, dim=-1) text_embeds = F.normalize(text_embeds, dim=-1) logits = image_embeds @ text_embeds.T / temperature # (batch, batch) labels = torch.arange(logits.shape[0], device=logits.device) loss_i2t = F.cross_entropy(logits, labels) loss_t2i = F.cross_entropy(logits.T, labels) return (loss_i2t + loss_t2i) / 2
Multi-Modal RAG Retrieval
Embed images and text into the same CLIP space so a text query can retrieve relevant images from a vector store.
python
import numpy as npfrom transformers import CLIPModel, CLIPProcessormodel = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")# offline: embed and index a corpus of imagesimage_inputs = processor(images=corpus_images, return_tensors="pt")image_embeds = model.get_image_features(**image_inputs).detach().numpy()index.add(image_embeds / np.linalg.norm(image_embeds, axis=1, keepdims=True))# online: embed the text query and search the same indextext_inputs = processor(text=["a red sports car at night"], return_tensors="pt")query_embed = model.get_text_features(**text_inputs).detach().numpy()top_k = index.search(query_embed / np.linalg.norm(query_embed), k=5)
Image-to-Image Diffusion Editing
Condition a diffusion pipeline on an existing image plus a text prompt to guide targeted edits instead of full generation.
python
from diffusers import StableDiffusionImg2ImgPipelineimport torchfrom PIL import Imagepipe = StableDiffusionImg2ImgPipeline.from_pretrained( "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")init_image = Image.open("room.png").convert("RGB").resize((512, 512))result = pipe( prompt="the same room with a modern minimalist sofa", image=init_image, strength=0.55, # lower = closer to original, higher = more creative freedom guidance_scale=7.5,).images[0]result.save("room_edited.png")
Evaluating Multi-Modal Outputs
Metrics and pitfalls specific to grading cross-modal systems, beyond plain text accuracy.
- CLIPScore- cosine similarity between a generated caption's text embedding and the image embedding, reference-free
- FID (Frechet Inception Distance)- measures how close generated image feature distributions are to real ones; lower is better
- Grounding accuracy- fraction of model claims about an image that are verifiably present in it (catches hallucinated objects)
- Modality gap- systematic offset between image and text embedding clusters in dual encoders, even after alignment training
- Hallucination rate (VLM)- percentage of described objects/attributes not actually present in the source image
- Token cost per image- most VLM APIs bill images as a fixed high-resolution tile count regardless of visual complexity
Pro Tip
When sending images to a VLM, resize to the model's documented optimal resolution before base64-encoding — oversized images burn tokens and cost without improving accuracy, since the model downsamples internally anyway.
Was this cheat sheet helpful?
Explore Topics
#MultiModalAIBasics#MultiModalAIBasicsCheatSheet#DataScience#Beginner#Prompt#Vision#Language#Model#MachineLearning#CheatSheet#SkillVeris
