Build a Complete Video Intelligence Pipeline in 20 Minutes

Summary: Video is the hardest data type in AI. A single video generates every other data type (audio, text, images, metadata, embeddings), each requiring different processing, different storage, and different retrieval patterns. Teams today stitch together 5+ services to handle this. This tutorial shows how to build the entire pipeline (from raw video to searchable, transcribed, analyzed, queryable output) in one system with Pixeltable.

What We're Building#

By the end of this tutorial, you'll have a video intelligence pipeline that:

• Extracts frames from videos at a configurable frame rate
• Runs object detection on every frame using YOLOX
• Generates descriptions of each frame using GPT-4o Vision
• Extracts and transcribes audio using OpenAI Whisper
• Creates searchable indexes for image similarity, text search, and audio segments
• Queries across modalities: find frames by visual content, by description text, or by what was said in the audio
• Explores alternative strategies: chunked audio search, cross-modal video embeddings via Twelve Labs and Gemini

All of this in one system. No S3 to configure, no Pinecone to sync, no Airflow DAGs to manage, no custom glue code. Just tables, computed columns, and queries.

Why Video Is the Ultimate Test for AI Infrastructure#

A single video generates every other data type:

1. Audio: extract the audio track, chunk it into segments
2. Text: transcribe the audio into searchable text
3. Images: extract keyframes, generating hundreds of images per video
4. Metadata: run object detection, scene classification on those frames
5. Embeddings: embed frames and text into vector space for similarity search
6. LLM outputs: generate descriptions, summaries, analysis

In the traditional approach, that's S3 for storage, FFmpeg for extraction, Whisper API for transcription, YOLOX for detection, an embedding service, Pinecone for vectors, and Airflow to orchestrate it all. Seven systems before you write a single line of application logic.

Let's do it in one.

Prerequisites#

You'll also need a sample video file. Any MP4 will work: a product demo, a meeting recording, a tutorial. If you don't have one handy, you can use a URL to a public video.

Step 1: Create the Video Table#

Start with a table that uses Pixeltable's native Video type. Your video files stay where they are. Pixeltable references them without copying.

Step 2: Extract Frames Automatically#

Create a view with a frame iterator. This declaratively defines "extract one frame per second from every video." When you insert a video, frames are extracted automatically.

Tip: For faster processing on long videos, use keyframes_only=True to extract only I-frames, or num_frames=10 to extract a fixed number of evenly-spaced frames. See the frame extraction cookbook for details.

Step 3: Add Object Detection#

Add a computed column that runs YOLOX object detection on every frame. This runs automatically on all existing and future frames.

The detections column now contains bounding boxes, labels, and confidence scores for every detected object in every frame. Pixeltable handles model loading, batching, and caching automatically.

Step 4: Generate Frame Descriptions with GPT-4o Vision#

Add another computed column that sends each frame to GPT-4o Vision for a natural language description. API calls are parallelized, rate-limited, and cached automatically.

Step 5: Extract Audio and Transcribe#

Back on the videos table, add computed columns that extract audio and transcribe it with Whisper:

The transcript column now contains the full text transcription of each video's audio track, with timestamps. See the audio extraction cookbook for more details.

Alternative: Chunked Audio for Granular Search#

The approach above transcribes the entire audio track as one block. For longer videos, you often want to search within specific segments: "what was said at minute 12?" To enable this, extract the audio, then create a view with the audio splitter to chunk it into segments, transcribe each chunk, and make them individually searchable:

Now you can search within the audio timeline: find the exact 30-second segment where a specific topic was discussed, rather than getting back the entire video transcript. Each chunk retains its start_time_sec and end_time_sec so you can jump to the right moment.

Step 6: Make Everything Searchable#

Now the powerful part: add embedding indexes so you can search across modalities. One line per index. They stay in sync automatically.

Alternative: Cross-Modal Video Embeddings#

The CLIP + OpenAI approach above indexes frames and text separately, which is great for many use cases, and CLIP is free/open-source. But what if you want to search video by sound, spoken words, or visual similarity all through one index? Two providers offer true cross-modal video embeddings:

Option A: Twelve Labs for Cross-Modal Video Search#

Twelve Labs projects text, images, audio, and video into the same semantic space. You split the video into segments and embed each one, then search using any modality (text, image, audio, or another video clip):

You can also focus embeddings on specific aspects (['visual'], ['audio'], or ['transcription']) depending on what matters for your use case. See the full Twelve Labs tutorial.

Option B: Gemini for Native Multimodal Embeddings#

Google's Gemini embed_content function natively accepts Video, Audio, Image, Text, and Document types. No need to extract frames first. Send the video directly:

Which approach should you choose?

• CLIP + OpenAI embeddings (Step 6 above): Free/open-source for image search, good text search. Best for budget-conscious setups.
• Twelve Labs: True cross-modal search across text, image, audio, and video. Best for rich video understanding with multi-query support.
• Gemini: Native multimodal embedding across all data types. Best if you're already in the Google ecosystem.
• Chunked audio + OpenAI (alternative above): Granular time-coded transcript search. Best for spoken-word content like meetings, lectures, and podcasts.

These approaches aren't mutually exclusive. Pixeltable supports multiple embedding indexes on the same table, so you can combine them and evaluate which works best for your use case.

Step 7: Insert Videos and Watch It Work#

Now insert your videos. Everything (frame extraction, object detection, vision descriptions, audio transcription, embedding indexes) triggers automatically.

Step 8: Query Across Modalities#

This is where the unified system pays off. You can query across every modality through the same interface:

Visual Search: Find frames by what they look like#

Text Search: Find frames by their description#

Structured Queries: Filter by detected objects#

Cross-Modal: Combine visual search with transcript context#

Step 9: Iterate (The Real Power)#

Now that your baseline pipeline is running, you can iterate on your data, not your infrastructure. Every change is incremental and versioned:

What You Just Built, Without Building#

Let's count what Pixeltable handled for you:

Capability	Traditional Approach	What You Wrote
Video storage	S3 bucket + IAM + upload scripts	pxt.create_table(..., {'video': pxt.Video})
Frame extraction	FFmpeg scripts + output management	frame_iterator(video=..., fps=1)
Object detection	Model serving + GPU management + batch scripts	add_computed_column(detections=yolox(...))
Vision descriptions	API client + retry logic + rate limiting + result storage	add_computed_column(description=openai.chat_completions(...))
Audio extraction	FFmpeg + temp file management	add_computed_column(audio=extract_audio(...))
Transcription	Whisper API client + chunking + storage	add_computed_column(transcript=openai.transcriptions(...))
Image search	CLIP embedding + Pinecone + sync scripts	add_embedding_index('frame', embedding=clip.using(...))
Text search	Another embedding pipeline + another index	add_embedding_index('description', string_embed=...)
Orchestration	Airflow DAG + dependency config + monitoring	Automatic: insert triggers everything
Versioning	Custom tracking across all services	Automatic: table.history(), table.revert()
Incremental updates	Custom diffing logic per service	Automatic: only new/changed rows process

Next Steps#

You now have a production-ready video intelligence pipeline. Here's where to go next:

• Scale it up: Insert hundreds of videos. Pixeltable processes them incrementally, parallelizing API calls and caching results.
• Add more analysis: Write custom UDFs for domain-specific processing (face detection, logo recognition, custom classifiers).
• Try cross-modal search: Twelve Labs integration for searching video by text, image, audio, or other video clips.
• Multimodal embeddings: Gemini embed_content for native video/audio/image/text embeddings, or Voyage AI for text + image cross-modal search.
• Build on top: Use the Starter Kit to wrap this in a FastAPI + React app.
• Explore cookbooks: Video frame extraction, audio extraction, scene detection, video similarity search.
• Try agents: Use this pipeline as the knowledge base for an AI agent that can answer questions about your video content.

Everything in this tutorial is open source under Apache 2.0. Start with pip install pixeltable or try the 10-Minute Tour.