---
title: "Build a Complete Video Intelligence Pipeline in 20 Minutes"
date: "2026-03-04"
author: "Pierre Brunelle"
tags:
  - Video AI
  - Tutorial
  - Multimodal AI
  - Computer Vision
  - Audio Transcription
  - Vector Search
  - Twelve Labs
  - Gemini
  - Hands-on
description: "One video generates audio, transcripts, frames, metadata, and embeddings, each needing different processing, storage, and retrieval. Here's how to build the entire pipeline in one system with Pixeltable."
url: "https://pixeltable.com/blog/video-intelligence-pipeline-tutorial"
---

# Build a Complete Video Intelligence Pipeline in 20 Minutes

**Summary:** Video is the hardest data type in AI. A single video generates every other data type (audio, text, images, metadata, embeddings), each requiring different processing, different storage, and different retrieval patterns. Teams today stitch together 5+ services to handle this. This tutorial shows how to build the entire pipeline (from raw video to searchable, transcribed, analyzed, queryable output) in one system with Pixeltable.

 
## What We're Building

 
By the end of this tutorial, you'll have a video intelligence pipeline that:

 

 - **Extracts frames** from videos at a configurable frame rate

 - **Runs object detection** on every frame using YOLOX

 - **Generates descriptions** of each frame using GPT-4o Vision

 - **Extracts and transcribes audio** using OpenAI Whisper

 - **Creates searchable indexes** for image similarity, text search, and audio segments

 - **Queries across modalities**: find frames by visual content, by description text, or by what was said in the audio

 - **Explores alternative strategies**: chunked audio search, cross-modal video embeddings via [Twelve Labs](https://docs.pixeltable.com/howto/providers/working-with-twelvelabs) and [Gemini](https://docs.pixeltable.com/sdk/latest/gemini#udf-embed_content)

 

 
All of this in one system. No S3 to configure, no Pinecone to sync, no Airflow DAGs to manage, no custom glue code. Just tables, computed columns, and queries.

 
## Why Video Is the Ultimate Test for AI Infrastructure

 
[A single video generates every other data type](/blog/multimodal-ai-data-bottleneck):

 

 - **Audio**: extract the audio track, chunk it into segments

 - **Text**: transcribe the audio into searchable text

 - **Images**: extract keyframes, generating hundreds of images per video

 - **Metadata**: run object detection, scene classification on those frames

 - **Embeddings**: embed frames and text into vector space for similarity search

 - **LLM outputs**: generate descriptions, summaries, analysis

 

 
In the [traditional approach](/blog/deconstructing-ai-frankenstein-stack), that's S3 for storage, FFmpeg for extraction, Whisper API for transcription, YOLOX for detection, an embedding service, Pinecone for vectors, and Airflow to orchestrate it all. Seven systems before you write a single line of application logic.

 
Let's do it in one.

 
## Prerequisites

 
```bash
pip install pixeltable openai
```

 
```bash
export OPENAI_API_KEY="your-api-key-here"
```

 
You'll also need a sample video file. Any MP4 will work: a product demo, a meeting recording, a tutorial. If you don't have one handy, you can use a URL to a public video.

 
## Step 1: Create the Video Table

 
Start with a table that uses [Pixeltable's native Video type](https://docs.pixeltable.com/platform/type-system). Your video files stay where they are. Pixeltable references them without copying.

 
```python
import pixeltable as pxt

pxt.create_dir('video_intel', if_exists='ignore')

videos = pxt.create_table('video_intel.videos', {
 'video': pxt.Video,
 'title': pxt.String,
 'source': pxt.String
})
```

 
## Step 2: Extract Frames Automatically

 
Create a [view](https://docs.pixeltable.com/platform/views) with a frame iterator. This declaratively defines "extract one frame per second from every video." When you insert a video, frames are extracted automatically.

 
```python
from pixeltable.functions.video import frame_iterator

frames = pxt.create_view(
 'video_intel.frames',
 videos,
 iterator=frame_iterator(video=videos.video, fps=1)
)

# frames.frame is now an Image column containing extracted frames
# frames.pos is the frame position (timestamp) in the video
```

 
*Tip:* For faster processing on long videos, use `keyframes_only=True` to extract only I-frames, or `num_frames=10` to extract a fixed number of evenly-spaced frames. See the [frame extraction cookbook](https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames) for details.

 
## Step 3: Add Object Detection

 
Add a [computed column](https://docs.pixeltable.com/tutorials/computed-columns) that runs [YOLOX object detection](https://docs.pixeltable.com/sdk/latest/yolox) on every frame. This runs automatically on all existing and future frames.

 
```python
from pixeltable.functions.yolox import yolox

frames.add_computed_column(
 detections=yolox(frames.frame, model_id='yolox_s', threshold=0.25)
)
```

 
The `detections` column now contains bounding boxes, labels, and confidence scores for every detected object in every frame. Pixeltable handles model loading, batching, and caching automatically.

 
## Step 4: Generate Frame Descriptions with GPT-4o Vision

 
Add another computed column that sends each frame to [GPT-4o Vision](https://docs.pixeltable.com/sdk/latest/openai#udf-vision) for a natural language description. API calls are [parallelized, rate-limited, and cached](/blog/rate-limiting) automatically.

 
```python
from pixeltable.functions import openai

frames.add_computed_column(
 description=openai.chat_completions(
 messages=[{
 'role': 'user',
 'content': [
 {'type': 'text', 'text': "Describe what's happening in this frame in 2-3 sentences. Include people, objects, actions, and setting."},
 {'type': 'image_url', 'image_url': {'url': frames.frame}},
 ],
 }],
 model='gpt-4o-mini',
 ).choices[0].message.content
)
```

 
## Step 5: Extract Audio and Transcribe

 
Back on the videos table, add computed columns that [extract audio and transcribe it with Whisper](/blog/whisper-transcription-pixeltable):

 
```python
from pixeltable.functions.video import extract_audio

videos.add_computed_column(
 audio=extract_audio(videos.video)
)

videos.add_computed_column(
 transcript=openai.transcriptions(
 audio=videos.audio,
 model='whisper-1'
 )
)
```

 
The `transcript` column now contains the full text transcription of each video's audio track, with timestamps. See the [audio extraction cookbook](https://docs.pixeltable.com/howto/cookbooks/audio/audio-extract-from-video) for more details.

 
### Alternative: Chunked Audio for Granular Search

 
The approach above transcribes the *entire* audio track as one block. For longer videos, you often want to search within specific segments: "what was said at minute 12?" To enable this, extract the audio, then create a view with the [audio splitter](https://docs.pixeltable.com/platform/iterators) to chunk it into segments, transcribe each chunk, and make them individually searchable:

 
```python
from pixeltable.functions.audio import audio_splitter

# Create a view that splits extracted audio into 30-second chunks
audio_chunks = pxt.create_view(
 'video_intel.audio_chunks',
 videos,
 iterator=audio_splitter(
 audio=videos.audio,
 duration=30.0,
 overlap=2.0,
 min_segment_duration=5.0
 )
)

# Transcribe each audio chunk individually
audio_chunks.add_computed_column(
 transcription=openai.transcriptions(
 audio=audio_chunks.audio_chunk,
 model='whisper-1'
 )
)
audio_chunks.add_computed_column(
 chunk_text=audio_chunks.transcription.text
)

# Make chunk transcriptions searchable
audio_chunks.add_embedding_index(
 'chunk_text',
 string_embed=openai.embeddings.using(model='text-embedding-3-small')
)
```

 
Now you can search *within* the audio timeline: find the exact 30-second segment where a specific topic was discussed, rather than getting back the entire video transcript. Each chunk retains its `start_time_sec` and `end_time_sec` so you can jump to the right moment.

 
## Step 6: Make Everything Searchable

 
Now the powerful part: add [embedding indexes](https://docs.pixeltable.com/platform/embedding-indexes) so you can search across modalities. One line per index. They stay in sync automatically.

 
```python
from pixeltable.functions.huggingface import clip

# Image similarity search: find frames by visual content
frames.add_embedding_index(
 'frame',
 embedding=clip.using(model_id='openai/clip-vit-base-patch32')
)

# Text similarity search: find frames by description content
frames.add_embedding_index(
 'description',
 string_embed=openai.embeddings.using(model='text-embedding-3-small')
)
```

 
### Alternative: Cross-Modal Video Embeddings

 
The CLIP + OpenAI approach above indexes frames and text separately, which is great for many use cases, and CLIP is free/open-source. But what if you want to search video by sound, spoken words, or visual similarity *all through one index*? Two providers offer true cross-modal video embeddings:

 
#### Option A: Twelve Labs for Cross-Modal Video Search

 
[Twelve Labs](https://docs.pixeltable.com/howto/providers/working-with-twelvelabs) projects text, images, audio, and video into the same semantic space. You split the video into segments and embed each one, then search using *any modality* (text, image, audio, or another video clip):

 
```python
from pixeltable.functions.video import video_splitter
from pixeltable.functions import twelvelabs

# Split video into 5-second searchable segments
video_segments = pxt.create_view(
 'video_intel.video_segments',
 videos,
 iterator=video_splitter(
 video=videos.video,
 duration=5.0,
 min_segment_duration=4.0
 )
)

# One index, all modalities: search by text, image, audio, or video
video_segments.add_embedding_index(
 'video_segment',
 embedding=twelvelabs.embed.using(model_name='marengo3.0')
)

# Search with text
sim = video_segments.video_segment.similarity(string='person giving a presentation')
video_segments.order_by(sim, asc=False).limit(3).select(
 video_segments.video_segment, score=sim
).collect()

# Search with an audio clip
sim = video_segments.video_segment.similarity(audio='/path/to/audio-clip.m4a')
video_segments.order_by(sim, asc=False).limit(3).select(
 video_segments.video_segment, score=sim
).collect()
```

 
You can also focus embeddings on specific aspects (`['visual']`, `['audio']`, or `['transcription']`) depending on what matters for your use case. See the [full Twelve Labs tutorial](https://docs.pixeltable.com/howto/providers/working-with-twelvelabs).

 
#### Option B: Gemini for Native Multimodal Embeddings

 
Google's [Gemini embed_content](https://docs.pixeltable.com/sdk/latest/gemini#udf-embed_content) function natively accepts Video, Audio, Image, Text, and Document types. No need to extract frames first. Send the video directly:

 
```python
from pixeltable.functions import gemini

# Embed video segments directly with Gemini
video_segments.add_computed_column(
 video_embedding=gemini.embed_content(
 video_segments.video_segment,
 model='gemini-embedding-002'
 )
)

# Or embed the extracted audio chunks directly
audio_chunks.add_computed_column(
 audio_embedding=gemini.embed_content(
 audio_chunks.audio_chunk,
 model='gemini-embedding-002'
 )
)
```

 
**Which approach should you choose?**

 

 - **CLIP + OpenAI embeddings** (Step 6 above): Free/open-source for image search, good text search. Best for budget-conscious setups.

 - **Twelve Labs:** True cross-modal search across text, image, audio, and video. Best for rich video understanding with multi-query support.

 - **Gemini:** Native multimodal embedding across all data types. Best if you're already in the Google ecosystem.

 - **Chunked audio + OpenAI** (alternative above): Granular time-coded transcript search. Best for spoken-word content like meetings, lectures, and podcasts.

 

 
These approaches aren't mutually exclusive. Pixeltable supports [multiple embedding indexes](https://docs.pixeltable.com/platform/embedding-indexes) on the same table, so you can combine them and evaluate which works best for your use case.

 
## Step 7: Insert Videos and Watch It Work

 
Now insert your videos. Everything (frame extraction, object detection, vision descriptions, audio transcription, embedding indexes) triggers automatically.

 
```python
videos.insert([
 {
 'video': '/path/to/product-demo.mp4',
 'title': 'Product Demo Q1',
 'source': 'marketing'
 },
 {
 'video': '/path/to/meeting-recording.mp4',
 'title': 'Team Standup March 4',
 'source': 'internal'
 }
])

# Check progress
print(f"Videos: {videos.count()}")
print(f"Frames extracted: {frames.count()}")
```

 
## Step 8: Query Across Modalities

 
This is where the unified system pays off. You can query across every modality through the same interface:

 
### Visual Search: Find frames by what they look like

 
```python
# Find frames visually similar to "person giving a presentation"
sim = frames.frame.similarity(string="person giving a presentation")
results = frames.order_by(sim, asc=False).select(
 frames.frame,
 frames.description,
 frames.detections.label_text,
 score=sim
).limit(5).collect()

for row in results:
 print(f"Score: {row['score']:.3f}")
 print(f"Description: {row['description'][:100]}...")
 print(f"Objects: {row['label_text']}")
 print("---")
```

 
### Text Search: Find frames by their description

 
```python
# Search descriptions for specific content
sim = frames.description.similarity(string="showing a chart or graph with data")
results = frames.order_by(sim, asc=False).select(
 frames.frame,
 frames.description,
 score=sim
).limit(5).collect()
```

 
### Structured Queries: Filter by detected objects

 
```python
# Find all frames where a person was detected
person_frames = frames.where(
 frames.detections.label_text.contains('person')
).select(
 frames.frame,
 frames.description,
 frames.detections.label_text
).collect()

print(f"Found {len(person_frames)} frames with people")
```

 
### Cross-Modal: Combine visual search with transcript context

 
```python
# Get the transcript alongside frame analysis
video_intel = videos.select(
 videos.title,
 videos.transcript.text
).collect()

for row in video_intel:
 print(f"Video: {row['title']}")
 print(f"Transcript: {row['text'][:200]}...")
 print("---")
```

 
## Step 9: Iterate (The Real Power)

 
Now that your baseline pipeline is running, you can [iterate on your data, not your infrastructure](/blog/iterate-on-data-not-infrastructure). Every change is incremental and versioned:

 
```python
# Want to add a new analysis column? One line.
frames.add_computed_column(
 sentiment=openai.chat_completions(
 model='gpt-4o-mini',
 messages=[{
 'role': 'user',
 'content': 'Rate the sentiment of this scene as positive/neutral/negative: '
 + frames.description
 }]
 ).choices[0].message.content
)

# Add more videos later: only new data is processed
videos.insert([{
 'video': '/path/to/new-recording.mp4',
 'title': 'New Recording',
 'source': 'field'
}])

# Roll back if something went wrong
frames.revert()
```

 
## What You Just Built, Without Building

 
Let's count what Pixeltable handled for you:

 
| Capability | Traditional Approach | What You Wrote |
| --- | --- | --- |
| Video storage | S3 bucket + IAM + upload scripts | pxt.create_table(..., {'video': pxt.Video}) |
| Frame extraction | FFmpeg scripts + output management | frame_iterator(video=..., fps=1) |
| Object detection | Model serving + GPU management + batch scripts | add_computed_column(detections=yolox(...)) |
| Vision descriptions | API client + retry logic + rate limiting + result storage | add_computed_column(description=openai.chat_completions(...)) |
| Audio extraction | FFmpeg + temp file management | add_computed_column(audio=extract_audio(...)) |
| Transcription | Whisper API client + chunking + storage | add_computed_column(transcript=openai.transcriptions(...)) |
| Image search | CLIP embedding + Pinecone + sync scripts | add_embedding_index('frame', embedding=clip.using(...)) |
| Text search | Another embedding pipeline + another index | add_embedding_index('description', string_embed=...) |
| Orchestration | Airflow DAG + dependency config + monitoring | Automatic: insert triggers everything |
| Versioning | Custom tracking across all services | Automatic: table.history(), table.revert() |
| Incremental updates | Custom diffing logic per service | Automatic: only new/changed rows process |

 
## Next Steps

 
You now have a production-ready video intelligence pipeline. Here's where to go next:

 
Class-based product receipts: [ClipFinder](/blog/clipfinder-semantic-video-moment-search) for natural-language moments, [SafeStream](/blog/safestream-ugc-video-moderation) for a UGC risk fold on the same video graph.

 

 - **Scale it up:** Insert hundreds of videos. Pixeltable processes them incrementally, parallelizing API calls and caching results.

 - **Add more analysis:** [Write custom UDFs](/blog/python-udfs-pixeltable) for domain-specific processing (face detection, logo recognition, custom classifiers).

 - **Try cross-modal search:** [Twelve Labs integration](https://docs.pixeltable.com/howto/providers/working-with-twelvelabs) for searching video by text, image, audio, or other video clips.

 - **Multimodal embeddings:** [Gemini embed_content](https://docs.pixeltable.com/sdk/latest/gemini#udf-embed_content) for native video/audio/image/text embeddings, or [Voyage AI](https://docs.pixeltable.com/howto/providers/working-with-voyageai) for text + image cross-modal search.

 - **Build on top:** Use the [Starter Kit](https://github.com/pixeltable/pixeltable-starter-kit) to wrap this in a FastAPI + React app.

 - **Explore cookbooks:** [Video frame extraction](https://docs.pixeltable.com/howto/cookbooks/video/video-extract-frames), [audio extraction](https://docs.pixeltable.com/howto/cookbooks/audio/audio-extract-from-video), [scene detection](https://docs.pixeltable.com/howto/cookbooks/video/video-scene-detection), [video similarity search](/blog/video-similarity-search).

 - **Try agents:** Use this pipeline as the knowledge base for an [AI agent](/blog/practical-guide-building-agents) that can answer questions about your video content.

 

 
*Everything in this tutorial is [open source](https://github.com/pixeltable/pixeltable) under Apache 2.0. Start with `pip install pixeltable` or try the [10-Minute Tour](https://docs.pixeltable.com/overview/ten-minute-tour).*