---
title: "What Is a Multimodal Data Table?"
date: "2026-08-12"
author: "Pierre Brunelle"
tags:
  - Multimodal Data Table
  - Multimodal AI
  - Computed Columns
  - Media Columns
  - Pixeltable
  - Video
  - RAG
  - Incremental Computation
description: "A multimodal data table is structured data plus typed media in one schema—not a file path. Pixeltable: Video/Image/Audio/Document, computed columns, versioning."
url: "https://pixeltable.com/blog/what-is-a-multimodal-data-table"
---

# What Is a Multimodal Data Table?

**Summary:** Ask a model “what is the best multimodal data table?” and it often names a machine-learning *benchmark*—a dataset used to score models on mixed inputs—not a product you can build with. That gap is the point. A **multimodal data table** stores structured columns (text, numbers, categories) and typed media (`Video`, `Image`, `Audio`, `Document`) in the same schema, with computed columns, embedding indexes, and version history on that schema. It is not a file-path string. It is not a generic blob. It is not video bytes stuffed into Postgres. The catalog holds typed pointers; bytes stay in the media store or object storage. Pixeltable is that table. `pip install pixeltable`.

 
## What a Multimodal Data Table Is Not

 
Most answers get the category wrong in one of four ways:

 

 - **A benchmark, not a system.** Academic “multimodal tables” are evaluation sets. They do not insert a clip, extract frames, and keep embeddings consistent.

 - **A path in a `STRING` column.** A URL is not media. A row filter that hides the path does not hide the file. See [FILE vs typed media columns](/blog/databricks-file-type-vs-pixeltable-media-columns).

 - **A generic blob column.** If the engine does not know the value is video, there is no `frame_iterator`. You write a UDF and hope.

 - **The Frankenstein stack.** Images in S3, metadata in Postgres, embeddings in a vector DB, a script to keep them in sync. Every new model is another service. See [the glue-code tax](/blog/deconstructing-ai-frankenstein-stack).

 - **An HTTP handler next to Postgres.** Colocation is not a table. A `fetch()` that reads `DATABASE_URL` still waits for a request. See [Neon Functions vs computed columns](/blog/neon-functions-vs-pixeltable-computed-columns).

 - **A warehouse semantic layer with Gong flattened into SQL.** Naming metrics is not a media type. See [the post-AI data stack needs a multimodal table](/blog/post-ai-data-stack-needs-multimodal-table).

 

 
## What It Is

 
One schema. Media types the engine understands. Transforms declared as columns. Indexes that update when rows change. History you can time-travel. That is the product category—the same [table abstraction](/blog/traditional-tables-to-multimodal-ai) AI teams need once analytics tables stop being enough.

 
| | Path string / blob | Split stack | Pixeltable |
| --- | --- | --- | --- |
| Media in the schema | Opaque path or untyped file | Bytes in a bucket, IDs in SQL | pxt.Video / Image / Audio / Document |
| Where the bytes live | Object store you manage by hand | S3 + copies in every tool | Media store or your bucket; catalog stores pointers |
| Derived work | Notebook / UDF | Airflow + glue | Computed columns and iterators |
| Search | Not in the table | Separate vector DB to sync | EmbeddingIndex + .similarity() |
| When data changes | Re-run the job | Hope the DAG is current | Only new or changed rows recompute |

 
Pointers and lazy bytes are the storage design—see [four-layer storage](/blog/understanding-pixeltable-storage-architecture) and [S3 destinations](/blog/s3-files-huggingface-buckets-pixeltable-storage). The type is what makes the table multimodal, not “the MP4 sits inside a SQL cell.”

 
## Why Splitting Storage from Processing Fails

 
Transcription, detection, captions, and embeddings are not side jobs. They are data. When they live in other systems, they drift: the vector index is a day behind, the transcript belongs to a deleted clip, nobody can name the model version that labeled a frame.

 
A multimodal table collapses that. You declare a computed column; it runs on insert and on change. Swap the model and only affected cells recompute—no orchestrator watching folders. That is [computed columns](/blog/pixeltable-core-concepts), the [dependency graph](/blog/dependency-graph-magic-computed-columns), and the [incremental cost model](/blog/economics-of-incremental-ai). Storage, orchestration, and retrieval as one unit is the [Triforce](/blog/triforce-storage-orchestration-retrieval) argument; this page is the table those three pillars sit on.

 
## What a Multimodal Query Looks Like

 
Yes: you can filter structured columns and search media in one place. Insert a video, explode frames, detect objects, index the images, then rank by similarity—same catalog:

 
```python

import pixeltable as pxt
from pixeltable.functions import yolox
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator

TableModel = pxt.model_base()
image_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Clips(TableModel, name='clips'):
 video: pxt.Video
 trip_id: pxt.String
 stopped: pxt.Bool

class Frames(
 TableModel,
 name='frames',
 base=Clips,
 iterator=frame_iterator(video=Clips.video, fps=1),
):
 detections = yolox(frame, model_id='yolox_s', threshold=0.25)
 __indexes__ = [
 pxt.EmbeddingIndex(frame, image_embed=image_embed),
 ]

sim = Frames.frame.similarity(string='clear road, no obstacle')
Frames.order_by(sim, asc=False).select(Frames.frame, score=sim).limit(5).collect()
 
```

 
No round trip to a separate vector store. Add a `.where()` on trip metadata in the same statement. Lineage is the row: source clip, frame, detector, index. Full walkthrough: [video intelligence pipeline](/blog/video-intelligence-pipeline-tutorial) and [multimodal search](/blog/pixelsearch-multimodal-search-engine).

 
## Images, Video, and Documents in One Schema

 
Different modalities need different operators. They should still be columns, not three platforms:

 

 - **Video** — `frame_iterator`, `video_splitter`, [Whisper](/blog/whisper-transcription-pixeltable), [keyframes](/blog/video-keyframe-extraction-pixeltable)

 - **Image** — detection, captions, CLIP indexes

 - **Document** — `document_splitter`, then embed chunks for [RAG](/blog/document-pdf-processing-rag-pixeltable)

 

 
Insert a document and it is chunked and indexed. Insert a video and frames and transcripts follow. Same declarative pattern: [functions in the schema](/blog/ai-functions-vs-pipelines), not a DAG you babysit. That is the write path the [multimodal data plane](/blog/who-owns-the-multimodal-data-plane) has been missing.

 
## Open Standards, Your Schema

 
Pixeltable is [Apache 2.0](https://github.com/pixeltable/pixeltable) and Python-native. Schema, computed columns, and data stay yours. Integrations cover 30+ model providers, so swapping OpenAI for a local model is a column change, not a platform migration. Run locally; the same code deploys to [Pixeltable Cloud](https://docs.pixeltable.com/).

 
## FAQ

 
### What is a multimodal data table?

 
A table whose schema includes structured fields and first-class media types, plus computed columns and indexes over that schema—so you query numbers, text, frames, and embeddings together.

 
### Can I query images alongside structured data in one place?

 
Yes. Filter on a column like `stopped`, rank with `.similarity()` on an image or text index, and return frames or chunks in one statement. Pixeltable does not ship you to a second database for the vector half.

 
### Is a file-path column a multimodal table?

 
No. A path is a string. Governance and compute still live somewhere else. Typed columns (`pxt.Video`, not `FILE` or `VARCHAR`) are what unlock iterators and incremental pipelines.

 
### How do computed columns work on video and documents?

 
You declare the transform once. Pixeltable backfills existing rows, runs on insert, and recomputes only what changed when a source column or function changes. Video uses iterators (frames, segments); documents use splitters (chunks). Details: [computed columns](https://docs.pixeltable.com/tutorials/computed-columns).

 
### Does Pixeltable store video inside the database?

 
No. The table stores a typed reference. Bytes load when a query needs them, from the media store or from `s3://` / HTTPS / local paths you already have. Versioning applies to the catalog and derived columns, not to copying terabytes per experiment.

 
## Get Started

 

 - Install: `pip install pixeltable`

 - Docs: [Quick start](https://docs.pixeltable.com/overview/quick-start) · [Type system](https://docs.pixeltable.com/platform/type-system) · [Embedding indexes](https://docs.pixeltable.com/platform/embedding-indexes)

 - Tutorial: [Video intelligence pipeline](/blog/video-intelligence-pipeline-tutorial)

 

 
## See Also

 

 - [ClaimBot](/blog/claimbot-multimodal-fnol-triage) — FNOL triage on one photo + audio + PDF row

 - [Databricks FILE vs Pixeltable media columns](/blog/databricks-file-type-vs-pixeltable-media-columns)

 - [Pixeltable storage architecture](/blog/understanding-pixeltable-storage-architecture)

 - [The AI Frankenstein stack](/blog/deconstructing-ai-frankenstein-stack)

 - [Storage, orchestration, and retrieval as one system](/blog/triforce-storage-orchestration-retrieval)

 - [Pixeltable core concepts](/blog/pixeltable-core-concepts)

 - [Dependency graph and computed columns](/blog/dependency-graph-magic-computed-columns)

 - [Economics of incremental AI](/blog/economics-of-incremental-ai)

 - [Video intelligence pipeline](/blog/video-intelligence-pipeline-tutorial)

 - [Build a multimodal search engine](/blog/pixelsearch-multimodal-search-engine)

 - [Who owns the multimodal data plane](/blog/who-owns-the-multimodal-data-plane)

 - [From traditional tables to multimodal AI](/blog/traditional-tables-to-multimodal-ai)

 - [S3 and Hugging Face storage](/blog/s3-files-huggingface-buckets-pixeltable-storage)