Summary: Building multimodal AI applications requires three capabilities: storing diverse data types, orchestrating transformations and model calls, and retrieving results efficiently. Today, each capability lives in a separate system: object stores for storage, DAG runners for orchestration, vector databases for retrieval. The result is a fragmented stack held together by glue code. This post argues that these three pillars form an indivisible unit (a Triforce), and that unifying them under a single abstraction is the architectural breakthrough that makes multimodal AI tractable.
The Three Pillars Every AI Application Needs#
Strip away the buzzwords and every AI application (whether it's a RAG pipeline, a video analysis system, or an autonomous agent) needs exactly three things from its infrastructure:
- Storage: a place to put raw data (videos, images, audio, documents) alongside structured metadata, computed results, and model outputs. Not just bytes in a bucket, but typed, versioned, queryable data.
- Orchestration: a way to define what should happen to that data: extract frames, transcribe audio, generate embeddings, call an LLM. And a way to ensure it happens reliably, incrementally, and in the right order.
- Retrieval: a way to get results back out: similarity search over embeddings, filtered queries over metadata, joins across modalities. Fast, consistent, and always in sync with the latest data.
These aren't independent concerns. They're deeply coupled. The embeddings you retrieve depend on the orchestration that produced them, which depends on the data in storage. Change any one piece and the other two must respond. They form a Triforce: three faces of a single system, not three systems bolted together.
How Teams Build Today: One Tool Per Pillar#
The industry's default approach is to pick the "best" tool for each pillar and integrate them yourself:
| Pillar | Typical Tools | What They Do Well | What They Don't Do |
|---|---|---|---|
| Storage | S3, GCS, Postgres | Store bytes and rows reliably | No native multimodal types, no computed transforms, no versioning across modalities |
| Orchestration | Airflow, Prefect, Temporal | Schedule and coordinate tasks | Don't own the data, can't do incremental updates, no built-in caching or lineage |
| Retrieval | Pinecone, Weaviate, Qdrant | Fast similarity search | Only store embeddings: know nothing about raw data, pipeline, or lineage |
Each tool is excellent in isolation. The problem is the space between them. We've written about this as the Frankenstein Stack: a Frankensteined architecture where 40-60% of the codebase is glue code to move data, handle retries, manage state, and keep indexes in sync.
Why Separating the Three Pillars Fails#
The Triforce metaphor isn't just aesthetic. There are deep technical reasons why these three pillars resist separation:
1. The Synchronization Problem#
When storage, orchestration, and retrieval live in different systems, you need to keep them in sync manually. Consider a simple operation: delete a video from your dataset.
- Delete the file from S3.
- Delete metadata rows from Postgres.
- Find and delete all derived embeddings in Pinecone.
- Invalidate cached inference results.
- Update any downstream views or indexes.
If step 3 fails, you have zombie vectors: embeddings that point to data that no longer exists. Your search returns broken results. Your RAG app hallucinates context from deleted documents. In a unified system, this is one atomic operation.
2. The Incrementality Problem#
Multimodal AI is inherently iterative. You change a prompt, swap an embedding model, adjust a chunking strategy, add new data. Each change should propagate efficiently, recomputing only what's affected, not re-running the entire pipeline.
But incrementality requires the system to understand the dependency graph: which embeddings depend on which frames, which frames came from which video, which transcriptions used which model version. When these relationships span three separate systems, incremental updates are essentially impossible. Teams fall back to expensive full reruns.
3. The Lineage Problem#
In regulated industries (healthcare, security, finance) you need to trace any output back to its source. Which model version produced this embedding? Which video frame was this detection derived from? Which prompt generated this summary?
When data flows through S3 → custom script → Pinecone → LangChain, lineage is a manual reconstruction exercise. When storage, orchestration, and retrieval are one system, lineage is automatic. It's just the dependency graph.
4. The Experimentation Problem#
The cost of separated pillars isn't just operational. It kills the feedback loop. Building multimodal AI is an experimental process. Which chunking strategy works best? Which embedding model? What retrieval threshold? You can only make good decisions with fast iteration and comparable results.
When your data lives across disconnected services, nothing is captured consistently. There's no versioning across the pipeline. No way to compare runs. Teams spend more time fighting infrastructure than actually experimenting, which is where all the value is.
What the Unified Triforce Looks Like#
If you designed a system from scratch to unify all three pillars, what would it look like? We think it looks like this:
Storage: Multimodal Types as First-Class Citizens#
Video, audio, images, and documents can't be opaque blobs. They need to be first-class column types with native operations. Your files stay where they are (local disk, S3, URLs). The system references them without copying. All computed outputs are stored and versioned automatically.
Orchestration: Declarative Computed Columns#
Instead of writing imperative scripts and DAGs, you declare what you want computed. Computed columns trigger automatically when data arrives, with incremental updates when anything upstream changes. Rate limiting, retries, caching, parallelization: all built in.
Retrieval: Built-In Embedding Indexes and Search#
Vector search isn't a separate service. It's a column operation. Embedding indexes stay in sync with the data automatically. No manual index maintenance. No sync jobs. No stale vectors.
What Changes When the Triforce Is Unified#
When all three pillars live in one system, things that were impossible become trivial:
| Operation | Separated Stack | Unified Triforce |
|---|---|---|
| Delete a video and all derivatives | 5-step distributed operation, risk of zombie vectors | One atomic delete, cascading cleanup |
| Swap the embedding model | Re-run entire pipeline, rebuild all indexes | One line change, incremental recomputation |
| Compare two chunking strategies | Build separate pipelines, manual diffing | Version the column, query both versions |
| Trace an output to its source | Manual reconstruction across 3+ systems | Automatic: follow the dependency graph |
| Add new data | Trigger DAG, wait for each step, hope nothing fails | table.insert(): everything propagates |
| Roll back a bad model update | Restore backups across S3 + Postgres + Pinecone | table.revert() |
The Full Picture: 47 Lines → 12 Lines#
Here's what this looks like in practice. A video processing pipeline with storage, orchestration, and retrieval, the traditional approach vs. the unified approach:
Traditional approach, with separate systems for each pillar:
Unified approach, one system for all three pillars:
A fraction of the code. But the real difference isn't the line count. It's that the unified version is auto-orchestrated, incremental, versioned, and queryable. The traditional version is none of those things.
Why Existing Tools Can't Just Add the Missing Pieces#
The natural question: why can't Pinecone add storage? Why can't Airflow add retrieval? Why can't Postgres add orchestration?
Because each tool was architecturally designed for one pillar:
- Vector databases started as single-purpose indexes. Adding storage, orchestration, and lineage means rebuilding as a general-purpose system, something they weren't designed for.
- Orchestrators coordinate existing systems. They don't own the data. Airflow knows the order of your tasks, but it doesn't know that changing your embedding model means recomputing your search index.
- Databases handle structured metadata brilliantly. But Postgres can't natively process a video, extract and transcribe audio, run object detection on frames, or manage embedding indexes.
- LLM frameworks like LangChain are stateless by design. They coordinate calls but don't persist data, don't version anything, and don't handle incremental updates.
Bolting capabilities onto a system designed for a different purpose creates the worst of both worlds. The Triforce must be designed as a unit from the ground up.
How Pixeltable Implements the Triforce#
Pixeltable is open-source Python infrastructure for incremental storage, transformation, indexing, and orchestration of multimodal data. It was designed from the start to unify all three pillars under a single table abstraction:
| Pillar | How Pixeltable Implements It |
|---|---|
| Storage | Native multimodal types (Video, Image, Audio, Document) with automatic versioning, four-layer storage architecture, and zero-duplication file references |
| Orchestration | Computed columns with automatic dependency tracking, incremental recomputation, rate limiting, caching, retries, and parallelization across 50+ AI providers |
| Retrieval | Built-in embedding indexes with automatic sync, similarity search, and structured queries: all through the same table interface |
Because all three live in one system, you get capabilities that are impossible with separated tools:
- Time travel: query any prior version of your data, schema, and computed results with
table.history()andpxt.get_table('my_table:472') - Automatic dependency tracking: the system knows your embedding depends on your frame, which depends on your video. Change anything and only the affected downstream results recompute.
- Prototype-to-production continuity: test a UDF on a sample with
t.sample(n=5).select(...), then commit it as a computed column. Same code, same system, no handoff. - Native audit trails: every row, every computed result, every model call is automatically versioned with full provenance.
Video: The Ultimate Test of the Triforce#
If you want to understand why the Triforce matters, look at video. A single video generates every other data type:
- Audio: extract the audio track, chunk it, transcribe with Whisper. Now you have text.
- Images: extract keyframes, generating hundreds of images per video.
- Metadata: run object detection, scene classification, face recognition on those frames.
- Embeddings: embed frames, audio chunks, and transcript segments into vector space for similarity search.
- LLM outputs: feed all of it into an LLM for summarization, analysis, or agentic workflows.
One input, five derived data types, each requiring different processing (orchestration), different storage, and different retrieval patterns. And you need lineage between all of them.
Teams building video pipelines today (in media, security, healthcare) are stitching together 5+ services to handle this. With a unified Triforce, it's one table, a few computed columns, and an embedding index. Insert a video and everything else happens automatically.
The Experimentation Loop: Where the Triforce Pays Off Most#
The biggest payoff of unification isn't operational. It's the feedback loop it enables.
Building multimodal AI is an experimental process. Which embedding model? What retrieval threshold? What prompt? These decisions compound, and you can only make good ones with fast iteration and comparable results. In a unified system, swapping a model or testing a new function is a one-line change, and the system handles incremental recomputation automatically:
Every decision is versioned. Every result is queryable. Every change propagates incrementally. This is only possible when storage, orchestration, and retrieval are one system, when the Triforce is complete.
Claim the Triforce#
The history of data infrastructure is clear: the systems that win are the ones that provide operational integrity under change across the full lifecycle. Snowflake didn't just store data. It handled access control, versioning, compute scaling, and governance. Databricks didn't just run Spark. It managed notebooks, lineage, model registries, and feature stores.
Multimodal AI needs the same. Storage, orchestration, and retrieval aren't three separate problems. They're one problem with three faces. Solve them together and you get a system where everything just works: atomic updates, incremental recomputation, automatic lineage, and a single interface from prototype to production.
Solve them separately and you get the Frankenstein Stack: 5+ services, thousands of lines of glue code, and an engineering team that spends more time on infrastructure than on AI.
The Triforce is waiting. pip install pixeltable.
Ready to see what unified infrastructure looks like? Start with the 10-Minute Tour, explore the video processing cookbooks, or dive into the RAG pipeline guide. Everything is open source under Apache 2.0.
