1. Introduction#
In June 2017, eight researchers at Google published a paper titled "Attention Is All You Need." It introduced the Transformer architecture, replacing recurrence with self-attention. The paper launched the era of GPT, BERT, LLaMA, and every large language model that followed.
The thesis was elegant: if you give the model the right architecture and enough data, you don't need the complex, hand-engineered systems of the past. Attention is all you need.
Eight years later, the thesis has been proven correct, so correct that it made itself obsolete. The Transformer won. Foundation models from OpenAI, Anthropic, Google, and Meta have commoditized intelligence. Inference costs collapsed. Capabilities converged. Embeddings, retrieval, vision, audio, code generation: all available as API calls.
The model era is over. The data era has begun.
If every company has access to the same models, the differentiator is no longer the model. It's the data you feed it. Specifically: how you store diverse multimodal assets, how you orchestrate transformations and AI calls across them, and how you retrieve the right context at the right time. The real bottleneck was never the attention mechanism. It's the data infrastructure around it.
2. The Transformer Analogy#
The Transformer paper made a structural observation: recurrent networks processed data sequentially, creating bottlenecks. Self-attention processed everything in parallel, enabling scale. The insight wasn't "attention is good." It was that the architecture of the system determined its limits, not the data or the training algorithm.
We make the same observation about AI application infrastructure.
Today, AI application teams process data sequentially through disconnected systems: S3 for storage, then a script to extract frames, then Pinecone for embeddings, then Airflow to orchestrate, then LangChain to retrieve. Each step is a handoff. Each handoff is a failure mode. The architecture creates bottlenecks that no amount of model intelligence can overcome.
Just as the Transformer replaced sequential recurrence with parallel attention, declarative data infrastructure replaces sequential pipeline handoffs with unified, parallel, incremental computation.
| Before Transformers | Before Declarative Infra | |
|---|---|---|
| Architecture | Sequential (RNNs, LSTMs) | Sequential (pipeline handoffs between services) |
| Bottleneck | Can't parallelize across time steps | Can't synchronize across services |
| Failure mode | Vanishing gradients, limited context | Stale indexes, zombie vectors, broken lineage |
| The fix | Self-attention (parallel, unified) | Computed columns (parallel, unified) |
| Key paper / system | "Attention Is All You Need" (2017) | Pixeltable (2024) |
3. The Five Problems Data Solves That Models Can't#
No matter how good your model is, these five problems determine whether your AI application works in production:
3.1 Context Quality#
An LLM is only as good as the context you feed it. If your RAG pipeline retrieves stale chunks, or your video search returns frames from a deleted video, the model produces garbage. Production RAG isn't a model problem. It's a data infrastructure problem. The embeddings must be fresh, the chunks must be correct, and the index must be in sync. That requires storage, orchestration, and retrieval working as one system.
3.2 Iteration Speed#
The competitive advantage in AI is iteration speed. Which chunking strategy works best? Which embedding model? What prompt? These decisions compound. But when your data lives across disconnected services, you can't iterate. You can only re-run. Change a model? Re-embed everything. Change a prompt? Re-process the entire dataset. The infrastructure eats the experiment budget.
3.3 Multimodal Complexity#
A single video generates every other data type: audio, text, images, metadata, embeddings. Each requires different processing, different storage, and different retrieval patterns. No existing system was designed to handle all of them together. Vector databases store embeddings. Object stores hold bytes. Orchestrators coordinate tasks. None provide unified multimodal data management.
3.4 Production Maintenance#
AI makes it easy to ship, not to maintain. A prototype works in a notebook. Production requires versioning, lineage, incremental updates, caching, error recovery, and retrieval that stays in sync when data changes. These are infrastructure properties, not model properties. No LLM can provide them. The data layer must.
3.5 Lineage and Trust#
In healthcare, security, and finance, you need to trace every output back to its source. Which model version produced this embedding? Which video frame generated this detection? Which prompt created this summary? When data flows through five separate services, lineage is a manual reconstruction exercise. When it flows through one system, lineage is automatic. It's just the dependency graph.
4. The Architecture: Declarative Multimodal Data Infrastructure#
Just as the Transformer paper proposed a specific architecture (multi-head self-attention, positional encoding, encoder-decoder stacks), we propose a specific architecture for multimodal data infrastructure:
4.1 Native Multimodal Types (Positional Encoding for Data)#
In the Transformer, positional encoding gives the model awareness of sequence position. In Pixeltable, native multimodal types give the system awareness of data modality. Video, Audio, Image, Document aren't opaque blobs. They're first-class column types with native operations. Your files stay where they are (local, S3, URLs). Pixeltable references them without copying, backed by a four-layer storage architecture that separates metadata, media, cache, and temp data.
4.2 Computed Columns (Self-Attention for Data)#
Self-attention lets every token attend to every other token in parallel. Computed columns let every transformation attend to every piece of data in the table: automatically, incrementally, in parallel. Define once, runs on all data, present and future.
Insert a video and everything downstream (frames, detection, descriptions, transcriptions) computes automatically. Change the model? Only affected rows recompute. Add new data? Only new data processes. This is the equivalent of going from O(n) recurrence to O(1) parallel attention, but for data pipelines. For a complete walkthrough, see Build a Video Intelligence Pipeline in 20 Minutes or explore the Whisper transcription and YOLOX object detection guides.
4.3 Embedding Indexes (Attention Heads for Retrieval)#
Multi-head attention lets the model attend to different aspects of the input simultaneously. Embedding indexes let you search across different modalities simultaneously (image similarity, text similarity, structured queries) all on the same data, always in sync.
4.4 Automatic Versioning (Residual Connections for Data)#
Residual connections in Transformers let the model preserve information from earlier layers. Automatic versioning in Pixeltable preserves every prior state of your data (schema changes, model swaps, data updates) so you can always go back. Unlike bolted-on version tracking with DVC or MLflow, this is structural: every mutation is captured at the storage layer.
4.5 Persistent Memory and State (KV Cache for Agents)#
In Transformers, the KV cache stores previous attention computations so the model doesn't recompute them on every token. In agentic AI, the equivalent is persistent memory: the accumulated state of tool outputs, intermediate artifacts, reasoning traces, and conversation history that an agent needs to function over time.
LLM frameworks treat agents as stateless function calls. Every invocation starts from scratch. But real agents are long-lived workflows: a customer support agent that remembers prior interactions, a research agent that accumulates findings across sessions, a code agent that tracks tool calls and their outcomes.
Pixeltable gives agents what the KV cache gives Transformers: persistent, queryable, versioned state, without the agent framework having to manage it.
Tool calls, LLM responses, decision traces: all stored as computed columns with full lineage. Every decision is traceable. Every interaction is reproducible. The agent's memory grows with the table, not with a separate memory service. See Pixelagent and team workflows for production implementations, or connect via MCP servers for Claude, Cursor, and other AI IDE integrations.
5. Experimental Results#
Unlike the original Transformer paper, our experimental results come not from BLEU scores on WMT benchmarks, but from production teams building real applications:
| Metric | Traditional Stack | Pixeltable | Source |
|---|---|---|---|
| Pipeline code | Baseline | 85% reduction | Obvio (CV pipelines) |
| Time to ship | 3 weeks | 3 days | Variata (computer use agents) |
| Infrastructure code | Baseline | 90% reduction | Obvio (production deployment) |
| Compute cost | Baseline | 70% reduction | Obvio (incremental processing) |
| Systems to manage | 5-8 services | 1 system | All design partners |
| Orchestration config | DAGs, retry logic, sync scripts | Zero | Architecture (computed columns) |
“Pixeltable has transformed our computer vision workflow. Before Pixeltable, our engineers spent 80% of their time on data plumbing. Now, they can focus on what truly matters: building better models and delivering value. We've reduced our infrastructure code by 90%, cut compute costs by 70%, and dramatically accelerated model iteration cycles.”
“Pixeltable has transformed how our ML engineers spend their time. We have a highly multimodal use case, and Pixeltable gives us full visibility into the input data, models, and incremental steps of our system's pipeline.”
6. Related Work#
We situate our contribution relative to existing systems:
- Vector Databases (Pinecone, Weaviate, Qdrant) solve the retrieval head but not the storage or orchestration heads. They are indexes, not systems of record. They know nothing about the raw video, the pipeline that produced the embedding, or the lineage between them.
- Orchestrators (Airflow, Prefect, Temporal) coordinate task execution but don't own the data. They are the glue between systems, not the foundation. Changing a model in Airflow doesn't trigger incremental recomputation. It triggers a full rerun.
- LLM Frameworks (LangChain, LlamaIndex) are stateless by design. They make it easy to wire up retrieval and generation, but they don't persist data, don't version anything, and don't handle incremental updates. Every run starts from scratch.
- Snowflake and Databricks own structured analytics and batch processing, respectively. But multimodal AI workloads are transactional (OLTP), not analytical (OLAP). Different engine, different consistency model, different access patterns.
- General-purpose frameworks coordinate existing systems but don't unify them. The gap between a Python library and a production data system is architectural, not incremental.
- Agent frameworks (CrewAI, AutoGen, custom) handle orchestration of LLM calls but not the persistent state underneath. They don't store tool outputs, don't version reasoning traces, and don't provide memory that survives between sessions. Pixeltable provides the invisible infrastructure that makes production agents reliable.
In the language of the original paper: these systems are the LSTMs and GRUs of AI infrastructure: useful, widely deployed, but architecturally constrained. Pixeltable is the Transformer: a ground-up redesign that removes the bottlenecks by unifying the three attention heads of data infrastructure: storage, orchestration, and retrieval. For a deeper comparison with specific tools, see our analyses of feature stores, Snowflake for AI, and the unified infrastructure approach.
7. Scaling Behavior#
The Transformer's key insight was that self-attention scales better than recurrence. Pixeltable's key insight is that incremental computation scales better than full reruns.
In a traditional pipeline, processing cost scales linearly with total dataset size. Every change triggers a full rerun. In Pixeltable, cost scales with the delta: only changed or new data processes. As datasets grow, the efficiency gap compounds:
- Add 10 videos to a 10,000-video dataset? Process 10, not 10,000.
- Swap an embedding model? Recompute embeddings, but reuse cached frames, detections, and transcriptions.
- Fix a prompt? Only the LLM column recomputes. Everything upstream is untouched.
This is the dependency graph in action: the system understands which outputs depend on which inputs and only recomputes what's necessary. Combined with built-in rate limiting, type safety, and automatic caching, the system handles the operational complexity that teams otherwise spend months building by hand. Just as Transformers made it feasible to train on internet-scale data, incremental computation makes it feasible to iterate on production-scale multimodal datasets.
8. Conclusion#
In 2017, the AI community learned that attention is all you need, for models. In 2026, as models converge and inference commoditizes, the lesson is different: data is all you need, for applications.
The quality of your AI application is determined by the quality of your data infrastructure: how you store multimodal assets, how you orchestrate transformations, and how you retrieve the right context. Models are powerful, ubiquitous, and interchangeable. Data infrastructure is not.
Every era of data produced an infrastructure owner. Oracle for relational. Snowflake for structured analytics. Databricks for batch processing. The multimodal data plane (video, audio, images, documents) has no owner. The workloads are too large, the pain is too acute, and the pattern from prior eras is too clear.
Pixeltable is the architecture proposed for this era: declarative, incremental, multimodal-native data infrastructure where computed columns replace ETL pipelines, embedding indexes replace standalone vector databases, automatic versioning replaces manual tracking, and persistent memory replaces stateless agent frameworks. One system for storage, orchestration, and retrieval.
Attention was all you needed. Now, data is all you need. And data needs infrastructure.
pip install pixeltable
Pixeltable is open source under Apache 2.0. Start with the 10-Minute Tour, build a complete video pipeline in 20 minutes, explore the agent architecture guide, or set up your AI coding tool for Pixeltable. For hands-on tutorials, try the smart image organizer, the document processing RAG pipeline, or the video keyframe extraction guide. The source code, documentation, PixelBot, and app template are all available on GitHub.
With thanks to Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin, for the paper, the architecture, and the title we couldn't resist borrowing.
