Summary: Building multimodal AI is an experimental process. Which chunking strategy? Which embedding model? What retrieval threshold? What prompt? These decisions compound, and you can only make good ones with fast iteration and comparable results. But when your data lives across 5+ disconnected services, nothing is captured consistently. There's no versioning across the pipeline and no way to compare runs. This post shows what a real experimentation loop looks like when storage, orchestration, and retrieval are unified.
The Hidden Cost of Fragmented Infrastructure#
When we talk to teams building multimodal AI applications, the first complaint is always about complexity: too many services, too much glue code, the Frankenstein Stack.
But the deeper you dig, the real pain isn't operational. It's that they can't iterate.
Building a multimodal AI application isn't a build-once-deploy-forever exercise. It's an experimental process with dozens of decisions:
- How should I chunk these documents: by sentence, by token count, with overlap?
- Which embedding model gives the best retrieval quality for my domain?
- What similarity threshold balances recall vs. precision?
- Which prompt gets the most useful LLM output?
- Does adding object detection to my video frames improve search quality?
Each of these decisions compounds. The chunking strategy affects the embeddings, which affect retrieval, which affects the context the LLM sees, which affects output quality. You can only make good decisions with fast iteration and comparable results.
Why Iteration Is Broken Today#
In the typical multimodal stack (S3 for media, Postgres for metadata, Pinecone for vectors, Airflow for orchestration, LangChain for retrieval), the experimentation loop is fundamentally broken:
1. No Versioning Across the Pipeline#
Each system versions independently, if at all. S3 has object versioning. Postgres has transactions. Pinecone has no built-in versioning. There's no way to say "show me the state of everything at 3pm Tuesday" (the version of your chunks, the embedding model used, the retrieval results, and the LLM output) as a coherent snapshot.
2. No Way to Compare Runs#
If you change your chunking strategy, you re-run your entire pipeline and eyeball the results. There's no queryable system that lets you say: "show me retrieval quality with 300-token chunks vs. 500-token chunks, same queries, same embedding model." You'd have to build that comparison infrastructure yourself.
3. Full Reruns on Every Change#
Changed the embedding model? Re-embed everything. Changed the chunking? Re-chunk, re-embed, re-index. Changed a prompt? Re-run every LLM call. The economics are brutal: most of the compute is wasted reprocessing data that didn't change.
4. Infrastructure Eats the Experiment Budget#
The time your team spends writing sync scripts, debugging stale indexes, and managing pipeline failures is time they're not spending on the experiments that actually improve your product. The infrastructure becomes the bottleneck, not the AI.
The Experimentation Loop in a Unified System#
When storage, orchestration, and retrieval are unified, the experimentation loop transforms. Here's what it actually looks like in Pixeltable:
Step 1: Build Your Baseline#
Start with a table, some computed columns, and an embedding index. This is your baseline: everything is stored, versioned, and queryable from the start.
Everything is now stored, versioned, and queryable. Check the current version:
Step 2: Test Before You Commit#
This is where it gets powerful. Before changing your pipeline, you can test on a sample: nothing is stored, API calls are parallelized and cached:
The key insight: sample(n=5).select(...) is ephemeral. It doesn't modify your table. It doesn't store results. But it does cache the API calls, so if you run it again with the same data, it's instant.
Step 3: Commit When You're Happy#
Once you've validated the improvement, commit it as a computed column. It runs on the full dataset but skips already-cached rows:
Step 4: Swap Models and Compare#
Want to try a different embedding model? Drop the old index and add a new one. Pixeltable recomputes incrementally:
Step 5: Compare Against Any Prior Version#
Every change is automatically versioned. You can query any prior state:
Why This Only Works in a Unified System#
Every step above is trivial individually. The reason it's transformative is that it all happens in the same system:
- Sample, test, commit: works because storage and compute are unified. The sample query, the UDF test, and the committed column all operate on the same data through the same interface.
- Model swapping: works because the embedding index is part of the table, not a separate service. Drop and re-add is atomic, not a distributed operation across Pinecone + custom sync scripts.
- Time travel: works because versioning spans the entire pipeline, not just one layer. You're comparing the full state (chunks, embeddings, model outputs), not just one piece.
- Incremental updates: work because the system tracks the full dependency graph. It knows which embeddings depend on which chunks, and only recomputes what's affected.
- Caching: works because results are persisted automatically. The 5-row sample test caches API responses that are reused when you commit to the full dataset.
In a separated stack, each of these would require custom engineering. Together, they form a fast, natural experimentation loop where the infrastructure gets out of your way.
The Multimodal Dimension: Where This Gets Even More Powerful#
The experimentation loop becomes even more valuable with multimodal data. Consider a video analysis pipeline:
You just went from basic frame extraction to a multi-modal searchable video pipeline, with every decision versioned and reversible. In a separated stack, each step would have been a separate service integration.
Real-World Impact#
This isn't theoretical. Teams using Pixeltable's experimentation loop report transformative changes:
"Pixeltable has transformed our computer vision workflow. Before Pixeltable, our engineers spent 80% of their time on data plumbing. Now, they can focus on what truly matters: building better models and delivering value. We've reduced our infrastructure code by 90%, cut compute costs by 70%, and dramatically accelerated model iteration cycles."
"Pixeltable has transformed how our ML engineers spend their time. We have a highly multimodal use case, and Pixeltable gives us full visibility into the input data, models, and incremental steps of our system's pipeline."
Because everything is declarative and versioned, anyone on the team can reproduce the full workflow without rebuilding anything. No tribal knowledge. No "it works on my machine."
The Loop, Summarized#
| Step | Separated Stack | Pixeltable |
|---|---|---|
| Build baseline | Configure 5+ services, write glue code | Define tables, computed columns, indexes |
| Test a change | Re-run entire pipeline on a fork | table.sample(n=5).select(...) |
| Commit | Update scripts, re-run, hope sync works | add_computed_column(): cached rows skipped |
| Compare | Manual diffing across services | Query current vs. prior version |
| Roll back | Restore backups across 3+ systems | table.revert() |
| Share results | Write docs, export CSVs, screenshot dashboards | Colleague queries the same versioned table |
Stop Fighting Infrastructure. Start Experimenting.#
The teams that win in multimodal AI aren't the ones with the most GPUs or the fanciest models. They're the ones that can iterate fastest: trying more ideas, comparing results, and converging on what works.
That's only possible when your infrastructure gets out of the way. When every decision is versioned, every result is queryable, and every change propagates incrementally, you stop fighting infrastructure and start experimenting.
Iterate on your data, not your infrastructure. pip install pixeltable.
Ready to see the experimentation loop in action? Start with the 10-Minute Tour, explore the iterative workflow cookbook, or try the embedding indexes notebook. Everything is open source under Apache 2.0.
Further Reading#
- AI Transformations Belong in the Schema: the architectural mechanism that makes this experimentation loop possible
- The Economics of Incremental AI: why incremental recomputation matters for cost
- Schema-Driven Infrastructure: the broader thesis on schema as infrastructure specification
