---
title: "Iterate on Your Data, Not Your Infrastructure: The Multimodal Experimentation Loop"
date: "2026-03-04"
author: "Pierre Brunelle"
tags:
  - AI Infrastructure
  - Multimodal AI
  - Experimentation
  - Incremental Computation
  - Developer Experience
description: "The biggest cost of fragmented AI infrastructure isn't operational. It's that you can't iterate. Here's how a unified system turns infrastructure friction into a fast, queryable experimentation loop."
url: "https://pixeltable.com/blog/iterate-on-data-not-infrastructure"
---

# Iterate on Your Data, Not Your Infrastructure: The Multimodal Experimentation Loop

**Summary:** Building multimodal AI is an experimental process. Which chunking strategy? Which embedding model? What retrieval threshold? What prompt? These decisions compound, and you can only make good ones with fast iteration and comparable results. But when your data lives across 5+ disconnected services, nothing is captured consistently. There's no versioning across the pipeline and no way to compare runs. This post shows what a real experimentation loop looks like when storage, orchestration, and retrieval are unified.

 
## The Hidden Cost of Fragmented Infrastructure

 
When we talk to teams building multimodal AI applications, the first complaint is always about complexity: too many services, too much glue code, the [Frankenstein Stack](/blog/deconstructing-ai-frankenstein-stack).

 
But the deeper you dig, the real pain isn't operational. It's that **they can't iterate**.

 
Building a multimodal AI application isn't a build-once-deploy-forever exercise. It's an experimental process with dozens of decisions:

 

 - How should I chunk these documents: by sentence, by token count, with overlap?

 - Which embedding model gives the best retrieval quality for my domain?

 - What similarity threshold balances recall vs. precision?

 - Which prompt gets the most useful LLM output?

 - Does adding object detection to my video frames improve search quality?

 

 
Each of these decisions compounds. The chunking strategy affects the embeddings, which affect retrieval, which affects the context the LLM sees, which affects output quality. You can only make good decisions with fast iteration and comparable results.

 
## Why Iteration Is Broken Today

 
In the typical [multimodal stack](/blog/deconstructing-ai-frankenstein-stack) (S3 for media, Postgres for metadata, Pinecone for vectors, Airflow for orchestration, LangChain for retrieval), the experimentation loop is fundamentally broken:

 
### 1. No Versioning Across the Pipeline

 
Each system versions independently, if at all. S3 has object versioning. Postgres has transactions. Pinecone has no built-in versioning. There's no way to say "show me the state of everything at 3pm Tuesday" (the version of your chunks, the embedding model used, the retrieval results, and the LLM output) as a coherent snapshot.

 
### 2. No Way to Compare Runs

 
If you change your chunking strategy, you re-run your entire pipeline and eyeball the results. There's no queryable system that lets you say: "show me retrieval quality with 300-token chunks vs. 500-token chunks, same queries, same embedding model." You'd have to build that comparison infrastructure yourself.

 
### 3. Full Reruns on Every Change

 
Changed the embedding model? Re-embed everything. Changed the chunking? Re-chunk, re-embed, re-index. Changed a prompt? Re-run every LLM call. [The economics are brutal](/blog/economics-of-incremental-ai): most of the compute is wasted reprocessing data that didn't change.

 
### 4. Infrastructure Eats the Experiment Budget

 
The time your team spends writing sync scripts, debugging stale indexes, and managing pipeline failures is time they're not spending on the experiments that actually improve your product. [The infrastructure becomes the bottleneck](/blog/multimodal-ai-data-bottleneck), not the AI.

 
## The Experimentation Loop in a Unified System

 
When [storage, orchestration, and retrieval are unified](/blog/triforce-storage-orchestration-retrieval), the experimentation loop transforms. Here's what it actually looks like in Pixeltable:

 
### Step 1: Build Your Baseline

 
Start with a table, some computed columns, and an embedding index. This is your baseline: everything is stored, versioned, and queryable from the start.

 
```python
import pixeltable as pxt
from pixeltable.functions import openai
from pixeltable.functions.huggingface import sentence_transformer
from pixeltable.functions.document import document_splitter

# Create document table
docs = pxt.create_table('experiments.docs', {
 'document': pxt.Document,
 'source': pxt.String
})

# Chunk with your initial strategy
chunks = pxt.create_view(
 'experiments.chunks', docs,
 iterator=document_splitter(
 document=docs.document,
 separators='sentence',
 limit=500
 )
)

# Add embeddings
chunks.add_embedding_index(
 'text',
 string_embed=sentence_transformer.using(
 model_id='all-MiniLM-L6-v2'
 )
)

# Add LLM-generated summaries
chunks.add_computed_column(
 summary=openai.chat_completions(
 model='gpt-4o-mini',
 messages=[{
 'role': 'user',
 'content': pxt.functions.string.format(
 'Summarize this text in one sentence: {text}',
 text=chunks.text
 )
 }]
 ).choices[0].message.content
)

# Insert your documents
docs.insert([
 {'document': '/data/reports/q1.pdf', 'source': 'quarterly'},
 {'document': '/data/reports/q2.pdf', 'source': 'quarterly'},
])
```

 
Everything is now stored, versioned, and queryable. Check the current version:

 
```python
# See all versions of your table
chunks.history()
```

 
### Step 2: Test Before You Commit

 
This is where it gets powerful. Before changing your pipeline, you can **test on a sample**: nothing is stored, API calls are parallelized and cached:

 
```python
@pxt.udf
def improved_summary(text: str) -> str:
 """A better summarization prompt"""
 import openai as oai
 response = oai.chat.completions.create(
 model='gpt-4o',
 messages=[{
 'role': 'user',
 'content': f'Write a concise, information-dense summary: {text}'
 }]
 )
 return response.choices[0].message.content

# Test on 5 random rows: nothing stored, calls parallelized and cached
results = chunks.sample(n=5).select(
 chunks.text,
 old_summary=chunks.summary,
 new_summary=improved_summary(chunks.text)
).collect()

# Compare side-by-side
for row in results:
 print(f"Old: {row['old_summary'][:100]}...")
 print(f"New: {row['new_summary'][:100]}...")
 print("---")
```

 
The key insight: `sample(n=5).select(...)` is ephemeral. It doesn't modify your table. It doesn't store results. But it does cache the API calls, so if you run it again with the same data, it's instant.

 
### Step 3: Commit When You're Happy

 
Once you've validated the improvement, commit it as a computed column. It runs on the full dataset but [skips already-cached rows](/blog/economics-of-incremental-ai):

 
```python
# Happy with the results? Commit to the full dataset
chunks.drop_column('summary')
chunks.add_computed_column(summary=improved_summary(chunks.text))

# Only uncached rows are recomputed: the 5 sample rows are instant
```

 
### Step 4: Swap Models and Compare

 
Want to try a different embedding model? Drop the old index and add a new one. Pixeltable recomputes incrementally:

 
```python
# Try a different embedding model
chunks.drop_embedding_index('text')
chunks.add_embedding_index(
 'text',
 string_embed=sentence_transformer.using(
 model_id='all-mpnet-base-v2'
 )
)

# Test retrieval quality with the new model
sim = chunks.text.similarity(string='What were the key revenue drivers?')
results = chunks.order_by(sim, asc=False).select(
 chunks.text,
 score=sim
).limit(5).collect()

for row in results:
 print(f"Score: {row['score']:.3f} | {row['text'][:80]}...")
```

 
### Step 5: Compare Against Any Prior Version

 
Every change is [automatically versioned](/blog/pixeltable-versioning-time-travel). You can query any prior state:

 
```python
# See all versions
chunks.history()

# Query a prior version (before the model swap)
old_chunks = pxt.get_table('experiments.chunks:3')
old_sim = old_chunks.text.similarity(string='What were the key revenue drivers?')
old_results = old_chunks.order_by(old_sim, asc=False).select(
 old_chunks.text,
 score=old_sim
).limit(5).collect()

# Compare old vs. new retrieval quality side by side
# Roll back if the new model is worse
chunks.revert()
```

 
## Why This Only Works in a Unified System

 
Every step above is trivial individually. The reason it's transformative is that it all happens **in the same system**:

 

 - **Sample, test, commit**: works because storage and compute are unified. The sample query, the UDF test, and the committed column all operate on the same data through the same interface.

 - **Model swapping**: works because the embedding index is part of the table, not a separate service. Drop and re-add is atomic, not a distributed operation across Pinecone + custom sync scripts.

 - **Time travel**: works because [versioning spans the entire pipeline](https://docs.pixeltable.com/platform/version-control), not just one layer. You're comparing the full state (chunks, embeddings, model outputs), not just one piece.

 - **Incremental updates**: work because the system tracks [the full dependency graph](/blog/dependency-graph-magic-computed-columns). It knows which embeddings depend on which chunks, and only recomputes what's affected.

 - **Caching**: works because results are persisted automatically. The 5-row sample test caches API responses that are reused when you commit to the full dataset.

 

 
In a separated stack, each of these would require custom engineering. Together, they form a fast, natural experimentation loop where the infrastructure gets out of your way.

 
## The Multimodal Dimension: Where This Gets Even More Powerful

 
The experimentation loop becomes even more valuable with multimodal data. Consider a video analysis pipeline:

 
```python
import pixeltable as pxt
from pixeltable.functions import openai, yolox
from pixeltable.functions.huggingface import clip
from pixeltable.functions.video import frame_iterator

videos = pxt.create_table('experiments.videos', {'video': pxt.Video})

frames = pxt.create_view(
 'experiments.frames', videos,
 iterator=frame_iterator(video=videos.video, fps=1)
)

# Baseline: YOLOX detection + CLIP embedding
frames.add_computed_column(
 objects=yolox(frames.frame, model_id='yolox_s')
)
frames.add_embedding_index(
 'frame',
 embedding=clip.using(model_id='openai/clip-vit-base-patch32')
)

videos.insert([{'video': '/data/surveillance/cam1.mp4'}])

# Experiment: does adding scene description improve search?
# Test on a sample first
results = frames.sample(n=10).select(
 frames.frame,
 frames.objects,
 description=openai.chat_completions(
 messages=[{
 'role': 'user',
 'content': [
 {'type': 'text', 'text': "Describe what's happening in this frame"},
 {'type': 'image_url', 'image_url': {'url': frames.frame}},
 ],
 }],
 model='gpt-4o-mini',
 ).choices[0].message.content
).collect()

# If descriptions look good, commit
frames.add_computed_column(
 description=openai.chat_completions(
 messages=[{
 'role': 'user',
 'content': [
 {'type': 'text', 'text': "Describe what's happening in this frame"},
 {'type': 'image_url', 'image_url': {'url': frames.frame}},
 ],
 }],
 model='gpt-4o-mini',
 ).choices[0].message.content
)

# Add text search on descriptions alongside image search
frames.add_embedding_index(
 'description',
 string_embed=openai.embeddings.using(model='text-embedding-3-small')
)

# Now you can search by image similarity OR text description
image_results = frames.order_by(
 frames.frame.similarity(string="person near a vehicle"), asc=False
).limit(5).collect()

text_results = frames.order_by(
 frames.description.similarity(string="person near a vehicle"), asc=False
).limit(5).collect()
```

 
You just went from basic frame extraction to a multi-modal searchable video pipeline, with every decision versioned and reversible. In a separated stack, each step would have been a separate service integration.

 
## Real-World Impact

 
This isn't theoretical. Teams using Pixeltable's experimentation loop report transformative changes:

 
> 
 
"Pixeltable has transformed our computer vision workflow. Before Pixeltable, our engineers spent 80% of their time on data plumbing. Now, they can focus on what truly matters: building better models and delivering value. We've reduced our infrastructure code by 90%, cut compute costs by 70%, and dramatically accelerated model iteration cycles."

 Adil Mohammad, Founding Engineer at [Obvio](https://obvio.ai/), formerly Senior Deep Learning Engineer at Nvidia
 

 
> 
 
"Pixeltable has transformed how our ML engineers spend their time. We have a highly multimodal use case, and Pixeltable gives us full visibility into the input data, models, and incremental steps of our system's pipeline."

 Denise Kutnick, CEO of [Variata](https://variata.ai/), formerly Director of MLSys Product at OctoAI
 

 
Because everything is declarative and versioned, anyone on the team can reproduce the full workflow without rebuilding anything. No tribal knowledge. No "it works on my machine."

 
## The Loop, Summarized

 
| Step | Separated Stack | Pixeltable |
| --- | --- | --- |
| Build baseline | Configure 5+ services, write glue code | Define tables, computed columns, indexes |
| Test a change | Re-run entire pipeline on a fork | table.sample(n=5).select(...) |
| Commit | Update scripts, re-run, hope sync works | add_computed_column(): cached rows skipped |
| Compare | Manual diffing across services | Query current vs. prior version |
| Roll back | Restore backups across 3+ systems | table.revert() |
| Share results | Write docs, export CSVs, screenshot dashboards | Colleague queries the same versioned table |

 
## Stop Fighting Infrastructure. Start Experimenting.

 
The teams that win in multimodal AI aren't the ones with the most GPUs or the fanciest models. They're the ones that can iterate fastest: trying more ideas, comparing results, and converging on what works.

 
That's only possible when your infrastructure gets out of the way. When every decision is versioned, every result is queryable, and every change propagates incrementally, you stop fighting infrastructure and start experimenting.

 
Iterate on your data, not your infrastructure. `pip install pixeltable`.

 
*Ready to see the experimentation loop in action? Start with the [10-Minute Tour](https://docs.pixeltable.com/overview/ten-minute-tour), explore the [iterative workflow cookbook](https://docs.pixeltable.com/howto/cookbooks/core/dev-iterative-workflow), or try the [embedding indexes notebook](https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/notebooks/feature-guides/embedding-indexes.ipynb). Everything is [open source](https://github.com/pixeltable/pixeltable) under Apache 2.0.*

 
## Further Reading

 

 - [AI Transformations Belong in the Schema](/blog/ai-transformations-in-the-schema): the architectural mechanism that makes this experimentation loop possible

 - [The Economics of Incremental AI](/blog/economics-of-incremental-ai): why incremental recomputation matters for cost

 - [Schema-Driven Infrastructure](/blog/schema-driven-infrastructure-ai): the broader thesis on schema as infrastructure specification