---
title: "What ML Infrastructure Engineers Actually Want: Design Principles for Modern AI Data Platforms"
date: "2025-12-10"
author: "Pixeltable Team"
tags:
  - ML Infrastructure
  - Data Engineering
  - Design Principles
  - MLOps
  - Best Practices
  - Multimodal AI
description: "Beyond feature checklists, what do practitioners really look for when evaluating ML data infrastructure? We explore the design principles that separate tools that 'work in practice' from those that don't."
url: "https://pixeltable.com/blog/ml-infrastructure-design-principles-evaluation"
---

# What ML Infrastructure Engineers Actually Want: Design Principles for Modern AI Data Platforms

When ML infrastructure engineers evaluate data platforms, they're not just checking feature boxes. They're looking for something deeper: a coherent design philosophy that translates into tools that actually work in practice. After countless conversations with practitioners evaluating Pixeltable alongside other solutions, we've distilled the key principles that separate effective ML data infrastructure from tools that look good on paper but fail in real workflows.

 
## Beyond Feature Checklists: What Really Matters

 
Feature comparison tables are everywhere, but they miss the point. A tool can check every box (versioning ✓, querying ✓, transformations ✓) and still be painful to use. What practitioners actually evaluate is:

 

 - **Does the mental model match how I think about data?**

 - **Can I express complex operations without fighting the API?**

 - **Does it work reliably at scale, or only in demos?**

 - **How much glue code do I need to write?**

 

 
These questions reveal whether a tool's design philosophy aligns with real ML workflows. Let's explore the principles that matter most.

 
## Tables as the Universal Interface

 
There's a reason SQL has persisted for 50 years: the table is an incredibly powerful abstraction. For ML practitioners, tables provide:

 

 - **Familiar mental model** - No new paradigms to learn

 - **Composable operations** - Filter, join, aggregate, transform

 - **Type safety** - Schema enforcement prevents silent failures

 - **Tooling compatibility** - Works with existing data tools

 

 
But here's where ML data platforms diverge: traditional databases treat tables as containers for scalar values. Modern AI workflows need tables that understand **multimodal data** (images, videos, audio, documents) as first-class citizens, not just BLOBs to be stored and retrieved.

 
 
```python

import pixeltable as pxt

# Tables that understand multimodal data natively
videos = pxt.create_table('content.videos', {
 'video': pxt.Video, # Native video type
 'audio': pxt.Audio, # Native audio type 
 'metadata': pxt.Json, # Structured metadata
 'title': pxt.String
})

# Query multimodal data with familiar operations
recent_videos = videos.where(
 videos.metadata['duration'] > 60
).order_by(
 videos.metadata['created_at'], asc=False
).limit(10)
 
```

 
 
The key insight: a unified data model across modalities eliminates the fragmentation that plagues most ML pipelines, where images live in one system, embeddings in another, and metadata in yet another.

 
## Operations That Actually Work

 
Many tools offer query APIs, but the devil is in the details. What practitioners need:

 
 
### Rich Querying

 
Not just basic filtering, but the full expressiveness of SQL-like operations:

 
```python

# Complex queries that feel natural
results = videos.select(
 videos.title,
 videos.metadata['category'],
 videos.transcript.word_count
).where(
 (videos.metadata['language'] == 'en') &
 (videos.transcript.word_count > 100)
).order_by(
 videos.metadata['views'], asc=False
)
 
```

 
 
### Batch Operations with Versioning

 
Real ML workflows require bulk updates, not just single-row operations. But updates without versioning are dangerous: you lose the ability to reproduce experiments or roll back mistakes.

 
```python

# Batch update that creates a versioned revision
videos.update(
 {'status': 'processed'},
 where=videos.metadata['processed'] == False
)

# Every update is tracked - you can always go back
previous_version = pxt.get_table('content.videos', version=5)
 
```

 
 
### First-Class CRUD

 
It sounds basic, but many ML-focused tools treat insert/update/delete as afterthoughts. When these operations are first-class citizens, you can build real applications, not just read-only analytics.

 
## Views and Computed Columns: The Declarative Advantage

 
This is where design philosophy matters most. There are two approaches to data transformations:

 
 
### The Imperative Approach (Script Everything)

 
```python

# Traditional approach: write scripts, manage state manually
for video in get_all_videos():
 frames = extract_frames(video)
 for frame in frames:
 embedding = compute_embedding(frame)
 save_to_database(video.id, frame.id, embedding)
 # What if this fails halfway? 
 # What if we add new videos later?
 # What if we change the embedding model?
 
```

 
 
### The Declarative Approach (Describe What You Want)

 
```python

# Declarative approach: describe the transformation
from pixeltable.functions.video import frame_iterator
from pixeltable.functions import huggingface

# Create a view that extracts frames
frames = pxt.create_view(
 'content.frames',
 videos,
 iterator=frame_iterator(video=videos.video, fps=1)
)

# Add computed column for embeddings
frames.add_computed_column(
 embedding=huggingface.clip_image(frames.frame)
)

# Pixeltable handles:
# ✅ Backfilling existing data
# ✅ Processing new videos automatically 
# ✅ Recomputing when the model changes
# ✅ Tracking dependencies between columns
 
```

 
 
The declarative approach isn't just cleaner. It's fundamentally more maintainable. When you describe *what* you want rather than *how* to get it, the system can optimize execution, handle failures gracefully, and maintain consistency automatically.

 
 
### UDFs: Pluggable Virtual Columns

 
The power of declarative pipelines comes from extensibility. User-defined functions (UDFs) let you plug any Python code into the dataflow:

 
```python

@pxt.udf
def custom_analysis(image: pxt.Image) -> dict:
 # Your custom logic here
 result = my_proprietary_model(image)
 return {'score': result.score, 'labels': result.labels}

# Use it like any built-in function
frames.add_computed_column(
 analysis=custom_analysis(frames.frame)
)
 
```

 
This creates a **pluggable virtual column**: the computation is defined declaratively but can contain arbitrary logic. For more on this pattern, see our guide on [Python UDFs in Pixeltable](/blog/python-udfs-pixeltable).

 
## The Versioning Story

 
ML practitioners are obsessed with reproducibility for good reason: models are only as trustworthy as the data they were trained on. But there's a design tension in how platforms approach versioning:

 
 
### Immutable Snapshots

 
Some systems create immutable snapshots for each version. Simple conceptually, but storage costs explode with large datasets, and comparing versions requires loading entire snapshots.

 
 
### Mutable with History

 
A more practical approach: data is mutable, but every change is tracked. You get the flexibility of updates with the safety of full history.

 
```python

# Current state
current_data = pxt.get_table('content.videos')

# Historical state at any point
yesterday = pxt.get_table('content.videos', version=42)

# Compare versions efficiently
changes = current_data.diff(yesterday)
 
```

 
This approach (automatic versioning with efficient storage) enables workflows that would be impractical with full snapshots: iterative data cleaning, A/B testing datasets, and debugging model regressions by tracing back through data history. Learn more about this in our post on [Pixeltable's versioning and time travel](/blog/pixeltable-versioning-time-travel).

 
## Data Sharing That Actually Works

 
Collaboration is where many ML data tools fall apart. The challenge: how do you share datasets without duplicating terabytes of data or breaking references when files move?

 
 
### The Publish and Replicate Pattern

 
Pixeltable uses a "publish and replicate" model:

 

 - **Publish**: A table (including dependent views and queries) is published to object storage

 - **Replicate**: Collaborators clone the table, creating local copies of metadata with efficient references to shared media

 

 
```python

# Publish a table directory to shared storage
pxt.publish('content', 's3://team-bucket/shared-content/')

# Collaborator replicates locally 
pxt.replicate('s3://team-bucket/shared-content/', 'content')

# External data can be shared without moving files
# Local caching prevents repeated downloads
 
```

 
This design avoids the extremes of "everyone has their own copy" (wasteful) and "everyone shares one mutable source" (dangerous). Read more about our data sharing capabilities in our [data sharing launch post](/blog/pixeltable-data-sharing-launch).

 
## Embeddings: Index-First Design

 
Every ML platform claims to "support embeddings," but the approaches vary wildly. Some treat embeddings as just another data type to store. Others focus on visualization and analysis. Pixeltable takes an **index-first** approach:

 

 - **Automatic indexing** - Embedding columns are indexed for similarity search by default

 - **Incremental updates** - New data is indexed without rebuilding from scratch

 - **Multimodal RAG** - Query across text, image, and audio embeddings in a single operation

 

 
```python

# Create embedding index
frames.add_embedding_index('embedding', metric='cosine')

# Similarity search across your entire dataset
similar_frames = frames.order_by(
 frames.embedding.similarity(query_embedding), 
 asc=False
).limit(10)

# Combine with filters for precise retrieval
relevant_frames = frames.where(
 frames.metadata['category'] == 'product'
).order_by(
 frames.embedding.similarity(query_embedding),
 asc=False
).limit(10)
 
```

 
This orientation toward retrieval and RAG workflows (rather than just storage) reflects where modern AI applications are heading. For more on this approach, see our guide on [incremental embedding indexes](/blog/incremental-embedding-indexes).

 
## Why Design Philosophy Matters

 
After evaluating tools, experienced practitioners often summarize their findings not as feature comparisons, but as design observations:

 
> 
 
"Solid and convincing design, simple and familiar API, good docs. Also seems to work in practice."

 

 
That last phrase, **"works in practice"**, is the ultimate test. It's the result of:

 

 - **Coherent abstractions** - Concepts that compose naturally

 - **Predictable behavior** - No surprises at scale

 - **Helpful errors** - When things go wrong, you know why

 - **Documentation that enables** - Not just reference docs, but guides for real workflows

 

 
These qualities emerge from design philosophy, not feature additions. You can't bolt them on later.

 
## A Practical Evaluation Framework

 
When evaluating ML data platforms, consider these questions:

 
 
| Principle | Questions to Ask |
| --- | --- |
| Data Model | Does it understand my data types natively, or treat everything as bytes? |
| Operations | Can I express complex queries without workarounds? Are batch operations first-class? |
| Transformations | Declarative or imperative? How much orchestration code do I need? |
| Versioning | Automatic or manual? Can I trace data lineage to any point in time? |
| Collaboration | How do I share datasets? What happens when data moves? |
| Extensibility | Can I plug in custom logic without leaving the framework? |
| Practice Test | Does a realistic proof-of-concept actually work? |

 
## Conclusion: Building for Real ML Workflows

 
The best ML data infrastructure doesn't just check feature boxes. It embodies design principles that make complex workflows tractable:

 

 - **Tables as the universal interface** for multimodal data

 - **Rich operations** that match how practitioners think

 - **Declarative transformations** that handle orchestration automatically

 - **Automatic versioning** for reproducibility without overhead

 - **Practical sharing** that enables collaboration at scale

 - **Index-first embeddings** for modern RAG workflows

 

 
These principles guide Pixeltable's design. We believe that when the fundamentals are right, everything else (performance, reliability, developer experience) follows naturally.

 
If you're evaluating ML data platforms, we invite you to [try Pixeltable](https://docs.pixeltable.com/overview/quick-start) and see how these principles translate to practice. Sometimes the best way to evaluate a design philosophy is to experience it firsthand.

 
 
## Further Reading

 

 - [Pixeltable Core Concepts](/blog/pixeltable-core-concepts) - The fundamentals of computed columns and declarative data processing

 - [Dependency Graph Magic](/blog/dependency-graph-magic-computed-columns) - How Pixeltable keeps your AI pipeline data consistent

 - [Declarative vs Imperative AI Pipelines](/blog/declarative-vs-imperative-ai-pipelines) - A deeper dive into the paradigm shift

 - [Navigating the ML Dataset Ecosystem](/blog/navigating-ml-dataset-ecosystem) - Finding your path in the data management landscape