The Infrastructure Trap: Building Custom Systems for Every LLM Output#
You're building an AI agent. The LLM returns structured outputs: maybe tool calls, maybe extracted entities, maybe reasoning steps. The schema isn't fixed. One request might return 3 tool calls, another might return none. One extraction has 5 fields, another has 12.
So you reach for the familiar fallback: unstructured storage. NoSQL databases. Document stores. Key-value pairs. Maybe you build a custom metadata layer. Perhaps you write bespoke serialization logic for every agent workflow.
You've just traded structured data's benefits (queryability, type safety, reproducibility) for the illusion of flexibility.
Here's the reality: Your "unstructured" AI outputs aren't actually unstructured. They have schemas, just non-deterministic ones. An LLM tool call has `name`, `arguments`, and `id`. An extraction has fields. Agent state has structure. You know this because you're writing code to parse, validate, and act on these outputs.
What you need isn't unstructured storage. You need structured storage that handles non-deterministic schemas. And that's exactly what Pixeltable's JSON columns provide, without reinventing your entire data infrastructure.
JSON Columns: Structure for Non-Deterministic Data#
Pixeltable's `pxt.Json` column type gives you the best of both worlds: the flexibility to store arbitrary JSON structures alongside the power of structured, queryable, versioned tables.
LLM Tool Calls: Variable Structure, Predictable Workflow#
Consider an AI agent that can call different tools based on context. The number and type of tool calls varies per request, but you need to track, query, and reproduce these decisions.
What just happened? Your non-deterministic LLM outputs (sometimes 0 tools, sometimes 3, with varying arguments) are captured in a structured, queryable, versioned table. You can filter by tool type, analyze usage patterns, and reproduce every agent decision without building custom infrastructure.
Structured Extraction: Variable Fields, Fixed Benefits#
You're extracting entities from documents. Different document types yield different fields. A research paper has authors, citations, and methodology. A legal document has parties, dates, and clauses. Traditional approaches force you to either:
- Create separate tables per document type (maintenance nightmare)
- Use wide tables with nullable columns for every possible field (sparse, inefficient)
- Abandon structured storage entirely (lose queryability)
Pixeltable's JSON columns eliminate this false choice:
One table. Multiple document types. Variable JSON schemas. Full queryability. No custom infrastructure required.
Agent State: The Dynamic Schema Myth#
Building stateful AI agents reveals the "dynamic schema problem" most acutely. Agent memory evolves. Conversation context grows unpredictably. Tool results vary wildly. Traditional database thinking says: "This needs a flexible schema database!"
But agent state isn't actually schema-less. It's compositional: each interaction adds structured data to an evolving context. The mistake is thinking you need different infrastructure. What you need is a JSON column within your structured workflow.
Building a Memory-Powered Agent with JSON Columns#
Every agent interaction is now:
- Stored with full context (in JSON columns)
- Queryable by session, turn, or content
- Reproducible via Pixeltable's versioning
- Part of a declarative DAG: add new computed columns without rewriting infrastructure
Why JSON Columns Win vs. "Flexible" Databases#
When faced with variable-schema AI outputs, developers often reach for document stores or graph databases. "We need flexibility!" But this trades away critical capabilities:
What You Lose with Document Stores#
| Capability | Document Store | Pixeltable JSON Columns |
|---|---|---|
| Declarative DAGs | ❌ No compute graphs | ✅ JSON outputs feed downstream columns |
| Type Safety | ❌ Everything is untyped | ✅ Top-level schema + flexible JSON |
| Incremental Compute | ❌ Manual change detection | ✅ Automatic recomputation |
| Versioning | ❌ Build yourself | ✅ Built-in with `table.revert()` |
| Cross-Modal | ❌ Separate systems | ✅ JSON + Image + Video in same table |
| Schema Evolution | ✅ Flexible | ✅ JSON fields can vary |
JSON Path Queries: Structured Access to Unstructured Data#
Pixeltable lets you query inside JSON structures using bracket notation, combining structured table operations with flexible JSON traversal:
Real-World Patterns: JSON Columns in Production Workflows#
Pattern 1: Structured Extraction with Validation#
Use Pixeltable's validation capabilities even with variable JSON outputs:
Pattern 2: Agentic Tool Execution with State Tracking#
Capture complete agent execution traces (reasoning, tool calls, and results) in JSON columns while maintaining workflow reproducibility:
Pattern 3: Multimodal Agent Outputs#
Combine structured columns (images, videos, audio) with JSON columns for non-deterministic metadata and decisions:
Incremental Computation with JSON: The Hidden Power#
Here's what makes Pixeltable's JSON columns truly powerful for AI workflows: incremental computation works with non-deterministic outputs.
When you add a new document, only that document's extraction runs. When you update an agent's tools, only affected rows recompute. Pixeltable tracks dependencies through your DAG, even when outputs are JSON with variable schemas.
Common Mistakes When "Going Unstructured"#
Mistake 1: Treating All AI Outputs as Unstructured#
Symptom: Storing every LLM response as a text blob or generic document.
Reality: LLM outputs have structure, even if non-deterministic. A JSON column captures this structure while allowing variability.
Fix:
Mistake 2: Building Separate Storage Per Agent Type#
Symptom: Different tables/databases for research agents, coding agents, data analysis agents.
Reality: All agents share core patterns: inputs, state, outputs. Only the JSON state structure varies.
Fix:
Mistake 3: Manual Change Detection for Dynamic Data#
Symptom: Writing custom code to detect when agent state changes and trigger recomputation.
Reality: Pixeltable's DAG handles this automatically, even with JSON columns.
Fix:
Production Patterns: JSON Columns at Scale#
Indexing JSON Outputs for Fast Queries#
Error Handling with JSON Validation#
When to Use JSON Columns vs. Separate Tables#
Use JSON columns when:
- Output structure varies per row (LLM tool calls, agent decisions)
- Schema evolves over time (learning systems, adaptive agents)
- You need flexibility but still want queryability
- Outputs are hierarchical or nested (conversation context, analysis results)
Use separate columns when:
- Fields are always present (core entity attributes)
- You need strong typing (numeric computations, media types)
- Performance-critical queries on specific fields
- Fields are frequently filtered or sorted
Best practice: Combine both. Use typed columns for core data, JSON for variable AI outputs.
Migrating from Unstructured Storage#
Already storing LLM outputs in document stores or custom systems? Here's how to migrate to Pixeltable while preserving your data:
The Mindset Shift: Structure Isn't the Enemy of Flexibility#
The fundamental insight: Non-deterministic outputs don't require unstructured databases. They require structured databases with JSON column support.
Pixeltable gives you:
- ✅ Declarative DAGs for reproducible workflows
- ✅ Incremental computation for efficiency
- ✅ Versioning and lineage for debugging
- ✅ Type safety for core data
- ✅ JSON flexibility for variable AI outputs
- ✅ Queryability across structured and unstructured
Stop rebuilding infrastructure every time your LLM returns a new field. Stop sacrificing queryability for flexibility. Structure the unstructured. JSON columns let you have both.
Getting Started with JSON Columns#
Ready to stop reinventing infrastructure for every agent workflow?
Learn more:
