---
title: "Structure the Unstructured: How JSON Columns Handle Non-Deterministic LLM Outputs"
date: "2025-01-15"
author: "Pixeltable Team"
tags:
  - AI Infrastructure
  - LLM
  - Agents
  - JSON
  - Data Architecture
description: "Stop reinventing infrastructure for dynamic schemas. Pixeltable's JSON columns let you capture unpredictable LLM outputs (tool calls, agent decisions, structured extractions) within declarative DAGs without sacrificing queryability or reproducibility."
url: "https://pixeltable.com/blog/structure-the-unstructured-json-llm-outputs"
---

# Structure the Unstructured: How JSON Columns Handle Non-Deterministic LLM Outputs

## The Infrastructure Trap: Building Custom Systems for Every LLM Output

 
 
You're building an AI agent. The LLM returns structured outputs: maybe tool calls, maybe extracted entities, maybe reasoning steps. The schema isn't fixed. One request might return 3 tool calls, another might return none. One extraction has 5 fields, another has 12.

 
 
So you reach for the familiar fallback: unstructured storage. NoSQL databases. Document stores. Key-value pairs. Maybe you build a custom metadata layer. Perhaps you write bespoke serialization logic for every agent workflow.

 
 
**You've just traded structured data's benefits (queryability, type safety, reproducibility) for the illusion of flexibility.**

 
 
Here's the reality: Your "unstructured" AI outputs aren't actually unstructured. They have schemas, just non-deterministic ones. An LLM tool call has `name`, `arguments`, and `id`. An extraction has fields. Agent state has structure. You know this because you're writing code to parse, validate, and act on these outputs.

 
 
What you need isn't unstructured storage. **You need structured storage that handles non-deterministic schemas.** And that's exactly what Pixeltable's JSON columns provide, without reinventing your entire data infrastructure.

 
## JSON Columns: Structure for Non-Deterministic Data

 
 
Pixeltable's `pxt.Json` column type gives you the best of both worlds: the flexibility to store arbitrary JSON structures alongside the power of structured, queryable, versioned tables.

 
 
### LLM Tool Calls: Variable Structure, Predictable Workflow

 
 
Consider an AI agent that can call different tools based on context. The number and type of tool calls varies per request, but you need to track, query, and reproduce these decisions.

 
 
```python

import pixeltable as pxt
from pixeltable.functions import openai

# Create table with JSON column for tool outputs
agent_log = pxt.create_table('agent_interactions', {
 'query': pxt.String,
 'tool_calls': pxt.Json, # Non-deterministic structure
 'final_answer': pxt.String
})

# Define tools as UDFs (Pixeltable pattern)
@pxt.udf
def search_database(query: str) -> list:
 """Search the knowledge base"""
 # Your search logic here
 results = knowledge_base.select(
 knowledge_base.content
 ).order_by(
 knowledge_base.content.similarity(string=query), asc=False
 ).limit(5).collect()
 return [r['content'] for r in results]

@pxt.udf
def analyze_data(dataset: str, metric: str) -> dict:
 """Perform statistical analysis"""
 # Your analysis logic here
 data = datasets.where(datasets.name == dataset).collect()
 return {
 'metric': metric,
 'value': calculate_metric(data, metric),
 'dataset': dataset
 }

# LLM decides which tools to call - non-deterministic
agent_log.add_computed_column(
 llm_response=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'user',
 'content': agent_log.query
 }],
 tools=pxt.tools(search_database, analyze_data)
 )
)

# Extract tool calls into JSON column - structure varies per request
agent_log.add_computed_column(
 tool_calls=agent_log.llm_response.choices[0].message.tool_calls
)

# Query specific tool usage - JSON is queryable
search_calls = agent_log.where(
 agent_log.tool_calls.astype(pxt.String).contains('search_database')
).select(agent_log.query, agent_log.tool_calls).collect()

```

 
 
**What just happened?** Your non-deterministic LLM outputs (sometimes 0 tools, sometimes 3, with varying arguments) are captured in a **structured, queryable, versioned table**. You can filter by tool type, analyze usage patterns, and reproduce every agent decision without building custom infrastructure.

 
### Structured Extraction: Variable Fields, Fixed Benefits

 
 
You're extracting entities from documents. Different document types yield different fields. A research paper has authors, citations, and methodology. A legal document has parties, dates, and clauses. Traditional approaches force you to either:

 
 

 - Create separate tables per document type (maintenance nightmare)

 - Use wide tables with nullable columns for every possible field (sparse, inefficient)

 - Abandon structured storage entirely (lose queryability)

 

 
 
Pixeltable's JSON columns eliminate this false choice:

 
 
```python

import pixeltable as pxt
from pixeltable.functions import openai

# One table handles all document types
documents = pxt.create_table('documents', {
 'doc': pxt.Document,
 'doc_type': pxt.String,
 'extracted_data': pxt.Json # Schema varies by document type
})

# Extraction prompt adapts to document type
@pxt.udf
def get_extraction_prompt(doc_type: str) -> str:
 prompts = {
 'research_paper': '''Extract: title, authors (array), abstract, 
 publication_year, citations (count), 
 methodology, key_findings (array)''',
 'legal_contract': '''Extract: parties (array), effective_date, 
 expiration_date, governing_law, 
 key_clauses (array), obligations (object)''',
 'financial_report': '''Extract: company, period, revenue, 
 expenses, profit_margin, 
 highlights (array), risks (array)'''
 }
 return prompts.get(doc_type, 'Extract all relevant structured information')

# Non-deterministic extraction - different fields per doc type
documents.add_computed_column(
 extracted_data=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'system',
 'content': get_extraction_prompt(documents.doc_type)
 }, {
 'role': 'user', 
 'content': documents.doc.astype(pxt.String)
 }],
 response_format={'type': 'json_object'}
 ).choices[0].message.content
)

# Query across variable schemas using JSON path expressions
papers_with_citations = documents.where(
 documents.doc_type == 'research_paper'
).select(
 documents.extracted_data['title'],
 documents.extracted_data['citations']
).collect()

# Different query for legal docs - same table, different JSON structure
contracts_expiring_soon = documents.where(
 documents.doc_type == 'legal_contract'
).select(
 documents.extracted_data['parties'],
 documents.extracted_data['expiration_date']
).collect()

```

 
 
One table. Multiple document types. Variable JSON schemas. Full queryability. **No custom infrastructure required.**

 
## Agent State: The Dynamic Schema Myth

 
 
Building stateful AI agents reveals the "dynamic schema problem" most acutely. Agent memory evolves. Conversation context grows unpredictably. Tool results vary wildly. Traditional database thinking says: "This needs a flexible schema database!"

 
 
But agent state isn't actually schema-less. It's **compositional**: each interaction adds structured data to an evolving context. The mistake is thinking you need different infrastructure. What you need is a JSON column within your structured workflow.

 
 
### Building a Memory-Powered Agent with JSON Columns

 
 
```python

import pixeltable as pxt
from pixeltable.functions import openai

# Agent conversation table with JSON for flexible state
conversations = pxt.create_table('agent_conversations', {
 'session_id': pxt.String,
 'turn': pxt.Int,
 'user_input': pxt.String,
 'agent_state': pxt.Json, # Evolving context - varies per agent
 'response': pxt.String
})

# Memory retrieval function - query past interactions
@pxt.query
def retrieve_memory(session_id: str, query: str, limit: int = 5):
 return conversations.where(
 conversations.session_id == session_id
 ).order_by(
 conversations.turn, asc=False
 ).limit(limit).select(
 conversations.user_input,
 conversations.agent_state,
 conversations.response
 )

# Build context from memory and current state
@pxt.udf
def assemble_context(
 current_input: str,
 session_id: str,
 past_interactions: list
) -> dict:
 # Non-deterministic: context structure varies by conversation
 return {
 'current_query': current_input,
 'conversation_history': past_interactions,
 'extracted_entities': [...], # Varies per session
 'user_preferences': {...}, # Learned over time
 'pending_tasks': [...] # Dynamic list
 }

# Agent response with full context
conversations.add_computed_column(
 context=assemble_context(
 conversations.user_input,
 conversations.session_id,
 retrieve_memory(conversations.session_id, conversations.user_input)
 )
)

conversations.add_computed_column(
 agent_state=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'system',
 'content': 'You are a stateful assistant. Update your internal state based on the conversation.'
 }, {
 'role': 'user',
 'content': f"Context: {conversations.context}\n\nUser: {conversations.user_input}"
 }],
 response_format={'type': 'json_object'} # Non-deterministic JSON output
 ).choices[0].message.content
)

# Extract response from state
conversations.add_computed_column(
 response=conversations.agent_state['response'].astype(pxt.String)
)

```

 
 
**Every agent interaction is now:**

 

 - Stored with full context (in JSON columns)

 - Queryable by session, turn, or content

 - Reproducible via Pixeltable's versioning

 - Part of a declarative DAG: add new computed columns without rewriting infrastructure

 

 
## Why JSON Columns Win vs. "Flexible" Databases

 
 
When faced with variable-schema AI outputs, developers often reach for document stores or graph databases. "We need flexibility!" But this trades away critical capabilities:

 
 
### What You Lose with Document Stores

 
 
| Capability | Document Store | Pixeltable JSON Columns |
| --- | --- | --- |
| Declarative DAGs | ❌ No compute graphs | ✅ JSON outputs feed downstream columns |
| Type Safety | ❌ Everything is untyped | ✅ Top-level schema + flexible JSON |
| Incremental Compute | ❌ Manual change detection | ✅ Automatic recomputation |
| Versioning | ❌ Build yourself | ✅ Built-in with `table.revert()` |
| Cross-Modal | ❌ Separate systems | ✅ JSON + Image + Video in same table |
| Schema Evolution | ✅ Flexible | ✅ JSON fields can vary |

 
 
### JSON Path Queries: Structured Access to Unstructured Data

 
 
Pixeltable lets you query inside JSON structures using bracket notation, combining structured table operations with flexible JSON traversal:

 
 
```python

# Query nested JSON from LLM outputs
agent_decisions = conversations.where(
 conversations.agent_state['confidence'] > 0.8
).where(
 conversations.agent_state['requires_human_review'] == False
).select(
 conversations.user_input,
 conversations.agent_state['decision'],
 conversations.agent_state['reasoning']
).collect()

# Filter by array elements in JSON
multi_tool_calls = agent_log.where(
 pxt.functions.json_array_length(agent_log.tool_calls) > 1
).collect()

# Extract specific fields across varying schemas
extracted_names = documents.select(
 documents.doc_type,
 documents.extracted_data['title'].astype(pxt.String),
 documents.extracted_data['authors'] # Might be array or single value
).collect()

```

 
## Real-World Patterns: JSON Columns in Production Workflows

 
 
### Pattern 1: Structured Extraction with Validation

 
 
Use Pixeltable's validation capabilities even with variable JSON outputs:

 
 
```python

import pixeltable as pxt
from pixeltable.functions import openai

# Table with validated JSON extraction
product_reviews = pxt.create_table('reviews', {
 'review_text': pxt.String,
 'extracted_insights': pxt.Json # Variable structure per review
})

# Extract with validation
@pxt.udf
def validate_extraction(extracted: dict) -> bool:
 # Core fields must exist, but additional fields are fine
 required = {'sentiment', 'rating', 'key_points'}
 return all(field in extracted for field in required)

product_reviews.add_computed_column(
 extracted_insights=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'system',
 'content': '''Extract: sentiment (positive/negative/neutral), 
 rating (1-5), key_points (array of strings).
 Include any other relevant insights.'''
 }, {
 'role': 'user',
 'content': product_reviews.review_text
 }],
 response_format={'type': 'json_object'}
 ).choices[0].message.content
)

# Validation column - marks invalid extractions
product_reviews.add_computed_column(
 is_valid=validate_extraction(product_reviews.extracted_insights)
)

# Query only valid, high-confidence extractions
valid_insights = product_reviews.where(
 product_reviews.is_valid == True
).where(
 product_reviews.extracted_insights['rating'] >= 4
).collect()

```

 
### Pattern 2: Agentic Tool Execution with State Tracking

 
 
Capture complete agent execution traces (reasoning, tool calls, and results) in JSON columns while maintaining workflow reproducibility:

 
 
```python

# Agent workflow table
agent_tasks = pxt.create_table('agent_tasks', {
 'task_description': pxt.String,
 'execution_trace': pxt.Json, # Tool calls, reasoning, state
 'final_result': pxt.String
})

# Step 1: Agent planning (returns JSON with variable tool sequence)
agent_tasks.add_computed_column(
 plan=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'system',
 'content': 'Plan task execution. Return JSON with: reasoning, tool_sequence (array), estimated_steps'
 }, {
 'role': 'user',
 'content': agent_tasks.task_description 
 }],
 response_format={'type': 'json_object'}
 ).choices[0].message.content
)

# Step 2: Define tools as UDFs
@pxt.udf
def search_knowledge(query: str) -> str:
 """Tool: Search internal knowledge base"""
 results = knowledge_base.select(
 knowledge_base.content
 ).order_by(
 knowledge_base.content.similarity(string=query), asc=False
 ).limit(3).collect()
 return "\n".join([r['content'] for r in results])

@pxt.udf
def fetch_external_data(api_endpoint: str) -> dict:
 """Tool: Fetch data from external API"""
 import requests
 response = requests.get(api_endpoint)
 return response.json()

# Tool execution with LLM orchestration
agent_tasks.add_computed_column(
 execution_result=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'system',
 'content': f"Plan: {agent_tasks.plan}\n\nExecute the planned tasks using available tools."
 }, {
 'role': 'user',
 'content': agent_tasks.task_description
 }],
 tools=pxt.tools(search_knowledge, fetch_external_data)
 )
)

# Capture execution trace in JSON
agent_tasks.add_computed_column(
 execution_trace=agent_tasks.execution_result.choices[0].message
)

# Step 3: Final synthesis using execution results
agent_tasks.add_computed_column(
 final_result=openai.chat_completions(
 model='gpt-4o',
 messages=[{
 'role': 'user',
 'content': f"Task: {agent_tasks.task_description}\n\nExecution: {agent_tasks.execution_trace}\n\nSummarize the result"
 }]
 ).choices[0].message.content
)

# Query: Find tasks that required >2 tools
complex_tasks = agent_tasks.where(
 pxt.functions.json_array_length(agent_tasks.plan['tool_sequence']) > 2
).select(
 agent_tasks.task_description,
 agent_tasks.execution_trace['tools_executed']
).collect()

```

 
### Pattern 3: Multimodal Agent Outputs

 
 
Combine structured columns (images, videos, audio) with JSON columns for non-deterministic metadata and decisions:

 
 
```python

# Multimodal agent processing table
media_analysis = pxt.create_table('media_tasks', {
 'media': pxt.Image, # Structured: native media type
 'analysis': pxt.Json, # Unstructured: variable AI insights
 'tags': pxt.Array[pxt.String] # Structured: queryable tags
})

# Vision analysis with non-deterministic output structure
media_analysis.add_computed_column(
 analysis=openai.chat_completions(
 messages=[{
 'role': 'user',
 'content': [
 {'type': 'text', 'text': '''Analyze this image. Return JSON with whatever insights 
 are relevant: objects (array), scene_type, people_count, 
 text_content, colors (array), mood, etc.
 Include only fields that apply.'''},
 {'type': 'image_url', 'image_url': {'url': media_analysis.media}},
 ],
 }],
 model='gpt-4o',
 ).choices[0].message.content
)

# Extract standard tags from variable analysis
@pxt.udf
def extract_tags(analysis: dict) -> list:
 tags = []
 if 'scene_type' in analysis:
 tags.append(analysis['scene_type'])
 if 'mood' in analysis:
 tags.append(analysis['mood'])
 if 'objects' in analysis:
 tags.extend(analysis['objects'][:3]) # Top 3 objects
 return tags

media_analysis.add_computed_column(
 tags=extract_tags(media_analysis.analysis)
)

# Query: Find images with people, regardless of analysis structure
images_with_people = media_analysis.where(
 (media_analysis.analysis.astype(pxt.String).contains('people')) |
 (media_analysis.analysis['people_count'].astype(pxt.Int) > 0)
).collect()

```

 
## Incremental Computation with JSON: The Hidden Power

 
 
Here's what makes Pixeltable's JSON columns truly powerful for AI workflows: **incremental computation works with non-deterministic outputs**.

 
 
When you add a new document, only that document's extraction runs. When you update an agent's tools, only affected rows recompute. Pixeltable tracks dependencies through your DAG, even when outputs are JSON with variable schemas.

 
 
```python

# Original workflow
documents.insert([
 {'doc': 'paper1.pdf', 'doc_type': 'research_paper'},
 {'doc': 'contract1.pdf', 'doc_type': 'legal_contract'}
])

# All extractions run, JSON columns populated

# Add new document - only ONE extraction runs
documents.insert([
 {'doc': 'paper2.pdf', 'doc_type': 'research_paper'}
])

# Update extraction logic - only NEW logic runs on new data
@pxt.udf
def enhanced_extraction(doc_type: str, previous: dict) -> dict:
 # Evolve your extraction without reprocessing everything
 base = previous
 if doc_type == 'research_paper':
 base['impact_score'] = calculate_impact(previous)
 return base

documents.add_computed_column(
 enhanced_insights=enhanced_extraction(
 documents.doc_type,
 documents.extracted_data
 )
)

# Only incremental computation - Pixeltable handles change propagation

```

 
## Common Mistakes When "Going Unstructured"

 
 
### Mistake 1: Treating All AI Outputs as Unstructured

 
 
**Symptom:** Storing every LLM response as a text blob or generic document.

 
 
**Reality:** LLM outputs have structure, even if non-deterministic. A JSON column captures this structure while allowing variability.

 
 
**Fix:**

 
```python

# ❌ BAD: Everything as text
llm_outputs = pxt.create_table('outputs', {
 'prompt': pxt.String,
 'response': pxt.String # Lost all structure!
})

# ✅ GOOD: Structured + JSON for variability
llm_outputs = pxt.create_table('outputs', {
 'prompt': pxt.String,
 'response_text': pxt.String, # For display
 'structured_output': pxt.Json, # For queryability
 'metadata': pxt.Json # Model, tokens, etc.
})

```

 
### Mistake 2: Building Separate Storage Per Agent Type

 
 
**Symptom:** Different tables/databases for research agents, coding agents, data analysis agents.

 
 
**Reality:** All agents share core patterns: inputs, state, outputs. Only the JSON state structure varies.

 
 
**Fix:**

 
```python

# ✅ One table for all agent types
unified_agents = pxt.create_table('agents', {
 'agent_type': pxt.String, # research, coding, analysis
 'task': pxt.String,
 'agent_state': pxt.Json, # Varies by agent type
 'result': pxt.Json # Varies by task complexity
})

# Query across agent types
high_value_tasks = unified_agents.where(
 unified_agents.agent_state['estimated_value'] > 1000
).select(
 unified_agents.agent_type,
 unified_agents.task,
 unified_agents.result
).collect()

```

 
### Mistake 3: Manual Change Detection for Dynamic Data

 
 
**Symptom:** Writing custom code to detect when agent state changes and trigger recomputation.

 
 
**Reality:** Pixeltable's DAG handles this automatically, even with JSON columns.

 
 
**Fix:**

 
```python

# JSON changes trigger downstream recomputation automatically
agent_conversations.add_computed_column(
 summary=generate_summary(agent_conversations.agent_state)
)

# When agent_state JSON changes, summary recomputes
# No manual change detection needed

```

 
## Production Patterns: JSON Columns at Scale

 
 
### Indexing JSON Outputs for Fast Queries

 
 
```python

# Extract commonly-queried JSON fields to indexed columns
documents.add_computed_column(
 title=documents.extracted_data['title'].astype(pxt.String)
)

documents.add_computed_column(
 pub_year=documents.extracted_data['publication_year'].astype(pxt.Int)
)

# Now these are indexed, queryable columns
# JSON still available for ad-hoc fields
recent_papers = documents.where(
 documents.pub_year >= 2023
).select(
 documents.title,
 documents.extracted_data # Full JSON when needed
).collect()

```

 
### Error Handling with JSON Validation

 
 
```python

@pxt.udf
def safe_json_extract(json_str: str, field: str, default: any = None):
 try:
 import json
 data = json.loads(json_str) if isinstance(json_str, str) else json_str
 return data.get(field, default)
 except:
 return default

# Safely extract fields that might not exist
documents.add_computed_column(
 safe_title=safe_json_extract(
 documents.extracted_data,
 'title',
 'Untitled'
 )
)

```

 
## When to Use JSON Columns vs. Separate Tables

 
 
**Use JSON columns when:**

 

 - Output structure varies per row (LLM tool calls, agent decisions)

 - Schema evolves over time (learning systems, adaptive agents)

 - You need flexibility but still want queryability

 - Outputs are hierarchical or nested (conversation context, analysis results)

 

 
 
**Use separate columns when:**

 

 - Fields are always present (core entity attributes)

 - You need strong typing (numeric computations, media types)

 - Performance-critical queries on specific fields

 - Fields are frequently filtered or sorted

 

 
 
**Best practice:** Combine both. Use typed columns for core data, JSON for variable AI outputs.

 
## Migrating from Unstructured Storage

 
 
Already storing LLM outputs in document stores or custom systems? Here's how to migrate to Pixeltable while preserving your data:

 
 
```python

# Import existing unstructured data
import pixeltable as pxt
import json

# Create structured table with JSON column
migrated_data = pxt.create_table('migrated_llm_outputs', {
 'original_id': pxt.String,
 'timestamp': pxt.Timestamp,
 'model': pxt.String,
 'output_data': pxt.Json, # Your existing unstructured data
 'metadata': pxt.Json
})

# Bulk import from your document store
existing_records = fetch_from_mongodb() # Or wherever
migrated_data.insert([
 {
 'original_id': record['_id'],
 'timestamp': record['created_at'],
 'model': record.get('model', 'unknown'),
 'output_data': record['llm_output'], # Preserve as JSON
 'metadata': record.get('metadata', {})
 }
 for record in existing_records
])

# Now add Pixeltable's incremental computation
migrated_data.add_computed_column(
 processed_output=process_and_validate(migrated_data.output_data)
)

# Get the benefits of structured storage + DAGs retroactively

```

 
## The Mindset Shift: Structure Isn't the Enemy of Flexibility

 
 
The fundamental insight: **Non-deterministic outputs don't require unstructured databases**. They require structured databases with JSON column support.

 
 
Pixeltable gives you:

 

 - ✅ **Declarative DAGs** for reproducible workflows

 - ✅ **Incremental computation** for efficiency

 - ✅ **Versioning and lineage** for debugging

 - ✅ **Type safety** for core data

 - ✅ **JSON flexibility** for variable AI outputs

 - ✅ **Queryability** across structured and unstructured

 

 
 
Stop rebuilding infrastructure every time your LLM returns a new field. Stop sacrificing queryability for flexibility. **Structure the unstructured**. JSON columns let you have both.

 
## Getting Started with JSON Columns

 
 
Ready to stop reinventing infrastructure for every agent workflow?

 
 
```python

# Install Pixeltable
pip install pixeltable

# Your first JSON column workflow
import pixeltable as pxt
from pixeltable.functions import openai

# Create table with JSON for flexible outputs
my_table = pxt.create_table('ai_workflow', {
 'input': pxt.String,
 'ai_output': pxt.Json # Non-deterministic schema
})

# LLM generates variable JSON
my_table.add_computed_column(
 ai_output=openai.chat_completions(
 model='gpt-4o',
 messages=[{'role': 'user', 'content': my_table.input}],
 response_format={'type': 'json_object'}
 ).choices[0].message.content
)

# Query the variable outputs
results = my_table.select(
 my_table.input,
 my_table.ai_output
).collect()

```

 
 
**Learn more:**

 

 - [Computed Columns Guide](https://docs.pixeltable.com/datastore/computed-columns)

 - [OpenAI Integration](https://docs.pixeltable.com/integrations/openai)

 - [Building Production AI Agents](/blog/practical-guide-building-agents)

 - [Production RAG Systems](/blog/production-rag-data-centric)