Summary: “Where does the agent run?” is a solved question. It runs in a sandbox—an ephemeral machine that boots, executes whatever the model just wrote, and dies without ceremony. Disposability is the point. That raises the question that actually matters for platforms and fleets: if the place the agent runs is built to be destroyed, where does the agent live? An agent is a storage-shaped problem. You can swap the sandbox, the harness, and even the model; the one part you cannot regenerate is the state. In Pixeltable that state is a queryable multimodal catalog—session log, memory, media, lineage, and versions under one snapshot boundary—not a process, and not five stores glued together with hope.
An Agent Is Four Things in Four Places#
Be precise about what “an agent” physically is. It is not one process on one machine. It decomposes into four pieces, hosted in four different places:
| Piece | Where it lives | Disposable? |
|---|---|---|
| Model | Inference provider (or your GPUs)—stateless per call | Yes |
| Harness | Your orchestration loop on app servers / workers | Yes—if state is externalized |
| Sandbox | Ephemeral compute for tools and generated code | Yes—by design |
| State | Pixeltable catalog (tables, media, versions) | No—this is the agent |
Three layers are replaceable at will. The fourth is the one you cannot rebuild from a prompt. The agent does not live in the sandbox, the harness, or the model; it lives in its state, and everything else is a way of moving that state forward in time. That is the same inversion as agents as data: the session log is the agent; the runtime is an interpreter.
The Agent’s World, Frozen Mid-Task#
If you froze an agent mid-task and took inventory of everything it would need to become itself again, you would find four kinds of durable state:
- Working files / media — repos, artifacts, images, video clips, half-finished outputs. Already bytes; in Pixeltable they are typed columns (
pxt.Image,pxt.Video,pxt.Document) referenced by the catalog. - Memory — conversation history, scratchpads, what session forty-one learned that session one did not. Append-only rows, not a hidden file inside a worker.
- Structured state — run metadata, tool results, config, task status. Queryable columns and
pxt.Json, not an opaque checkpoint blob. - Provenance — what the world looked like before a destructive step. Lineage,
table.history(), time travel, and immutable snapshots.
Notice what this inventory has in common: all four are first-class data under one catalog boundary (PIXELTABLE_HOME). The catalog at that moment is the agent at that moment. Object storage may hold media bytes; the catalog is the consistent world—see storage architecture.
Today, That World Is Scattered Across Five Systems#
That is the theory. In most deployments the fourth layer looks nothing like one namespace:
| State | Where it usually sits |
|---|---|
| Conversation history | Postgres messages table your framework created on first boot |
| Memory / embeddings | A vector DB + Redis hot cache + maybe another memory product |
| Checkpoints | Framework checkpointer blobs tied to one library version |
| Working files | An S3 bucket, copied in and out of the sandbox |
| Run metadata | Temporal history, a Celery backend, or a runs table you bolted on |
Each piece can be fine alone. The problem is the seams: there is no moment at which you can capture all five in one consistent state. You cannot snapshot “the agent,” because the agent has no snapshot boundary. You cannot cleanly fork two continuations or replay Tuesday byte-for-byte. That is the Frankenstein stack—and the multimodal version of the same ownership gap in who owns the multimodal data plane.
“But My Sandbox Has Snapshots”#
It does, and the objection deserves a direct answer. Sandbox volumes and machine snapshots persist compute across restarts. If “the VM still has files” were the whole requirement, you could stop there.
It is not. A sandbox snapshot is a photograph of a machine: it restores one environment and hauls that image around. What you want constantly branched is the world—memory, structured state, media references, and provenance as one unit—while the machine stays disposable. Compute checkpoints the machine; the data plane checkpoints the agent. Pixeltable owns the second job. Dashboards that only replay traces still leave you a bystander; the sibling argument is data plane, not dashboard.
One Catalog, Four Kinds of State#
Pixeltable does not run your agent loop or replace your sandbox. The loop lives in Pixelagent, LangGraph, or your code; ephemeral compute stays ephemeral. Pixeltable is the agent harness’s data plane: the durable world those layers orbit.
Minimal durable state#
Read the world before the next step#
Same pattern as the agent memory cookbook and stateful agents. Isolation for concurrent runs is session_id (and dirs)—many agents against one catalog, not a pretend copy-on-write object-store fork API.
Provenance without a fifth system#
Versioning and immutable snapshots are how you get “Tuesday back” without bolting on a separate provenance store—see time travel. For scripted rollbacks, use the CLI (pxt revert). Persist the catalog so sandboxes can die freely: Docker + PIXELTABLE_HOME.
Fleets Are More Sessions, Not More Machines#
A platform running a thousand concurrent agents does not need a thousand machine images of “the agent.” It needs a thousand scoped worlds—session logs and artifacts that cannot corrupt each other—while workers and sandboxes stay cheap to kill. Put identity in the catalog; let harness replicas pick up any session whose state still exists. Kill the model connection, the worker, or the sandbox mid-task and you lose a restart, not three days of context.
FAQ#
How is this different from “agents as data”?#
Same foundation. That post defines the session log as the system of record. This post places that log in the four-layer map (model / harness / sandbox / state) and contrasts disposable compute with a durable catalog boundary.
How is this different from “data plane, not dashboard”?#
The data-plane post argues observability without queryable state is bystander mode. This post answers where that state should live when sandboxes are designed to die.
Does Pixeltable sandbox my agents?#
No. Bring your own ephemeral compute. Pixeltable holds the world those sandboxes mount conceptually—by reading and writing tables—not by replacing the microVM.
Shouldn’t working files just live in S3?#
Media bytes often do. The failure mode is splitting conversation, embeddings, checkpoints, files, and run metadata across five systems with no shared snapshot. Keep one catalog as the boundary; point media columns at object storage when scale demands it.
Get Started#
An agent you cannot version is an agent you cannot trust. Treat the catalog as the agent, and replay, experiment, and recovery stop being afterthoughts.
- Deep dive: Agents as Data
- Sibling: Data plane, not dashboard
- Cookbook: agent memory pattern
- Ship an app: Starter Kit · GitHub
- Install:
pip install pixeltable· docs.pixeltable.com
See Also#
- Agents as data / session log
- Your agent needs a data plane, not a dashboard
- Pixeltable as the agent harness
- Building memory-powered stateful agents
- Versioning and time travel
- Who owns the multimodal data plane
- Pixelagent launch
- Operate the catalog from the terminal
- Docker + PIXELTABLE_HOME
- Storage architecture
