---
title: "AI Agents & MCP: Give Your Agents Persistent Multimodal Memory"
description: "Build AI agents with durable memory and tool-calling capabilities using Pixeltable and Model Context Protocol (MCP). Store conversations, images, and documents as queryable tables that agents can read from and write to."
keywords:
  - AI agents MCP
  - model context protocol
  - agent memory
  - tool calling AI
  - persistent agent memory
  - MCP server python
  - agentic AI pipeline
  - multimodal agent
complexity: "intermediate"
estimated_time: "30 min"
url: "https://pixeltable.com/use-cases/ai-agents-mcp"
---

# AI Agents & MCP: Give Your Agents Persistent Multimodal Memory

Build AI agents with durable memory and tool-calling capabilities using Pixeltable and Model Context Protocol (MCP). Store conversations, images, and documents as queryable tables that agents can read from and write to.

## Prerequisites

- Familiarity with AI agents and LLMs
- Basic Python programming

## The Problem

AI agents lose context between sessions, struggle with multimodal data, and have no durable memory. Building tool-calling agents requires wiring together separate systems for memory, retrieval, and data storage. MCP servers need a backend that can handle structured data, media, and semantic search.

## The Solution

Pixeltable serves as the persistent memory and data layer for AI agents. Tables store conversations, media, and structured data. Computed columns handle embeddings and AI inference. MCP integration exposes tables as tools that agents can query, insert into, and search semantically.

## Implementation

### Agent Memory Table

Create a persistent memory store for agent conversations and context.

```python
import pixeltable as pxt
from pixeltable.functions.huggingface import sentence_transformer

# Persistent agent memory
memory = pxt.create_table('app.agent_memory', {
    'session_id': pxt.String,
    'role': pxt.String,        # 'user', 'assistant', 'tool'
    'content': pxt.String,
    'timestamp': pxt.Timestamp,
    'metadata': pxt.Json,
})

# Semantic search over memory
memory.add_embedding_index(
    'content',
    string_embed=sentence_transformer.using(
        model_id='sentence-transformers/all-MiniLM-L6-v2'
    )
)

# Agents can now search their own history
relevant = memory.select(
    memory.content, memory.role, memory.session_id
).order_by(
    memory.content.similarity(string='what did we discuss about pricing?'),
    asc=False
).limit(10)
```

Unlike in-memory chat history, this memory persists across sessions and supports semantic retrieval.


### Knowledge Base

Give agents access to a searchable knowledge base with documents and media.

```python
from pixeltable.functions.document import document_splitter

# Agent knowledge base: documents, images, and metadata
knowledge = pxt.create_table('app.knowledge', {
    'document': pxt.Document,
    'title': pxt.String,
    'category': pxt.String,
})

# Automatic chunking and indexing
chunks = pxt.create_view(
    'app.knowledge_chunks',
    knowledge,
    iterator=document_splitter(
        document=knowledge.document,
        separators='sentence',
        limit=512
    )
)

chunks.add_embedding_index(
    'text',
    string_embed=sentence_transformer.using(
        model_id='sentence-transformers/all-MiniLM-L6-v2'
    )
)

# Reusable retrieval query for agents
@pxt.query
def search_knowledge(question: str, n: int = 5):
    return chunks.select(
        chunks.text, chunks.title
    ).order_by(
        chunks.text.similarity(string=question), asc=False
    ).limit(n)
```

Agents get grounded answers from your documents. New documents are indexed automatically.


### MCP Server

Expose Pixeltable tables as MCP tools for any agent framework.

```python
from pixeltable.mcp import create_mcp_server

# Create an MCP server from your Pixeltable tables
server = create_mcp_server(
    name="knowledge-agent",
    tables={
        'memory': memory,
        'knowledge': knowledge,
    },
    queries={
        'search': search_knowledge,
    }
)

# Agents (Claude, GPT, etc.) can now:
# - Insert into memory: {"tool": "memory.insert", "args": {...}}
# - Search knowledge: {"tool": "search", "args": {"question": "..."}}
# - Query tables: {"tool": "memory.select", "args": {...}}

# Run the MCP server
if __name__ == "__main__":
    server.run()
```

Any MCP-compatible agent (Claude Desktop, GPT, custom) can use your Pixeltable tables as tools.


### Multimodal Tools

Let agents work with images, video, and audio, not just text.

```python
from pixeltable.functions.huggingface import clip

# Multimodal asset table
assets = pxt.create_table('app.assets', {
    'image': pxt.Image,
    'caption': pxt.String,
    'tags': pxt.Json,
    'uploaded_by': pxt.String,
})

# Visual search index
assets.add_embedding_index(
    'image',
    image_embed=clip.using(
        model_id='openai/clip-vit-base-patch32'
    )
)

# Agent tool: find images by description
@pxt.query
def find_images(description: str, n: int = 5):
    return assets.select(
        assets.image, assets.caption, assets.tags
    ).order_by(
        assets.image.similarity(string=description), asc=False
    ).limit(n)

# Now agents can: "Find me photos of the product launch event"
# and get back actual images with metadata
```

Agents aren't limited to text; they can search and retrieve images, video, and audio through the same interface.


## Benefits

- Persistent multimodal memory that survives across sessions
- Semantic search over agent history and knowledge bases
- MCP integration: works with Claude, GPT, and any MCP-compatible agent
- Multimodal tools: agents work with images, video, audio, and documents
- Automatic indexing keeps agent knowledge always current

## Use Cases

- Customer support agents with persistent context
- Research assistants with document retrieval
- Content creation agents with media search
- DevOps agents with log analysis and monitoring
- Sales agents with CRM data and document access

## Performance


| Metric | Value | Description |

| --- | --- | --- |

| Setup Time | 15 min | To working MCP server with memory + search |

| Code Reduction | 90% | vs building custom agent infrastructure |

## Requirements

- Python 3.9+
- MCP-compatible agent (Claude Desktop, etc.)
- API keys for embedding models

## Resources

- [Pixeltable MCP Servers](https://pixeltable.com/blog/pixeltable-mcp-servers) - Building MCP servers with Pixeltable
- [AI Agents with Persistent Memory](https://docs.pixeltable.com) - Architectural patterns for agent memory
- [Model Context Protocol](https://modelcontextprotocol.io) - Official MCP specification