--- title: "Build a Multimodal AI App in 4 Steps Without Writing Infrastructure Code" date: "2026-05-20" author: "Pierre Brunelle" tags: - Developer Experience - AI Agents - Starter Kit - Tutorial - Multimodal AI - RAG - Video AI - CLI - Open Source description: "Install the AI Skill. Ask your assistant to build something. It scaffolds a complete project from the Pixeltable Starter Kit. Run it. Storage, orchestration, and retrieval are solved. No Airflow, no Pinecone, no glue code. The AI assistant IS the installer." url: "https://pixeltable.com/blog/build-multimodal-ai-app-four-steps" --- # Build a Multimodal AI App in 4 Steps Without Writing Infrastructure Code **Summary:** There is a new distribution model for developer tools: the AI coding assistant IS the installer. The human says what they want. The assistant knows how to build it (via the [Pixeltable Skill](https://github.com/pixeltable/pixeltable-skill)). It scaffolds a complete project (via [pixeltable-new](https://github.com/pixeltable/pixeltable-new)). The human runs it. No reading docs. No comparing frameworks. Four steps from zero to a running multimodal AI application with storage, orchestration, and retrieval solved. ## The Four Steps ### Step 1: Teach Your AI Assistant ```bash npx skills add pixeltable/pixeltable-skill ``` This installs the [Pixeltable Skill](https://github.com/pixeltable/pixeltable-skill) into your AI coding environment. It works with Cursor, Claude Code, Windsurf, Copilot, Cline, and any tool that supports the `SKILL.md` standard. The Skill teaches your assistant correct Pixeltable patterns, 25+ AI provider integrations, the top 5 anti-patterns to avoid, and a task router that maps what you want to build to the right code. For Claude Code, you can also install it as a native plugin: ```bash claude plugin add pixeltable-skill@pixeltable-skill ``` ### Step 2: Ask Your Assistant to Build Something Open your AI coding assistant and type something like: > "Build me a video search app with Pixeltable. I want to upload videos, extract frames, transcribe audio, and search across everything with text queries." The assistant, now equipped with the Pixeltable Skill, knows that the right tool is `pixeltable-new`. It knows the available templates. It knows to scaffold from the [Starter Kit](https://github.com/pixeltable/pixeltable-starter-kit) rather than writing LangChain + Pinecone + FFmpeg glue from scratch. ### Step 3: The Assistant Scaffolds Your Project The assistant runs: ```bash uvx pixeltable-new --template video-intel my-video-app ``` This fetches a complete project from the Starter Kit. No `pip install` required for the scaffolder itself (`uvx` runs it in an isolated environment). What you get: ```text my-video-app/ pyproject.toml # dependencies schema.py # tables, views, computed columns, indexes app.py # FastAPI endpoints + UI functions.py # custom UDFs .env.example # API keys template README.md # how to run ``` The `schema.py` is the key file. It defines your entire AI pipeline declaratively: ```python import pixeltable as pxt from pixeltable.functions.video import frame_iterator from pixeltable.functions import openai # Create tables videos = pxt.create_table('video_intel.videos', { 'video': pxt.Video, 'title': pxt.String, }, if_exists='ignore') # Extract frames at 1fps frames = pxt.create_view('video_intel.frames', videos, iterator=frame_iterator(video=videos.video, fps=1.0), if_exists='ignore') # Transcribe audio videos.add_computed_column( transcript=openai.transcriptions(audio=videos.video, model='whisper-1'), if_exists='ignore') # Embed for search videos.add_embedding_index('transcript', idx_name='transcript_idx', string_embed=openai.embeddings.using(model='text-embedding-3-small'), if_exists='ignore') ``` That schema IS your infrastructure. No Airflow DAG. No Pinecone cluster. No S3 bucket configuration. No FFmpeg scripts. Computed columns handle transcription, frame extraction, and embedding. The embedding index handles vector search. [The schema drives everything](/blog/schema-driven-infrastructure-ai). ### Step 4: Run It ```bash cd my-video-app uv sync # install dependencies cp .env.example .env # add your API keys uv run python schema.py # create tables, views, indexes uv run uvicorn app:app # start the API ``` That is it. You have a running video intelligence application. Upload a video and Pixeltable automatically extracts frames, transcribes audio, and indexes everything for search. Query with text and get relevant video segments back. ## What Just Happened Four commands. Zero infrastructure code. Here is what you did NOT have to do: | Infrastructure Concern | Traditional Stack | What You Did | | --- | --- | --- | | Media storage | Configure S3 bucket + IAM roles + upload scripts | Nothing. Pixeltable stores media in ~/.pixeltable | | Video frame extraction | Install FFmpeg + write extraction script + handle formats | frame_iterator(video=t.video, fps=1.0) | | Audio transcription | Whisper API client + retry logic + rate limiting | openai.transcriptions(audio=t.video) | | Embedding + vector search | Pinecone cluster + embedding pipeline + upsert logic | add_embedding_index('transcript') | | Orchestration | Airflow DAG + task dependencies + error handling | Nothing. Computed columns fire on insert | | Incremental updates | Custom checkpointing + "what changed?" logic | Built in. Add a video, only new rows process | | Caching + retries | Redis + exponential backoff + dead letter queue | Built in. Rate limiting + retry + cache automatic | | HTTP API | FastAPI routes + Pydantic models + file upload handling | Template ships it. Or: FastAPIRouter in app.py plus pxt service update | ## All Six Templates Video intelligence is one of six application templates. Each scaffolds a complete, runnable project: | Template | Command | What You Get | | --- | --- | --- | | Multimodal RAG | uvx pixeltable-new --template multimodal-rag my-kb | Upload docs, images, video, audio. Unified cross-modal search. LLM Q&A. | | Video Intelligence | uvx pixeltable-new --template video-intel my-app | Frame extraction, CLIP visual search, transcription, object detection. | | Agent | uvx pixeltable-new --template agent my-agent | Tool-calling agent with persistent memory and knowledge base. | | Audio Intelligence | uvx pixeltable-new --template audio-intel my-audio | Audio upload, Whisper transcription, sentence-level search. | | Content Pipeline | uvx pixeltable-new --template content-pipeline my-pipeline | Auto-detect media type, process, export to Parquet/SQL. | | Data Lab | uvx pixeltable-new --template data-lab my-lab | Image dataset management, CLIP search, detection, PyTorch/COCO export. | ## The Same App, Two Ways To understand what the Skill + scaffolder eliminates, here is what building the same video search app looks like with the traditional stack: ### The Traditional Stack ```bash # 1. Storage aws s3 mb s3://my-videos # write upload script, handle multipart, configure CORS # 2. Frame extraction pip install opencv-python ffmpeg-python # write extraction script: cv2.VideoCapture, frame sampling, format handling # ~80 lines of Python # 3. Transcription pip install openai # write Whisper client: chunking, retry logic, rate limiting # ~60 lines of Python # 4. Embeddings + vector search pip install pinecone-client # create index, write embedding pipeline, handle upserts, metadata # ~100 lines of Python # 5. Orchestration pip install apache-airflow # write DAG: task dependencies, error handling, retries, scheduling # ~120 lines of Python + YAML # 6. API pip install fastapi uvicorn python-multipart # write endpoints: upload, search, status # ~150 lines of Python + Pydantic models # Total: 6 services, ~510 lines of infrastructure code, 3+ config files ``` ### The Pixeltable Way ```bash # 1. Install skill (one time) npx skills add pixeltable/pixeltable-skill # 2. Scaffold uvx pixeltable-new --template video-intel my-app # 3. Run cd my-app && uv sync && uv run python schema.py && uv run uvicorn app:app # Total: 1 system, schema.py + app.py, 0 infrastructure code ``` The delta is not "slightly less code." It is a different category. One system instead of six. [Declarative instead of imperative](/blog/declarative-vs-imperative-ai-pipelines). [Incremental instead of full-reprocess](/blog/economics-of-incremental-ai). [Versioned instead of overwrite-in-place](/blog/pixeltable-versioning-time-travel). ## Why the Skill Is the Distribution Channel Here is the insight that changes how developer tools get adopted in 2026. The old distribution model: developer reads a blog post, visits your docs, evaluates your framework, writes code. Weeks from awareness to first line of code. The new distribution model: developer installs a Skill. The next time they ask their AI assistant to build something in your domain, the assistant reaches for your framework instead of LangChain. Minutes from install to running application. The Skill does not just teach the assistant *how* to use Pixeltable. It teaches the assistant *when* to use Pixeltable. The task router in our [SKILL.md](https://github.com/pixeltable/pixeltable-skill) maps intents to patterns: | If the User Asks For | The Skill Routes To | | --- | --- | | "Build a RAG app" | uvx pixeltable-new --template multimodal-rag + chunking + embedding index patterns | | "Analyze videos" | uvx pixeltable-new --template video-intel + frame_iterator + CLIP patterns | | "Build an agent with memory" | uvx pixeltable-new --template agent + memory table + tool-calling patterns | | "Process audio files" | uvx pixeltable-new --template audio-intel + transcription patterns | | "Build a data pipeline" | uvx pixeltable-new --template content-pipeline + export_sql patterns | | "Manage an ML dataset" | uvx pixeltable-new --template data-lab + annotation + export patterns | Without the Skill, the same AI assistant would reach for LangChain + Pinecone + Airflow. Not because those tools are better, but because they are what the model saw most during training. The Skill [deflects that prior](/blog/designing-software-for-llms-as-customers) and redirects to the idiomatic path. ## What Happens After Scaffolding The scaffolded project is a starting point, not a locked artifact. The Skill continues to help as you customize: - **Add a modality.** "Add image classification to my video app." The assistant adds a computed column with the right Hugging Face model, not a new service. - **Change a model.** "Switch from Whisper to Gemini for transcription." The assistant replaces one computed column. Existing data stays. [Only new rows reprocess](/blog/economics-of-incremental-ai). - **Add search.** "Let me search video frames by visual content." The assistant adds a CLIP embedding index on the frames view. One line. - **Deploy.** "Deploy this to production." The assistant knows about `pxt service update` locally, `pxt schema update app.py pxt://org:db` for hosted tables, Docker configs in the [Starter Kit](https://github.com/pixeltable/pixeltable-starter-kit), and the Cloud dashboard for hosted HTTP. Every modification follows the same pattern: add or change a computed column, view, or embedding index. The assistant knows these patterns because the Skill covers them. No new services. No new DAG tasks. No new glue code. ## Try It Now ```bash # 1. Install the skill (works with Cursor, Claude Code, Windsurf, Copilot) npx skills add pixeltable/pixeltable-skill # 2. Scaffold any template uvx pixeltable-new --template multimodal-rag my-kb # or: video-intel, agent, audio-intel, content-pipeline, data-lab # 3. Run cd my-kb && uv sync && cp .env.example .env # Add your API keys to .env uv run python schema.py && uv run uvicorn app:app ``` That is the entire onboarding flow. The AI assistant evaluated the framework for you, scaffolded the project for you, and the infrastructure handles itself. You just write the parts that are specific to your application. ## Resources - [Pixeltable Skill](https://github.com/pixeltable/pixeltable-skill): `npx skills add pixeltable/pixeltable-skill` - [pixeltable-new](https://github.com/pixeltable/pixeltable-new): `uvx pixeltable-new --template multimodal-rag my-kb` - [Pixeltable Starter Kit](https://github.com/pixeltable/pixeltable-starter-kit): 3 patterns + 6 application templates - [Pixeltable MCP Server](https://github.com/pixeltable/mcp-server-pixeltable-developer): 32 tools for LLM-powered exploration - [Designing Software for LLMs as Customers](/blog/designing-software-for-llms-as-customers): the five-stage framework - [What We Learned Shipping Templates for AI Agents](/blog/shipping-application-templates-for-ai-agents): the testing field report - [Five Things Pixeltable Does That Competitors Cannot](/blog/five-things-pixeltable-does-competitors-cant): the compounding capabilities - [Pixeltable Documentation](https://docs.pixeltable.com) - [Pixeltable on GitHub](https://github.com/pixeltable/pixeltable) - [Discord](https://discord.gg/QPyqFYx2UN)