PixelAssist Retrospective: ~2,100 Lines of TypeScript vs. ~40 Lines of Pixeltable

Summary: We shipped PixelAssist (the AI help chat on pixeltable.com) as a single Next.js API route in TypeScript. It works, but the core file grew to ~1,330 lines, the route handler to ~780, and every anti-hallucination patch added more glue: OPML parsing, URL allowlisting, a name harvester, a PascalCase suffix rule, an audit-retry loop. A dedicated cleanup pass pulled ~100 lines of dead code and duplication out, extracted shared helpers (collectPxtVars, CODE_BLOCK_RE, isPythonishLang), converted a suffix Map to a Set, and collapsed redundant regex passes; the total still sits around ~2,100 lines because the complexity is structural, not stylistic. This post compares what that same feature looks like written natively on Pixeltable: the core pipeline collapses to roughly 40 lines because Pixeltable already provides what we hand-rolled: tables, embedding indexes, tool calling, provenance, and upstream rate-limit / retry handling. This is not a rewrite pitch. It is a retrospective about PixelAssist specifically: which layer of the stack should own your RAG logic, what Pixeltable does not solve for a chat widget, and whether a TypeScript SDK would let us stay in Next.js.

What PixelAssist Is, In One Paragraph#

PixelAssist is an embedded chat widget on pixeltable.com. Users can ask "how do I build a RAG pipeline?" and the assistant replies grounded in the Pixeltable docs, skill references, and the public API OPML. It runs on Groq with openai/gpt-oss-120b, supports tool calling to fetch specific references, audits every Python code block the model emits against the public API, and streams tokens back to the browser as NDJSON frames. The frontend repo is closed-source. The line counts and design choices in this post come from the implementation as we ship it internally.

Architecture Today: TypeScript-First#

PixelAssist lives entirely inside a Next.js API route. The chat handler is /api/help/chat/route.ts, shared types and helpers are in src/lib/help-chat.ts. On every turn the handler fetches references (llms.txt, skill pages, public_api.opml), runs a tool-call loop against Groq, audits the generated code, and streams NDJSON frames back to the browser.

text

The important thing about this diagram: PixelAssist is fundamentally a RAG pipeline, but it has no persistent store. Every request refetches references, rebuilds the inventory in memory, and throws it all away when the function returns. Retrieval, grounding, auditing: all of it is reimplemented inside a single stateless handler.

What We Actually Built in TypeScript#

The PixelAssist route went through five iterations as failure modes surfaced. Each iteration added a guardrail:

v1: Groq with static prompt. Hallucinated API names freely. Added llms.txt fetch and a URL allowlist.
v2: static context injection. Fetched llms.txt, SKILL.md, and public_api.opml on every request, cached for 1h, stuffed into the system prompt as a "DATA ONLY" block. Better URL grounding, but the model still invented function signatures.
v3: tool calling. Replaced static context with fetch_skill_reference, fetch_doc_page, lookup_api_symbol tools. Switched to openai/gpt-oss-120b for reliable tool routing. Fewer URL hallucinations, but the model occasionally emitted code without retrieving first.
v4: post-generation audit. Parsed the OPML into a classified inventory (moduleCallables vs classMethods), extracted Python code blocks, scanned every pxt.X and <var>.method( call, flagged anything not in the inventory, and injected a correction prompt on the next iteration.
v5: harvest + three-tier acceptance. Turned out the OPML does not list every legitimate type alias (pxt.String, pxt.Document). The audit was over-rejecting valid names and the model capitulated with "I cannot express this." Added a per-turn name harvester that pulled verified surface from tool results, plus a PascalCase suffix rule for re-exports. Audit passes once the model fetches the right reference.
v6: cleanup + dedup pass. Once the pipeline was stable, we did a dedicated audit of help-chat.ts and route.ts. Removed the static SKILL.md fetch path (dead after moving to tool-call retrieval), deleted the skillGuide field, inlined the extractPublicApiSymbols wrapper, extracted shared helpers (collectPxtVars, CODE_BLOCK_RE, isPythonishLang) used by both the harvester and the auditor, replaced a suffix Map with a Set, and made redactSecrets do a single pass per pattern. Net: ~40 lines out of help-chat.ts, no behavior change, the system prompt no longer references stale <pixeltable_skill> tags. Worth doing, but a reminder that none of it would exist in a data-plane that owned the retrieval.

The file structure at the end of this:

File	Lines	What it does
`src/lib/help-chat.ts`	~1,330	Types, limits, system prompt, tool definitions, context fetcher (llms.txt + OPML), sanitizers, OPML parser, Python block extractor, API inventory builder, name harvester, auditor, link rewriter
`src/app/api/help/chat/route.ts`	~780	Auth / rate-limit / sanitize, tool-call loop, audit-retry logic, streaming NDJSON frames, guest-quota cookies, security headers

Not all 2,100 lines are retrieval logic. A generous breakdown:

Concern	Lines (approx.)	Could Pixeltable replace it?
Tool-call loop + retrieval	~200	Yes: `invoke_tools()`
OPML parsing + API inventory	~120	Yes: tables are the inventory
Name harvester + audit + retry	~180	Yes: retrieval tool IS the allowlist
URL allowlist + link rewriter	~140	Yes: same reason
Context fetcher + cache + sanitize	~180	Yes: `refs.insert()` ingests once
Streaming NDJSON framing	~120	No: Pixeltable returns final values
Prompt-injection defenses	~150	No: still needed in FastAPI
Rate limit + guest quota cookies	~80	No: still needed
System prompt + tool schemas + types	~400	Partial: declarative pipeline replaces most of this

The honest math: roughly 820 lines are pipeline logic Pixeltable eliminates; another ~350 are security and streaming concerns that stay wherever the handler runs. The headline "40 lines" below refers to the pipeline portion: apples to apples.

The Pixeltable-Native Architecture (PixelAssist Only)#

Here is the same PixelAssist feature drawn with Pixeltable as a first-class backend. The Next.js widget stays, the tiny FastAPI handler exists only to translate HTTP into a table insert, and everything PixelAssist-specific (references, chunks, embeddings, the retrieval tool, chat rows) is Pixeltable.

text

A note on the Postgres box. Pixeltable is a multi-user database (concurrent writers, isolation, snapshots, the usual expectations), and under the hood it uses Postgres as its storage engine. That is an implementation detail: the Python SDK does not expose Postgres as a user-facing surface, and the RDBMS is not pluggable today. You talk to Pixeltable (create_table, insert, select, @pxt.query, add_computed_column), not to SQL.

Every concern PixelAssist hand-rolls in TypeScript is a first-class primitive here: the knowledge base is a table, the retrieval tool is a @pxt.query, the audit is "the retrieval tool returned real rows," the chat history is the chat table, and re-running with a new model is one call. The FastAPI handler is three statements because Pixeltable exposes the right primitives directly.

PixelAssist in ~40 Lines of Pixeltable#

This is the exact same feature (ingest references, chunk, embed, tool-call retrieval, LLM answer, grounded output) expressed declaratively. It is taken directly from patterns in the Pixeltable skill reference, specifically the tool-calling agent workflow.

python

And the FastAPI handler:

python

That is the feature. Everything else (logging every turn, replaying with a new model, searching past conversations, tracking per-cell errors) is free because each row is a table row.

Concern-by-Concern: Side-by-Side#

The clean way to evaluate this is not by line count; it is by asking, for each concern we built into the TypeScript version, whether Pixeltable makes it structurally unnecessary or just differently-implemented.

Concern	TypeScript version (what we did)	Pixeltable version
URL hallucination	Fetch `llms.txt`, parse URLs, build `allowedUrls` Set, grow it from tool results, run a streaming `MarkdownLinkRewriter` that strips anything not in the set.	The retrieval tool `search_docs` returns rows from the `refs` table. The model physically cannot cite a URL that is not a `refs.url` value. The allowlist is the column.
API hallucination	Parse `public_api.opml` into a classified inventory, extract Python blocks, track `pxt`-typed variables, check module-style and method-style calls against three tiers: OPML exact, harvested names, PascalCase suffix.	The model calls `@pxt.query` functions that return real rows. If a function does not exist, the code does not compile. The "API inventory" is the Python module itself.
Retrieval-before-code	Count `retrievalCallsMade`, detect Python blocks in output, inject a synthetic correction user-message if code was emitted without retrieval, retry once, fall back to a warning footer.	Retrieval is a computed column: `chat.add_computed_column(retrieved=invoke_tools(...))`. It runs before `answer` by construction of the dependency graph.
Ingest new references	Add URL, wait for the 1-hour cache to expire, serve stale in the meantime. Cache lives in Next.js memory so each instance warms separately.	`refs.insert([{'url': '...', 'content': 'path/to/new.md'}])` → chunking + embedding run incrementally. Previous rows are untouched.
Provenance	Not stored anywhere. Logs are the only trace.	`chat` rows have `plan`, `retrieved`, `answer` columns. `chat.describe()` shows the full dependency graph. Time-travel is free.
Analytics	Grep logs, hope you kept enough detail.	`chat.where(chat.answer.errortype != None).collect()`. A dashboard query.
Re-run with a new model	Redeploy the route, replay historical prompts you hopefully saved somewhere.	`chat.recompute_columns(['answer'])` after changing the model string.
Chat history	Store separately, manage sessions by hand.	The `chat` table is the history. `session_id` filters it.
AI provider integration	`npm i groq-sdk`, manage the SDK shape, handle errors, add each new provider manually. Our current route has zero retry logic: a Groq 429 or 5xx becomes a 502 to the user.	25+ providers already wrapped in `pixeltable.functions.*` with consistent signatures.
Rate limits, retries, transient failures from the LLM provider	Not handled. One `fetch` to Groq per turn; if Groq 429s we surface the error. Batching / backoff / per-request fallback would be new code we haven't written.	Handled at the column layer. `pixeltable.functions.groq.chat_completions` (and the equivalents for OpenAI / Anthropic / Gemini / etc.) manage rate-limit retries with exponential backoff and respect provider headers, batch requests efficiently, and contain failures per row via `errortype` / `errormsg` columns so one 429 doesn't fail the whole pipeline. Cross-provider fallback composes as a second computed column that `coalesce`s the primary result with a backup model.
Streaming tokens to browser	Custom NDJSON framer, `ReadableStream`, stream-safe `<think>` stripper, streaming link rewriter.	Not natively. Pixeltable computed columns return one final value. Either drop streaming or call the LLM directly from FastAPI for the streamed turn and use Pixeltable only for logging and retrieval (hybrid).
Prompt-injection defense	Unicode scrub, credential redaction, tag neutralization, same-origin check, DOMPurify + `html-react-parser` allowlist on the client.	Same. These apply regardless of backend. Move to FastAPI middleware of equivalent length.
Rate limit + guest quota	In-memory limiter + signed cookie.	Same. FastAPI middleware; storage can be Redis or a Pixeltable table of its own.

Of the thirteen concerns: eight become either trivial or structurally impossible in the Pixeltable version (the seven retrieval / provenance items plus upstream rate-limit and retry handling, which our TypeScript route does not handle at all today). Three stay the same regardless of backend (prompt-injection defense, end-user rate limiting, guest quota). Two (streaming, provider diversity at the browser edge) require an explicit design choice, covered below.

The Retrieval Invariant: Why URL Hallucination Goes Away#

This one is worth dwelling on because it is the single most valuable structural property we lost by staying in TypeScript.

In our TS implementation, the model can output any URL it wants. We defend after the fact: parse llms.txt into an allowlist, grow it from tool results, run every emitted link through a streaming rewriter, strip anything that is not on the list. It works. But it is a classic "forbidden list" problem: you need to know the answer before you can check it.

In Pixeltable, search_docs is a @pxt.query. Its return shape is select(chunks.url, chunks.text, sim). When the LLM calls the tool, it receives rows from the refs table. The URLs in those rows are, by construction, the only URLs that exist in the knowledge base. The model's job is to cite them, not invent them. If it ignores the retrieved context and writes a URL from pre-training, we still catch it, but now "valid URL" has a positive definition (it is a value in a column), not a negative one (it is not in a hand-maintained deny-rewriter).

The same argument applies to API hallucinations. Our TS auditor has three tiers: OPML exact match, harvested names from tool results, PascalCase suffix match. Each tier was added because the previous one over-rejected valid code. In Pixeltable, the "API" is a Python module: if the function exists, it exists; if it does not, the code raises AttributeError at add_computed_column definition time, long before a user sees the answer.

What Pixeltable Does Not Replace (For PixelAssist)#

Even scoped to PixelAssist alone, some concerns do not disappear when you move the pipeline into Pixeltable. They just live in the FastAPI handler in front of it.

Area	Status	Why
Token streaming to the browser	Still needed	Pixeltable computes `answer` as a single column value. Interactive token streaming requires calling the LLM directly from the handler for the live turn (hybrid pattern below).
Prompt-injection defense (unicode scrub, tag neutralization, DOMPurify on the client)	Still needed	These are HTTP/client concerns that apply regardless of backend. Move to FastAPI middleware of equivalent length.
Rate limit + guest quota cookie	Still needed	Per-IP and per-session throttling for the chat endpoint. Lives in the handler; storage can be Redis or a small Pixeltable table.
Auth on the chat endpoint (when enabled for signed-in users)	Still needed	HTTP-layer concern. The handler validates the session before inserting into `chat`.

The cutoff is clean: everything that is pipeline (retrieval, grounding, provenance, re-run, history) moves. Everything that is transport (streaming, rate limit, injection scrub, auth) stays in the handler. That is the right seam.

The Streaming Tradeoff, Honestly#

Our current NDJSON streaming produces visible text as tokens arrive from Groq. Users see "thinking → first word → second word" in under a second. Pixeltable computes answer as a single column value. chat.insert() blocks until the whole chain finishes, then the handler reads one row.

Three ways to handle this:

Give up streaming. Show a loader; the answer appears when done. Acceptable for documentation Q&A where 2–4s is tolerable.
Hybrid. Stream the LLM call directly from FastAPI for the live turn; write the final row to Pixeltable asynchronously for logging, retrieval corpus, and re-runs. You keep 100% of the UX and about 70% of the code savings.
Insert-then-poll. Handler inserts the row, returns a row ID. Client polls chat.where(id == X).select(answer). Simpler for batch workflows, worse for chat.

For PixelAssist we would pick option 2. The pipeline definition does not change; the handler does two things in parallel instead of one.

Would a TypeScript SDK for Pixeltable Change the Answer?#

Yes, and it is on the roadmap. The plan is to rebuild the full Pixeltable surface in TypeScript, not a read-only subset: table DDL, inserts, queries, @pxt.query, UDFs, computed-column pipelines, and the provider integrations. It does not exist today, but when it lands PixelAssist collapses into a single Next.js app with Pixeltable as the backing store: no FastAPI service in front, no language split.

Layer	Today (Python only)	With the TS SDK (planned)
Schema definition / DDL	Python	TypeScript: typed table schemas generated from the catalog
Insert / update / delete (`chat.insert(row)`)	Python	TypeScript, typed against the schema
Query (`select`, `where`, `similarity`, `collect`)	Python	TypeScript query builder: biggest ergonomic win
`@pxt.query` (retrieval tools)	Python	TypeScript: `search_docs` defined in the same file as the route
UDFs (`@pxt.udf`)	Python	TypeScript for TS-native transforms; Python still available where the AI ecosystem lives (numpy, PIL, HF)
AI provider integrations (Groq, OpenAI, Anthropic, …)	Python	TypeScript wrappers with the same semantics (retries, backoff, per-row error containment)
Computed column pipelines (`chat.add_computed_column(plan=…)`)	Python	TypeScript: pipeline defined next to the handler

Concretely for PixelAssist, once the TS SDK ships the Next.js route looks like this, with no separate Python service and no JSON translation layer:

typescript

The pipeline itself (refs, chunks, the embedding index, search_docs, the plan/retrieved/answer computed columns) is defined once (in TS, in the same repo, next to the route). The retrieval invariant survives the language switch: search_docs still returns rows from chunks, so the URL allowlist is still the column by construction. The post-hoc OPML parser, name harvester, link rewriter, and audit-retry loop stay deleted regardless of which language the handler is written in. That is the point.

Decision Framework#

Stay in TypeScript when#

The feature is a thin wrapper over an external LLM API with no knowledge corpus to maintain (a one-shot classifier, a "summarize this input" button).
Streaming tokens to the browser is the core UX, and you do not need to log, replay, or re-rank the output.
The team has no Python deployment target and will not add one for this feature alone.
The scope fits in a single route handler and is unlikely to grow guardrails.

Move to Pixeltable when#

The feature is RAG, tool-calling agents, stateful memory, or any pipeline with multiple AI stages.
You want every past turn auditable, re-runnable with a new model, and searchable by similarity.
You need to ingest new references incrementally without redeploying.
You care about structurally eliminating URL and API hallucination rather than chasing each failure with another post-hoc check.
You already run Python somewhere, or are willing to add a small FastAPI service next to Next.js.

Hybrid when#

The UX requires token streaming AND you want the provenance / auditability benefits. Stream from the handler, persist to Pixeltable.
The chat endpoint already exists in TS and is stable. Only the pipeline moves; the HTTP layer stays put.

What We Would Do Differently#

Given the choice to redo PixelAssist with what we now know:

Put the pipeline in Pixeltable. Ingest llms.txt, the skill references, and public_api.opml as rows in a refs table. Build the chat pipeline as four computed columns (plan, retrieved, answer, and a hybrid streamed_answer for logging).
Keep a thin handler in front. FastAPI or Next.js, depending on TS SDK availability. Streaming stays in the handler (direct to Groq for the live turn); we fire-and-forget a row into Pixeltable for logging and replay.
Stop reinventing the allowlist. The retrieval function returns real rows from a real table. URL and API hallucinations become structurally impossible, not post-hoc-caught.
Treat the chat table as the audit log. Every failed audit, every unverified symbol, every ambiguous answer is a row we can SELECT and learn from, without inventing a separate logging pipeline.

This is not a rewrite pitch. It is the natural place for this specific code to live given what it actually does. The TS version works today, and the lesson is about where we put the next AI feature that is still living as an ad-hoc tool loop in an API route.

Takeaways#

The "2,100 vs 40" comparison is real but narrow. Pipeline logic collapses ~8–10×. Transport concerns (streaming, rate limit, prompt-injection defense, auth) stay wherever the handler runs: for PixelAssist that is ~350 lines either way.
Declarative data pipelines turn anti-hallucination from a patch loop into an invariant. Declarative beats imperative when the thing you are trying to prove is "this output references only things that exist."
The retrieval tool is the allowlist. A @pxt.query that returns rows from a table cannot return a value that is not in that table. That is the one structural property our TS auditor cannot give us no matter how many tiers we add.
A TypeScript SDK for Pixeltable is on the roadmap: not a read-only subset, the full surface (DDL, queries, @pxt.query, UDFs, providers, computed columns). When it lands, PixelAssist becomes a single Next.js app with Pixeltable as the backing store; no Python service in front, no JSON translation layer, and the retrieval invariant is language-independent.
Six iterations of guardrails (OPML parsing, URL allowlist, name harvester, PascalCase suffix rule, audit-retry loop, dedup pass) are six symptoms of one problem: there was no data-plane for the knowledge base.

Get Started with Pixeltable#

Pixeltable Starter Kit: production-ready FastAPI + React app with multimodal upload, cross-modal search, and tool-calling agents.
Pixeltable Skill: the reference bundle PixelAssist retrieves from. Useful whether your LLM tool is a chat widget or an editor.
Documentation: full API reference, tutorials, deployment guides.
Pixeltable Core Concepts: tables, computed columns, views, and embedding indexes explained.
GitHub: star the repo, file issues, contribute.