---
title: "MedDossier: Clinical Intake on One Audio + PDF + Image Row"
date: "2026-09-17"
author: "Pierre Brunelle"
tags:
  - Healthcare
  - Multimodal AI
  - TableModel
  - Document
  - Audio
  - Pixeltable Cloud
  - Triage
description: "Dictation, lab PDF, and a scan on the same row. Whisper and Gemini fill columns. A UDF flags contradictions. Not a diagnostic device. Not three microservices."
url: "https://pixeltable.com/blog/meddossier-clinical-intake-triage"
---

# MedDossier: Clinical Intake on One Audio + PDF + Image Row

**Summary:** MedDossier is clinical intake as a table row: doctor dictation, lab PDF, scan image. Whisper transcribes the audio. Gemini reads the PDF and the image. A UDF (or a third Gemini call) compares the three and writes `triage_level`. You do not store “a list of files” and hope a pipeline joins them later. Same shape as [ClaimBot](/blog/claimbot-multimodal-fnol-triage) — different job. `pip install pixeltable`.

 
## The Product

 

 - Create a case with `case_id`

 - Attach `pxt.Audio`, `pxt.Document`, `pxt.Image`

 - Fill notes, lab summary, scan notes on insert

 - Flag `routine` / `priority` / `urgent` and a contradiction bit

 - Search similar scans (CLIP) and similar notes (MiniLM)

 

 
ClaimBot is FNOL: photo, memo, report, insurance triage. This post is the clinical analog — still one heterogeneous row, not an image array.

 
## The Stitch You Delete

 
One bucket for WAV. One for PDFs. One for DICOM exports you flattened to PNG. Three workers. A chart field you type by hand. Pixeltable is one insert. Delete the case; derived cells are not orphans in two other systems.

 
## The Receipt

 
```python
import pixeltable as pxt
from pixeltable.catalog.model import Column, EmbeddingIndex
from pixeltable.functions import gemini, whisper
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

COMPARE_PROMPT = (
 'Compare the dictation transcript, the lab-PDF summary, and the scan notes. '
 'Reply with one token: routine, priority, or urgent, then CONTRADICTION or OK, '
 'then a short reason. Do not diagnose. Do not invent a radiology model name.'
)

@pxt.udf
def parse_triage(raw: str) -> dict:
 blob = raw.lower()
 level = 'priority'
 if 'urgent' in blob:
 level = 'urgent'
 elif 'routine' in blob:
 level = 'routine'
 return {
 'triage_level': level,
 'contradiction': 'contradiction' in blob,
 'reason': raw,
 }

class Cases(TableModel, name='cases'):
 case_id = Column(pxt.String, primary_key=True)
 patient_id: pxt.String
 dictation_audio: pxt.Audio
 lab_report: pxt.Document
 scan_image: pxt.Image

 dictation_notes = whisper.transcribe(dictation_audio, model='base')
 lab_summary = gemini.generate_content(
 [lab_report, 'Summarize lab values and flags in one short paragraph.'],
 model='gemini-2.5-flash',
 )
 scan_notes = gemini.generate_content(
 [scan_image, 'Describe visible findings in the scan image. Do not diagnose.'],
 model='gemini-2.5-flash',
 )
 compare_raw = gemini.generate_content(
 [dictation_notes, lab_summary, scan_notes, COMPARE_PROMPT],
 model='gemini-2.5-flash',
 )
 triage = parse_triage(compare_raw)

 __indexes__ = [
 EmbeddingIndex(column=dictation_notes, embedding=text_embed, name='md_dictation_idx'),
 EmbeddingIndex(column=scan_image, embedding=clip_embed, name='md_scan_clip_idx'),
 ]

@pxt.query
def similar_scans(ref_scan: pxt.Image, top_k: int = 5):
 sim = Cases.scan_image.similarity(ref_scan)
 return (
 Cases.order_by(sim, asc=False)
 .limit(top_k)
 .select(Cases.case_id, Cases.triage, Cases.scan_notes)
 )

api = FastAPIRouter(name='meddossier')
api.add_insert_route(
 Cases,
 path='/cases',
 inputs=[
 Cases.dictation_audio,
 Cases.lab_report,
 Cases.scan_image,
 Cases.case_id,
 Cases.patient_id,
 ],
)
api.add_query_route(path='/similar-scans', query=similar_scans)
```

 
Gemini on the image is a description column, not a shipped radiology API. CLIP on `scan_image` uses `embedding=` so a reference shot and a text query can share the index — same form as [SnapCatalog](/blog/snapcatalog-product-visual-search). Same file on Cloud: db → schema → service.

 
## What This Is Not

 
Not a diagnostic device. Not medical advice. Not HIPAA-as-a-product — your BAA, retention, and EHR stay where they are. Not ClaimBot; that post is insurance FNOL. The multimodal row is intake evidence, not a finding.

 
## People Also Ask

 
**Why not `pxt.Image[]`?** One dictation, one lab PDF, one scan. Arrays hide the join. Many images are a child table.

 
**Do I chunk the lab PDF?** Only for passage citations. That receipt is [DocuVision](/blog/docuvision-pdf-chart-qa). Start with Gemini on `lab_report`.

 
**How do I go to Cloud?** Same `app.py`. `PIXELTABLE_API_KEY`, `pxt://org:db`, then `pxt db update` → `pxt schema update` → `pxt service update`.

 
**Is this an AI automation workflow?** Yes. Insert audio, PDF, and scan; intake columns run on the row. The category is [AI automation workflow](/blog/ai-automation-workflow).

 
## Three Types, Clinical Job

 
Declare the case. Apply the file. Insert audio, PDF, image. Read `triage`. Keep the intake. Delete the three-bucket stitch.

 

 - **[Pixeltable on GitHub](https://github.com/pixeltable/pixeltable)**: `pip install pixeltable`

 - **[ReelForge](/blog/reelforge-podcast-chaptering-viral-clips)** — podcast chapters

 - **[LectureSync](/blog/lecturesync-slide-lecture-qa)** — slides + lecture video

 - **[AdRadar](/blog/adradar-creative-fatigue-visual-search)** — creative fatigue

 - **[ClaimBot](/blog/claimbot-multimodal-fnol-triage)** — FNOL triage on the same row shape

 - **[Pixeltable documentation](https://docs.pixeltable.com)**