The easiest way to cut your labeling bill in half isn't to negotiate with your vendor. It's to stop sending them redundant data. Here's how to use semantic deduplication to prune your datasets without losing signal.
The Duplicate Problem#
In video datasets (dashcams, security footage, drones), "duplicates" aren't just exact file matches. They are:
- Sequential Frames: Frame 100 and Frame 101 of a car stopped at a red light are 99.9% identical.
- Burst Mode: 10 photos taken in 1 second.
- Recurring Backgrounds: The same empty warehouse floor seen for 8 hours.
Standard hash-based deduplication (MD5/SHA) fails here because the pixels are slightly different. You need semantic deduplication.
Semantic Deduplication with Embeddings#
Embeddings (from models like CLIP or ResNet) map images to vectors. Similar images map to nearby vectors. By calculating the distance between vectors, we can find semantic duplicates.
Step 1: Generate Embeddings#
First, we compute embeddings for our entire dataset. In Pixeltable, this is one line.
Step 2: Find Clusters#
We can use a simple clustering approach or a nearest-neighbor search to find groups of similar images.
Step 3: Prune the Dataset#
We can create a "curated" view that filters out the duplicates, keeping only the most representative image from each cluster.
Visual Querying: Finding the Needle#
Sometimes you don't want to remove duplicates. You want to find more of a specific rare case. "Show me all images that look like this broken connector."
Curate, Don't Just Collect#
Data volume is easy. Data quality is hard. By using embeddings to understand the content of your data, you can build datasets that are smaller, higher quality, and cheaper to label, all while training better models.
