---
title: "Stop Labeling Duplicates: Semantic Deduplication for Multimodal Datasets"
date: "2025-01-31"
author: "Marcel Kornacker"
tags:
  - Data Curation
  - Embeddings
  - Cost Optimization
  - Computer Vision
description: "How to use embeddings to find and remove near-duplicates from your training data, saving 30-50% on labeling costs."
url: "https://pixeltable.com/blog/semantic-deduplication-curation"
---

# Stop Labeling Duplicates: Semantic Deduplication for Multimodal Datasets

The easiest way to cut your labeling bill in half isn't to negotiate with your vendor. It's to stop sending them redundant data. Here's how to use semantic deduplication to prune your datasets without losing signal.
 

 
## The Duplicate Problem

 
 

 In video datasets (dashcams, security footage, drones), "duplicates" aren't just exact file matches. They are:
 

 
 

 - **Sequential Frames:** Frame 100 and Frame 101 of a car stopped at a red light are 99.9% identical.

 - **Burst Mode:** 10 photos taken in 1 second.

 - **Recurring Backgrounds:** The same empty warehouse floor seen for 8 hours.

 

 

 Standard hash-based deduplication (MD5/SHA) fails here because the pixels are *slightly* different. You need **semantic deduplication**.
 

 
## Semantic Deduplication with Embeddings

 

 Embeddings (from models like CLIP or ResNet) map images to vectors. Similar images map to nearby vectors. By calculating the distance between vectors, we can find semantic duplicates.
 

 
### Step 1: Generate Embeddings

 
 

 First, we compute embeddings for our entire dataset. In Pixeltable, this is one line.
 

 
```python
import pixeltable as pxt
from pixeltable.functions.huggingface import clip

# Create table and add images
t = pxt.create_table('dataset', {'image': pxt.Image})

# Add computed column for embeddings
t.add_computed_column(embedding=clip.image_embedding(t.image))
```

 
### Step 2: Find Clusters

 
 

 We can use a simple clustering approach or a nearest-neighbor search to find groups of similar images.
 

 
```python
# Find images that are very close to each other
# (Pseudo-code for logic you can implement with UDFs)
duplicates = t.select(t.image).order_by(t.embedding).limit(100)
```

 
### Step 3: Prune the Dataset

 
 

 We can create a "curated" view that filters out the duplicates, keeping only the most representative image from each cluster.
 

 
```python
# Keep one representative per near-duplicate cluster
t.add_computed_column(is_representative=mark_cluster_representative(t.embedding))

curated = pxt.create_view(
 'curated_dataset',
 t,
 predicate=t.is_representative,
)
```

 
## Visual Querying: Finding the Needle

 

 Sometimes you don't want to remove duplicates. You want to find *more* of a specific rare case. "Show me all images that look like this broken connector."
 

 
```python
# Visual similarity search
broken_connector = 'path/to/example.jpg'

results = t.select(t.image).order_by(
 t.embedding.similarity(image=broken_connector)
).limit(50)
```

 
## Curate, Don't Just Collect

 

 Data volume is easy. Data quality is hard. By using embeddings to understand the *content* of your data, you can build datasets that are smaller, higher quality, and cheaper to label, all while training better models.