I run 3mins.news, an AI-powered news aggregator that processes 180+ RSS sources across 17 languages. Every hour, hundreds of new articles flow in — and the same event gets reported by BBC in English, NHK in Japanese, Le Monde in French, and Xinhua in Chinese.
The hardest engineering problem isn't fetching RSS feeds or rendering a nice UI. It's deduplication: figuring out that these four articles, written in four different languages with zero shared tokens, are actually about the same event.
This post covers the approach I built, the tradeoffs involved, and how the entire pipeline runs on Cloudflare Workers + PostgreSQL for about $100/month.
The Textbook Answer: MinHash + LSH
If you search "document deduplication at scale," you'll find MinHash + Locality Sensitive Hashing (LSH). It's elegant:
- Shingling: Break each document into overlapping n-grams (e.g., 3-word sequences)
- MinHash: Hash each shingle set into a compact signature (~100-200 values)
- LSH: Split signatures into bands, hash each band into buckets. Similar documents land in the same bucket.
The key insight: the probability that two MinHash signatures share a minimum value equals the Jaccard similarity of the original shingle sets. You get approximate similarity without comparing raw text.
This is what early Google News and web crawlers used. For monolingual, same-script dedup, it works beautifully. Several Google research papers from the 2000s describe exactly this pipeline.
Why It Breaks for Multilingual News
Consider this real scenario from a single morning:
| Source | Language | Headline |
|---|---|---|
| Reuters | English | "EU Approves New AI Regulation Framework" |
| NHK | Japanese | "EU、新たなAI規制枠組みを承認" |
| Le Monde | French | "L'UE adopte un nouveau cadre réglementaire pour l'IA" |
| Xinhua | Chinese | "欧盟批准新人工智能监管框架" |
These four articles describe the exact same event. But run 3-word shingling on them:
- English shingles:
{"EU Approves New", "Approves New AI", "New AI Regulation", ...} - Japanese shingles:
{"EU、新たな", "新たなAI規制", "AI規制枠組み", ...} - French shingles:
{"L'UE adopte un", "adopte un nouveau", ...}
Jaccard similarity = 0. Zero token overlap. MinHash would correctly report that these documents share nothing — because at the token level, they don't.
You could try translating everything to a common language first, but that adds latency, cost, and introduces translation errors that propagate into your similarity scores.
The Solution: Multilingual Embeddings
Modern multilingual embedding models solve this by mapping semantically equivalent text to nearby points in vector space, regardless of language. The sentence "EU approves AI regulation" and "欧盟批准AI监管" produce vectors with cosine similarity > 0.85 — because the models are trained on parallel corpora across languages.
I use a multilingual embedding model that produces 1536-dimensional vectors. Each incoming article gets embedded once at ingestion time. From that point on, all dedup and clustering operates in vector space where language barriers don't exist.
Cost reality: Embedding is cheap. At ~$0.04 per million tokens, embedding thousands of articles daily costs pennies.
Two-Pass Clustering with pgvector + UnionFind
Here's where it gets interesting. Raw embedding similarity isn't enough — you need to actually group articles into coherent "stories" (clusters of articles about the same event). I use a two-pass approach:
Pass 1: Match Against Existing Stories
Most incoming articles belong to events we're already tracking. A new Reuters update about an ongoing crisis should merge into the existing story cluster, not create a new one.
For each unassigned article, I run a KNN query against existing story embeddings:
SELECT item_src.item_id,
story_cand.story_id,
1 - (story_cand.embedding_vec <=> item_src.embedding_vec) AS sim
FROM item item_src
JOIN LATERAL (
SELECT story.story_id, story.embedding_vec
FROM story
WHERE story.embedding_vec IS NOT NULL
AND story.created_at >= $cutoff -- only recent stories
AND ABS(item_src.published_at -
story.latest_published_at) <= $max_gap -- time proximity
ORDER BY story.embedding_vec <=> item_src.embedding_vec
LIMIT 10
) story_cand ON true
WHERE item_src.item_id = ANY($batch_ids)
AND item_src.embedding_vec IS NOT NULL
Key constraints:
- Similarity threshold: 0.7 — tuned through experimentation. Lower catches more but introduces false merges.
- Time gap ≤ 18 hours — prevents merging "same topic, different month" (e.g., two separate earthquake events).
- Story age ≤ 36 hours — old stories stop absorbing new articles.
This pass typically catches ~70-80% of incoming articles. They get assigned to existing stories, and the story's embedding is updated as a sliding window average of its most recent articles.
Pass 2: Cluster Remaining Articles with UnionFind
Articles that didn't match any existing story might be part of a new, emerging event. To find these clusters, I run item-to-item KNN:
SELECT item_src.item_id AS src_id,
item_cand.item_id AS dst_id,
1 - (item_cand.embedding_vec <=> item_src.embedding_vec) AS sim
FROM item item_src
JOIN LATERAL (
SELECT item_id, embedding_vec
FROM item AS item_cand
WHERE item_cand.embedding_vec IS NOT NULL
AND item_cand.published_at >= $window_start
AND item_cand.item_id <> item_src.item_id
ORDER BY item_cand.embedding_vec <=> item_src.embedding_vec
LIMIT 10
) item_cand ON true
WHERE item_src.item_id = ANY($batch_ids)
This gives me a similarity graph: nodes are articles, edges are similarity scores ≥ 0.7. Now I need to extract connected components — groups of articles that are transitively similar to each other.
UnionFind (disjoint set) is perfect here:
class UnionFind {
private parent: number[]
private rank: number[]
constructor(size: number) {
this.parent = Array.from({ length: size }, (_, i) => i)
this.rank = new Array(size).fill(0)
}
find(node: number): number {
if (this.parent[node] !== node) {
this.parent[node] = this.find(this.parent[node]) // path compression
}
return this.parent[node]
}
union(a: number, b: number): void {
let rootA = this.find(a)
let rootB = this.find(b)
if (rootA === rootB) return
if (this.rank[rootA] < this.rank[rootB]) [rootA, rootB] = [rootB, rootA]
this.parent[rootB] = rootA
if (this.rank[rootA] === this.rank[rootB]) this.rank[rootA]++
}
}
For each edge with similarity ≥ 0.7, call union(src, dst). Then extract components. Any component with ≥ 2 articles becomes a new story.
Why UnionFind over DBSCAN or hierarchical clustering?
- It's O(α(n)) per operation — essentially O(1) in practice
- Pure CPU computation, no database round-trips
- Works naturally with the edge-list output from KNN queries
- Deterministic — same edges always produce the same clusters
Story Embedding: Sliding Window Average
Each story's embedding is the average of its 3 most recent articles' embeddings:
WITH latest AS (
SELECT item.embedding_vec,
ROW_NUMBER() OVER (
ORDER BY item.published_at DESC NULLS LAST
) AS rn
FROM story_item si
JOIN item ON si.item_id = item.item_id
WHERE si.story_id = $story_id
AND item.embedding_vec IS NOT NULL
)
UPDATE story
SET embedding_vec = (SELECT avg(embedding_vec) FROM latest WHERE rn <= 3)
WHERE story_id = $story_id
Why top-3 instead of all articles? As a story evolves, its focus shifts. "EU proposes AI regulation" becomes "EU votes on AI regulation" becomes "EU AI Act signed into law." The sliding window keeps the story embedding aligned with its current state, not its historical average.
pgvector: The Engine Behind It All
All vector operations happen inside PostgreSQL using the pgvector extension. No dedicated vector database, no external ML infrastructure.
HNSW Indexes
CREATE INDEX ON item USING hnsw (embedding_vec vector_cosine_ops);
CREATE INDEX ON story USING hnsw (embedding_vec vector_cosine_ops);
HNSW (Hierarchical Navigable Small World) provides approximate nearest neighbor search. For my dataset size (~tens of thousands of items in the active window), it's fast and accurate enough.
One gotcha with connection pooling: SET hnsw.ef_search = 64 gets reset when the connection returns to the pool. The fix is SET LOCAL inside an explicit transaction:
BEGIN;
SET LOCAL hnsw.ef_search = 64;
-- KNN queries here
COMMIT;
This ensures the parameter only lives for the duration of the transaction, which is exactly what you want with pooled connections.
Why Not a Dedicated Vector Database?
I considered Pinecone, Weaviate, and Qdrant. But:
- One database to manage — joins between vectors and relational data (story metadata, publication times, source info) are trivial in PostgreSQL, painful across systems
- Cost — pgvector is free, included in any PostgreSQL instance
- Good enough — at my scale (tens of thousands of active vectors), HNSW in pgvector handles sub-100ms KNN queries easily
- Transactional consistency — embedding writes and metadata updates happen in the same transaction
The Full Pipeline
Here's the complete data flow, from RSS feed to curated news:
RSS Sources (180+, 17 languages)
│
▼
[Fetch] ─── Parse RSS ─── Deduplicate by URL ─── Compute Embedding
│
▼
[Cluster] ─── Pass 1: Match existing stories (KNN)
│ Pass 2: UnionFind new clusters (KNN + connected components)
│
▼
[Score] ─── LLM evaluates significance (0-10) across 6 dimensions
│ LLM rewrites headline + summary
│ Translate to 10 languages
│
▼
[Event] ─── Group related stories into multi-day events
│ Generate evolving narrative (4-section structure)
│ Translate narrative to 16 languages
│
▼
[Publish] ─── Select top stories ─── Render multilingual emails
Batch send via email API
Each stage is a separate Cloudflare Workflow — a durable, retryable execution unit. If an LLM call fails, only that step retries. If the entire worker crashes mid-pipeline, execution resumes from the last completed step.
LLM Cost Optimization
The LLM pipeline uses different models for different tasks:
| Task | Model Choice | Why |
|---|---|---|
| Scoring + Rewriting | DeepSeek | Cheap, good at structured output |
| Translation | Qwen Plus | Better multilingual quality |
| Embedding | Qwen3 Embedding 8B | Good cross-lingual alignment, low cost |
Translation batching: Instead of 10 separate API calls (one per language), I batch 2 languages per call across 5 batches. Each batch writes to the database independently — if batch 3 fails, batches 1-2 are already persisted.
Failure isolation: Scoring and translation are tracked with separate flags (needs_score, needs_translate). A translation failure doesn't re-trigger scoring. A permanent failure gets flagged with -1 and never retried.
Running on Cloudflare Workers at $100/month
The entire backend runs on Cloudflare's serverless stack:
| Service | Role | Why |
|---|---|---|
| Workers | All compute | No servers to manage |
| Workflows | Pipeline orchestration | Durable execution, built-in retry |
| Queues | Decoupling | Scheduler → Workflow fanout |
| KV | Runtime state | Concurrency guards, rate limiting |
| Hyperdrive | DB connection pooling | Connection reuse for PostgreSQL |
| Cron Triggers | Scheduling | Minute-level pipeline orchestration |
Cost Breakdown
| Item | Monthly Cost | Notes |
|---|---|---|
| LLM (scoring + rewriting) | ~$55 | DeepSeek, with prompt caching |
| LLM (translation) | ~$12 | Qwen Plus, resource package |
| LLM (embedding) | ~$8 | Minimal per-token cost |
| Email delivery | $20 | Resend Pro plan |
| Cloudflare Workers | $5 | Paid plan |
| PostgreSQL (managed) | ~$0 | Free tier sufficient at current scale |
| Total | ~$100 |
The key insight: cost doesn't scale with users. The LLM cost is driven by content volume (180 sources × daily articles × 10 translation languages), not by how many people read the output. Whether I have 10 users or 10,000 users, the LLM bill is the same. Only email delivery and database connections scale with users.
Fitting in 128MB Memory
Cloudflare Workers have a 128MB memory limit. That's tight when you're doing vector operations on thousands of articles.
The solution: push all vector computation into PostgreSQL. The Worker never holds embedding vectors in memory — it only passes around item IDs. The KNN queries, similarity calculations, and average computations all happen inside pgvector.
The Worker's job is orchestration: deciding which items to cluster, batching them into manageable groups (200 items per KNN batch), and writing results back. The heavy math stays in the database.
Other constraints I work around:
- Step output ≤ 1MiB: Page-load items at 5,000 per page (~760KB)
- 50 subrequests per invocation: Batch SQL operations using
unnest()arrays instead of individual INSERTs - Connection pooling resets session state: Use
SET LOCALin transactions, neverSET
Lessons Learned
1. Cosine similarity thresholds are domain-specific. 0.7 works for news dedup because news articles about the same event share vocabulary and structure. For more diverse content, you'd need different thresholds.
2. Time constraints matter as much as similarity. Without the 18-hour gap limit, the system would merge "Japan earthquake 2025" with "Japan earthquake 2026." Semantic similarity alone isn't enough.
3. Two-pass beats single-pass. Matching against existing stories first is both faster (fewer comparisons) and more accurate (story embeddings are refined over time) than clustering everything from scratch.
4. PostgreSQL is underrated for vector workloads. The ability to combine vector KNN with relational filters (time ranges, source IDs, status flags) in a single query is a genuine advantage over dedicated vector databases.
5. Batch everything. On Cloudflare Workers, every subrequest counts. Batch your SQL operations with unnest(), batch your LLM calls (multiple languages per request), batch your email sends. The difference between per-item and batched operations is the difference between hitting limits and running smoothly.
If you're interested in the end result, check out 3mins.news — AI-curated global news in 3 minutes, available in 17 languages. The entire pipeline described above runs every hour, processing hundreds of articles into a handful of curated stories.