The Problem: Serverless Doesn't Mean Unlimited
I run 3mins.news, an AI-powered news aggregator built entirely on Cloudflare Workers. The backend has 10+ cron triggers running every few minutes — fetching RSS feeds, clustering articles, calling LLMs, sending emails. It works great until you realize that Cloudflare's Workers Paid Plan has hard monthly limits:
| Resource | Monthly Limit | Overage Cost |
|---|---|---|
| Workers Requests | 10M | $0.30/M |
| Workers CPU Time | 30M ms | $0.02/M ms |
| KV Reads | 10M | $0.50/M |
| KV Writes | 1M | $5.00/M |
| KV Deletes | 1M | $5.00/M |
| KV Lists | 1M | $5.00/M |
| Queue Operations | 1M | $0.40/M |
| Observability Events | 20M | $0.60/M |
If a bug causes a retry loop or an unexpected traffic spike burns through KV writes, you either eat the overage bill or your app degrades unpredictably. Cloudflare doesn't auto-pause your workers when you hit limits — it just starts charging overages.
AWS has Budget Alerts. But those are passive notifications. By the time you read the email and react, you've already accumulated charges. What I wanted was active, application-level self-protection: the system should automatically scale back non-critical work before hitting the ceiling.
The Insight: Circuit Breaker, But Inward
The circuit breaker pattern is well-established — Netflix's Hystrix popularized it for protecting against cascading downstream failures. The idea: if a downstream service fails repeatedly, stop calling it for a while and fail fast instead.
But here, the threat isn't downstream. It's my own resource consumption against a hard budget ceiling. The circuit breaker needs to face inward.
The mental model:
Traditional: My App ──→ [Circuit Breaker] ──→ Downstream Service
"stop calling if it's failing"
Usage-aware: [Circuit Breaker] ──→ My App's Scheduled Tasks
"stop running if we're burning through our budget"
Architecture
The circuit breaker runs as part of the cron scheduler. Every 5 minutes, it queries Cloudflare's GraphQL API and Observability Telemetry API for current-month usage across all 8 resource dimensions. Between checks, it reads cached state from KV.
┌─────────────────────────────────────────────────┐
│ Cron Scheduler │
│ │
│ ┌──────────────────┐ │
│ │ Circuit Breaker │ │
│ │ │ │
│ │ Every 5 min: │ ┌─────────────────┐ │
│ │ Query CF APIs ───┼────→│ CF GraphQL API │ │
│ │ Cache to KV │ │ CF Telemetry API│ │
│ │ │ └─────────────────┘ │
│ │ Between checks: │ │
│ │ Read KV cache │ │
│ └────────┬─────────┘ │
│ │ │
│ tripped? ──── yes ──→ Skip all tasks │
│ │ │
│ no │
│ │ │
│ Run scheduled tasks normally │
└─────────────────────────────────────────────────┘
Per-Resource Thresholds
Not all resources are equally dangerous. Workers Requests at $0.30/M is cheap — a 10% overage costs $0.30. KV Writes at $5.00/M is expensive — a 10% overage costs $5.00. And some resources (like CPU time) can't really be "stopped" without stopping everything.
So I designed per-resource threshold configurations:
const RESOURCE_THRESHOLDS: Record<string, ResourceThresholds> = {
// Warn-only: cheap overage or can't selectively reduce
'Workers Requests': { warn: 80, trip: null, recover: null },
'Workers CPU': { warn: 80, trip: null, recover: null },
// Can trigger circuit breaker: expensive overage
'KV Reads': { warn: 80, trip: 90, recover: 85 },
'KV Writes': { warn: 80, trip: 90, recover: 85 },
'KV Deletes': { warn: 80, trip: 90, recover: 85 },
'KV Lists': { warn: 80, trip: 90, recover: 85 },
'Queue Operations': { warn: 80, trip: 90, recover: 85 },
'Observability': { warn: 80, trip: 90, recover: 85 },
}
The trip: null configuration is key. It means "warn me at 80%, but never trigger a full circuit break." Workers Requests and CPU are in this category — they're either cheap or can't be selectively reduced. The circuit breaker only trips on resources where overages are expensive AND the system can meaningfully reduce consumption by pausing scheduled tasks.
Three States with Hysteresis
The breaker has three states:
any resource >= 90% (trip)
Normal ─────────────────────────────────────→ Tripped
↑ │
│ all trippable resources < 85% │
└──────────────────────────────────────────────┘
(recover)
The 5% gap between trip (90%) and recover (85%) prevents oscillation. Without it, the system would trip at 90%, stop tasks, recover to 89%, resume tasks, immediately climb back to 90%, trip again — a classic flapping problem.
Usage %
95 ─ ─ ─ ─ ─ ─ ┐
90 ─ ─ ─ ─ ─ ─ ┤ ← trip threshold
85 ─ ─ ─ ─ ─ ─ ┤ ← recover threshold
│
Without hysteresis: With hysteresis:
↗90 trip ↗90 trip
↘89 recover ↘89 still tripped
↗90 trip ↘87 still tripped
↘89 recover ↘84 recover
(flapping!) (stable)
Alert Deduplication
Nobody wants 288 identical alert emails per day (24 hours × 12 checks/hour). The system deduplicates alerts at two levels:
- Warn alerts: Per-resource, per-month. If KV Reads hits 80% and you get an alert, the same resource won't alert again that month.
- Trip alerts: Global, per-month. One trip notification per month is enough.
Deduplication uses KV with TTL:
// Per-resource dedup key: "circuit:alert:warn:KV Reads:2026-03"
const kvKey = `circuit:alert:warn:${resource.name}:${monthKey}`
const alreadySent = await env.RUNTIME_KV.get(kvKey)
if (alreadySent) continue // Skip this resource
// After sending, mark as sent (TTL: 3 days for safety)
await env.RUNTIME_KV.put(kvKey, String(Date.now()), {
expirationTtl: 259200
})
The Check Flow
Here's the complete decision flow, executed every 5 minutes:
flowchart TD
Start[Cron trigger fires] --> IsCheckTime{minute % 5 == 0?}
IsCheckTime -->|No| ReadCache[Read KV cache]
ReadCache --> ReturnCached{Cached tripped?}
ReturnCached -->|Yes| SkipTasks[Skip all tasks]
ReturnCached -->|No| RunTasks[Run tasks normally]
IsCheckTime -->|Yes| QueryAPI[Query CF GraphQL + Telemetry API]
QueryAPI --> CacheUsage[Cache usage data to KV]
CacheUsage --> EvalTrip{Any trippable resource >= 90%?}
EvalTrip -->|Yes| WriteTripped[Write tripped flag to KV]
WriteTripped --> SendTripAlert[Send trip alert]
SendTripAlert --> SkipTasks
EvalTrip -->|No| CheckCurrentState{Currently tripped?}
CheckCurrentState -->|Yes| EvalRecover{All trippable < 85%?}
EvalRecover -->|Yes| ClearTripped[Delete tripped flag from KV]
ClearTripped --> RunTasks
EvalRecover -->|No| SkipTasks
CheckCurrentState -->|No| EvalWarn{Any resource >= 80%?}
EvalWarn -->|Yes| SendWarnAlert[Send warn alert]
EvalWarn -->|No| RunTasks
SendWarnAlert --> RunTasks
A few design details worth noting:
API failure tolerance: If the Cloudflare API query fails, the circuit breaker maintains its current state. It never assumes "everything is fine" on API failure — it reads the last known state from KV. This prevents a monitoring outage from masking a real usage spike.
Minimal overhead: Between 5-minute check windows, the circuit breaker is a single KV read — sub-millisecond, essentially free. The full API check only runs 12 times per hour.
Recovery-only on trippable resources: When evaluating recovery, only resources with trip != null participate. If Workers CPU is at 95% but all KV/Queue metrics are below 85%, the breaker recovers. CPU can't be reduced by pausing tasks anyway.
Querying Usage: Two APIs in Parallel
Cloudflare doesn't have a single "give me all my usage" endpoint. I piece together data from two APIs:
GraphQL API — covers Workers requests, CPU time, KV operations (by action type), and Queue operations:
query CfUsage($accountTag: String!, $since: Date!, $until: Date!) {
viewer {
accounts(filter: { accountTag: $accountTag }) {
workersInvocationsAdaptive(
filter: { date_geq: $since, date_leq: $until }
limit: 1
) {
sum { requests cpuTimeUs }
}
kvOperationsAdaptiveGroups(
filter: { date_geq: $since, date_leq: $until }
limit: 10
) {
dimensions { actionType }
sum { requests }
}
queueMessageOperationsAdaptiveGroups(
filter: { date_geq: $since, date_leq: $until }
limit: 1
) {
sum { billableOperations }
}
}
}
}
Observability Telemetry REST API — covers Observability events (logs/traces), which aren't exposed in the GraphQL API:
const telemetryResponse = await fetch(CF_TELEMETRY_ENDPOINT, {
method: 'POST',
headers: { Authorization: `Bearer ${token}` },
body: JSON.stringify({
timeframe: { from: monthStartMs, to: nowMs },
view: 'calculations',
parameters: {
calculations: [{ operator: 'COUNT' }],
datasets: [],
limit: 1,
},
ignoreSeries: true,
}),
})
Both requests run in parallel. If the Telemetry API fails, the system still works — it just reports Observability as 0 and relies on the other metrics.
Alerting: Email + Feishu
When a threshold is crossed, alerts go out via two channels in parallel:
- Email via Resend API — for persistent, searchable records
- Feishu (Lark) webhook — for immediate mobile notification
const results = await Promise.allSettled([
resendFetch({
apiKey: env.RESEND_API_KEY,
path: '/emails',
body: {
from: '3mins.news <alert@3mins.news>',
to: ['admin@example.com'],
subject: title,
html: buildAlertHtml(level, resources, env.ENVIRONMENT),
},
}),
env.FEISHU_WEBHOOK_URL
? sendFeishuMessage(env.FEISHU_WEBHOOK_URL, buildAlertCard(level, resources, env.ENVIRONMENT))
: Promise.resolve(true),
])
The Feishu card includes a visual progress bar for each resource:
[Production] Usage Warning
KV Reads ■■■■■■■■□□ 82.3% (8.2M/10M)
KV Writes ■■■■■■■■■□ 87.1% (871K/1M)
Promise.allSettled ensures one channel failing doesn't block the other. Dedup markers are only written after the email succeeds — so if email delivery fails, the next check cycle will retry.
Testing
The circuit breaker has comprehensive unit tests covering all state transitions:
describe('checkCircuitBreaker', () => {
// Non-check periods: read KV cache
it('returns false when no cached tripped state')
it('returns true when cached tripped state exists')
// Check periods: full evaluation
it('usage < 80%: no alert, returns false')
it('80-90%: sends warn alert, returns false')
it('>= 90%: trips circuit, writes KV flag, returns true')
it('tripped + all < 85%: recovers, deletes KV flag')
it('tripped + 85-90%: stays tripped (hysteresis)')
// Per-resource thresholds
it('Workers CPU >= 90%: warn only, no trip (trip=null)')
it('Workers CPU 95% + KV Reads 70%: no trip')
it('Workers CPU 95% + KV Writes 93%: trips on KV Writes only')
it('tripped + CPU still high + KV recovered: recovers')
// Alert dedup
it('same resource, same month: no duplicate warn')
it('resource A already warned + resource B new: warns B only')
// Edge cases
it('API failure while not tripped: returns false')
it('API failure while tripped: returns true (maintains state)')
it('no CF_AE_TOKEN configured: skips check')
it('caches usage data to KV for reporting')
})
The key test for hysteresis:
it('tripped + 85-90% → stays tripped (hysteresis)', async () => {
// KV Reads at 87% — below trip (90%) but above recover (85%)
vi.mocked(queryCfUsage).mockResolvedValue(
makeUsageData({ 'KV Reads': { percent: 87 } })
)
// Already tripped
const mockKV = {
get: vi.fn().mockResolvedValue(
JSON.stringify({ tripped: true, resources: ['KV Reads'], trippedAt: 1000 })
),
put: vi.fn(), delete: vi.fn(),
}
const result = await checkCircuitBreaker(makeEnv({ RUNTIME_KV: mockKV }), new Date())
expect(result).toBe(true) // Still tripped
expect(mockKV.delete).not.toHaveBeenCalled() // Flag not cleared
})
Lessons Learned
1. Per-resource thresholds are essential. A single global threshold doesn't work when resources have wildly different overage costs. $0.30/M for requests vs $5.00/M for KV writes is a 16x difference.
2. Hysteresis prevents flapping. The 5% gap between trip and recover thresholds was added after I watched the breaker oscillate in testing. It's a well-known technique in control systems, but easy to overlook in software.
3. "Fail safe" on monitoring failure. When the usage API is down, maintaining the last known state (rather than defaulting to "everything is fine") prevents the worst case: a monitoring outage hiding a usage spike.
4. KV is perfect for this. The circuit breaker state needs to persist across Worker invocations but doesn't need strong consistency. KV's eventual consistency is fine — a few seconds of stale state during a 5-minute check cycle is irrelevant.
5. Alert fatigue is real. Without per-resource monthly deduplication, the first time KV Reads hit 80%, I would get 8,640 alert emails for the rest of the month (12 checks/hour × 24 hours × 30 days). The dedup logic was not optional.
Applicability Beyond Cloudflare
This pattern applies to any serverless platform with usage-based pricing and hard limits:
- AWS Lambda: Monthly compute time, concurrent executions
- Vercel: Serverless function invocations, bandwidth
- Supabase: Database connections, storage, bandwidth
- Any metered API: OpenAI, Stripe, Twilio — anywhere you have a budget ceiling
The core idea is simple: treat your own resource budget as a health signal, just like you'd treat a downstream service's error rate. When the signal degrades, trip the breaker and gracefully degrade before the platform does it for you (or your wallet does).
The circuit breaker has been running in production for two weeks. It caught a KV Reads spike at 82% early in the month — I got a single warning email, investigated, fixed the root cause, and never hit the trip threshold. That's the outcome I wanted: early detection with zero downtime.
If you're running non-trivial workloads on any serverless platform, consider building something similar. The implementation is straightforward, and it's a much better safety net than platform billing alerts that arrive after the damage is done.