CF GarageThis case study is published with the client anonymised. Volumes, signatures and reasoning are reproduced as measured; the domain name, the targeted pages and the identifiers of the rules in place are not, because the campaign was still running at the time of writing. The dashboard screenshots have been cropped with the same goal in mind.
The site and the symptom
A European marketplace on the Cloudflare Business plan: a public catalogue of several hundred thousand offer pages, roughly five million requests a day, plus a mobile application talking to the same domain. The operator noticed a spike in unique visitors on the Cloudflare dashboard and asked whether it was good news or bad news.
Day Requests Page views Unique visitors
D-4 5.33 M 443 219 43 309
D-3 5.23 M 429 719 44 173
D-2 4.96 M 409 933 64 532
D-1 2.37 M 230 294 47 137
D 3.27 M 208 304 816 627
D+1 6.28 M 444 606 721 136
D+2 6.03 M 460 856 522 820
D+3 5.48 M 420 887 378 014The whole diagnosis is almost entirely in the shape of those three columns. Unique visitors were multiplied by eight to nineteen depending on which day you compare. Page views did not move, and on day D they actually went down. Traffic that arrives without generating page views is traffic that does not navigate: it fetches one page, it leaves, it comes back under another address. The hourly series then pinned the switch to the hour, from 30 223 requests at 02:00 UTC to 80 908 at 03:00, with no return to baseline afterwards.
Why the usual readings showed nothing
Three separate things were hiding this campaign, and each of them is common enough to be worth naming.
- The source networks looked like the customer base. Over one four-hour sample, the top source networks were a Vietnamese operator, three French consumer ISPs, a Romanian and Portuguese operator, an Emirati ISP, a Moroccan incumbent, a Brazilian carrier, a Pakistani operator, a Turkish one and a Mexican one, with cloud providers such as Google, Microsoft and Amazon making up a minority. Every one of those is plausible for a site with real traffic in those markets. Residential bot pools are sold precisely so that a country or ASN reading returns nothing.
- The threat counter was already high, and already explained. The zone had a custom rule blocking web scraping which, over two days, matched more than 1.5 million requests. Seeing a large number rise on a counter that is supposed to rise is not an alarm. It reads as the protection doing its job, not as evidence that a second campaign is passing beside it.
- The origin server was fine. No server alarm, no slow page, no CPU spike. That was no accident, and we come back to it below: after two days, the campaign had moved most of its volume into the edge cache, without ever waking up origin monitoring.
The reading that made it visible
Rather than slicing by country or network, we crossed four dimensions in a single query: the browser as parsed by Cloudflare, the HTTP protocol version, the TLS version, and the source of Cloudflare's bot classification. Over a four-hour window, restricted to requests that actually reached the origin:
191 225 Chrome HTTP/1.1 TLSv1.3 heuristics
10 966 Chrome HTTP/1.1 TLSv1.3 machine_learning
6 944 Chrome HTTP/2 TLSv1.3 machine_learningThe same bucket, at the same hours, the previous day: 4 444 requests. A factor of forty-three, in one line. Three elements converge on the same reading.
- A protocol contradiction. A real Chrome negotiates HTTP/2 through ALPN against a Cloudflare edge and does not spontaneously fall back to HTTP/1.1. "Chrome over HTTP/1.1" is the signature of an HTTP library, whether Go
net/http, Pythonrequests,curlornode-fetch, carrying a Chrome user agent string. - Deterministic classification. Cloudflare labelled this traffic
heuristics, meaning it matched a known automation signature, rather thanmachine_learning, which is the probabilistic scoring applied to ambiguous traffic. The edge had already recognised the shape; nothing had been configured to act on it. - A hard-coded user agent pool. Fourteen distinct user agent strings, each carrying between 14 059 and 14 451 requests. Real traffic is never that evenly distributed; a three percent spread across fourteen values is a round-robin through a list. The announced versions were old and inconsistent with each other, Chrome 99 to 116 on Windows and 119 to 136 on Mac, while the site's legitimate traffic was dominated by Chrome 152.
A second discriminator: the missing referer
Filtering that same window down to Chrome over HTTP/1.1 classified as heuristics, and grouping by referer host:
191 228 no referer (99.99 %)
14 referer presentReal navigation to a deep catalogue page almost always carries a referer, internal or external. This pool never sent one. That matters for what comes next: the referer signal survives a switch to HTTP/2, which the protocol clause does not. It is not free of false positives, since it also catches direct entry through a typed URL or a bookmark, measured at roughly 3 500 requests over twenty hours on this zone. Which is exactly why it gets measured before it gets enforced.
Flat around the clock, then absorbed by the cache
Three days after the switch, the campaign was still running, but it no longer looked the same depending on where you observed it from. On total traffic, the suspect bucket stayed flat: around 25 000 requests an hour day and night, 471 370 requests over a full day, 8.5% of the zone's traffic, with no trace of a day and night cycle and an unexplained trough between 09:00 and 12:00 UTC. Over the same day, legitimate traffic varied by a factor of six between its 02:00 trough and its 14:00 peak. A curve that ignores the cycle of its own audience is automation, on its own, without any other evidence.
On the traffic that actually reached the origin, by contrast, the same curve had collapsed after two days on the plateau: most of the sweep was now being served by the edge cache, from pages that had already been requested. Nothing had changed on the attacker's side; the edge was doing the absorbing. This is the part that catches most operators out: against a cached public catalogue, scraping can keep running at full speed without moving a single server monitoring curve.
Two traps on the way to a fix
Per-URL rate limiting does nothing here
Of the 1 544 207 requests the existing rule had blocked over two days, the fifty most-hit paths accounted for fewer than 55 700, under four percent of the volume. The rest was a long tail of detail pages enumerated by sequential identifier across a range of roughly 39 000 values, each individual page sitting at 400 to 430 hits over two days. Every single URL stayed below any threshold you could reasonably set, so a per-URL rate limit would have matched nothing at all. Counting has to aggregate by URI pattern or by source IP. This is the single most common reason a rate limiting rule ships and changes nothing.
Read the actions before you read the volumes
Skip rule carries more traffic than the blocking rule; its volume alone makes it neither a flaw nor a candidate for tightening. The volumes shown cover a different window than the ones quoted in the text.
The zone had a second rule matching more than 1.4 million requests on the mobile application's endpoints. Judged on volume alone, that looks like a large amount of unfiltered traffic and a candidate for tightening. It was in fact a deliberate Skip allowlist, put there so that the mobile app would never be challenged. Turning it into a block would have taken the application down for every customer. We raised it as a candidate mid-analysis and withdrew the suggestion as soon as we identified which rule carried which action. Event counts tell you what a rule matched, never what it did with it; on a zone you did not configure yourself, establish the action of every rule before you propose touching any of them.
The recommendation: four layers, not one rule
The protocol rule stops this campaign, not this attacker. The day the operator switches their library to HTTP/2, which curl --http2, Python httpx and Go net/http all do natively, the contradiction disappears and the rule matches nothing. Layering is not belt and braces here; it is the only approach that stays true against a rotating bot pool, because what it attacks is the economics. Every layer makes each fresh IP address more expensive, and the attacker is paying for that traffic.
- Layer one, the protocol contradiction.
http.user_agent contains "Chrome" and http.request.version eq "HTTP/1.1", with a Managed Challenge action rather than a block: some corporate proxies do downgrade to HTTP/1.1, and a real browser behind one will solve the challenge while a Go client will not. The mobile app is unaffected, its user agent contains no browser string, and Android WebView traffic speaks HTTP/2 and escapes the clause. - Layer two, a challenge lock on the targeted section. Requests to the enumerated section without a
cf_clearancecookie get a Managed Challenge. This is the main lever against IP rotation: every new address has to solve a challenge before seeing a single page, which is what makes a metered residential bot pool expensive. It needs a clause exempting the application's own session cookie, so that signed-in customers do not meet a challenge on their first click. - Layer three, a rate limit aggregated by IP on the enumerated pattern. Counting on the URI pattern rather than the exact URL, at a threshold of fifteen to twenty requests per IP over ten minutes, action Managed Challenge. A human consults a handful of offers; a sequential sweep across tens of thousands of identifiers crosses that line even at a deliberately slow pace.
- Layer four, the user agent pool itself. Enumerating the fourteen hard-coded strings costs nothing, catches the attacker if they change their protocol without refreshing their pool, and has a near-zero legitimate residual given how obsolete the announced versions are. It is fragile by construction, since the pool can be regenerated in an afternoon, and it is worth deploying anyway because it is free.
Underneath all four, Super Bot Fight Mode set to challenge definitely-automated traffic acts as the net if everything above is circumvented. And every one of these rules goes out in Log action first, for a few hours, so the legitimate residual can be measured against the baseline before anything is enforced. One change at a time, never several, or no effect can be attributed to its cause.
The item that outranked every rule
The enumerated pages sat under an account path and were addressed by sequential identifier. If pages like that are served without checking that the record belongs to the authenticated user, there is an access control problem that exists independently of traffic volume, and no amount of edge filtering will fix it: an attacker who slows down to one request per minute still walks the whole catalogue. Verifying that ranked above deploying any security rule, and we said so before proposing the rules. If those pages require authentication, the scraping becomes pointless and the four layers become a comfort rather than a necessity.
This is a recurring pattern worth stating plainly. Edge filtering buys time and raises cost; it does not replace an authorisation check. When a diagnostic surfaces something the application should be doing instead, saying so is part of the job, even when it shortens the engagement.
What this case shows
- Unique visitors rising while page views stay flat is a signature. Traffic that does not navigate is not an audience. Before investigating anything else, compare those two curves.
- Source geography is the first thing an attacker dilutes. Country and network readings are cheap to defeat with a residential bot pool. Protocol coherence, the agreement between the claimed browser, the negotiated HTTP version and the headers actually sent, is far harder to forge and far more discriminating.
- A quiet origin server proves nothing. Against a cached public catalogue, scraping is served by the edge and never appears in server monitoring. Here, the campaign ended up drawing most of its requests from cache without changing a single parameter.
- Measure in Log, then enforce. The cost of a badly scoped security rule is paid by real customers, worldwide, within seconds of deployment. A few hours of measurement is cheap insurance.
If you are seeing the same shape on your own zone, the two related guides below cover the fixed parts of this work in tutorial form: rate limiting and bot rules on sensitive endpoints, and separating crawlers you want from crawlers you do not.
Frequently asked questions
Why did the country and ASN breakdowns show nothing?
Because the campaign ran through rotating residential bots located in the markets the site actually serves. Consumer ISPs carried more of the traffic than datacenter networks, so every geographic and network reading looked plausible. Source location is the first dimension an attacker dilutes, because it is the cheapest one to fake. Protocol coherence is much harder to disguise, which is why crossing browser, HTTP version and TLS version made the campaign readable in one query.
How can a browser user agent over HTTP/1.1 be a reliable signal?
A real Chrome negotiates HTTP/2 through ALPN against a Cloudflare edge and will not fall back to HTTP/1.1 on its own. A user agent claiming Chrome while speaking HTTP/1.1 is almost always an HTTP library wearing a browser string. The signal is not perfect: some corporate proxies downgrade connections, which is exactly why the rule should use a Managed Challenge rather than an outright block.
Why does per-URL rate limiting fail against a scraper?
Because a scraper enumerating a catalogue spreads its requests across tens of thousands of distinct URLs. In this case the fifty most-hit paths accounted for less than four percent of the blocked volume, and every individual page sat at a few hundred hits over two days, far below any sane threshold. Rate limiting has to aggregate by URI pattern or by source IP, never by exact URL.
Does the origin server load tell you whether you are being scraped?
No. Once the edge cache warms up on a public catalogue, most scraping requests are answered by Cloudflare and never reach the origin. Server monitoring stays quiet while the content is harvested at full speed. In this specific case, the campaign even moved most of its volume into the cache after two days without changing a single parameter on its side.