How to Block AI Crawlers in Cloudflare Without Blocking Googlebot

GPTBot, ClaudeBot, PerplexityBot, Bytespider and a dozen others read your site every day. Some of them send visitors back, some absorb your content into a model, some act for a human in real time. This guide separates the three, gives the current user agent inventory with sources, and shows the Cloudflare configuration that stops what you chose to stop without ever touching verified search crawlers.

Last reviewed on 12 September 2026. This area moves fast: Cloudflare changed its bot taxonomy on 1 July 2026 and is changing the defaults for newly onboarded domains on 15 September 2026, so check the dashboard values against what you read here rather than assuming.

Three kinds of crawler, three different decisions

The single most expensive mistake on this topic is treating "bot" as one category. A request from Googlebot, a request from GPTBot and a request from ChatGPT-User have almost nothing in common beyond being automated, and blocking all three with one rule is how sites lose search traffic while still being scraped.

Indexing crawlers. Googlebot, Bingbot, Applebot, DuckDuckBot. They read your pages to build a search index, and the exchange is explicit: they consume bandwidth, you get listed and receive clicks. Blocking them costs you traffic and nothing else changes.
AI training crawlers. GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, meta-externalagent. They read your pages so the content can be absorbed into model weights. There is no referral, no citation and no link back, now or later. This is the category most site owners actually mean when they say "block AI crawlers".
AI search and answer crawlers. OAI-SearchBot, PerplexityBot, Claude-SearchBot, meta-webindexer. They index your content so an answer engine can quote it and link to it. Economically these sit much closer to Googlebot than to GPTBot, which is why blocking them is a business decision rather than a hygiene measure.
Real-time AI agents. ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User. A human asked an assistant a question and the assistant fetched one page to answer it. Volume is low, intent is human, and several operators state explicitly that robots.txt does not apply to these fetches precisely because a person requested them.

Cloudflare formalised the same distinction on 1 July 2026. Bots are now classified by behaviour rather than by a single "AI bot" label, and three of those behaviours are exposed to every customer as presets: Search (collects or indexes your content so it can answer questions about it later), Agent (automated activity acting in real time on a person's behalf) and Training (crawls your content to train or fine-tune a model). Cloudflare tracks other behaviours too, including Transact, Data Collection, Security Testing, SEO, Ads Verification, Social and Link Preview, Feed Fetching, and Monitoring and Operations.

Before touching a single rule, write down one line per category: allow, block, or rate limit. Everything below is just the mechanics of enforcing that sentence.

The current inventory: who operates what

The list below is taken from each operator's own documentation, checked on 12 September 2026. Names are case-sensitive and must be reproduced exactly; a typo in a robots.txt token silently does nothing.

OpenAI

GPTBot: crawls content that may be used to train OpenAI's generative foundation models. Honours robots.txt.
OAI-SearchBot: surfaces websites in ChatGPT's search features. Not used for training. Honours robots.txt. Allowing this one while disallowing GPTBot is explicitly supported by OpenAI.
ChatGPT-User: fetches a page because a user or a custom GPT action asked for it. Not used for training. Because it is user-initiated, OpenAI states that robots.txt rules may not apply.
OAI-AdsBot: checks advertising landing pages.

Anthropic

ClaudeBot: collects web content that may contribute to training. Honours robots.txt.
Claude-SearchBot: crawls to improve search result quality inside Claude.
Claude-User: fetches a page when an individual asks Claude a question.

Anthropic publishes its crawler source IP ranges at https://claude.com/crawling/bots.json, which lets you verify a request rather than trust the header.

Perplexity

PerplexityBot: indexes pages so they can be surfaced and linked in Perplexity results. Stated as indexing only, not training.
Perplexity-User: visits a page to answer a specific user question, and by Perplexity's own documentation generally ignores robots.txt because a human requested the fetch.

This is also the clearest documented case of a crawler evading site preferences. In August 2025 Cloudflare published an investigation showing an undeclared Perplexity crawler presenting itself as Chrome on macOS, rotating through IPs and ASNs outside the published ranges after the declared agent had been blocked, and fetching pages on test domains whose robots.txt disallowed everything. Cloudflare removed Perplexity from its verified bot list and added detection signatures for all customers, including Free. Treat a user agent string as a claim, never as proof.

Google

Google-Extended: a robots.txt token, not a crawler. Google states that it does not have a separate HTTP request user agent string and that crawling is done with existing Google user agent strings. It controls whether your content can be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

The consequence is worth stating plainly because it is the single most common configuration error on this subject: an edge rule matching http.user_agent contains "Google-Extended" matches nothing at all, ever. No request carries that string. The only place the token has any effect is robots.txt.

Apple

Applebot: the crawler that feeds Siri, Spotlight and Apple search features.
Applebot-Extended: again a robots.txt token, not a crawler. Apple states that it does not crawl webpages and is only used to determine how to use the data already crawled by Applebot, and that pages disallowing it can still be included in search results.

Common Crawl

CCBot: operated by the non-profit Common Crawl Foundation, identifying itself as CCBot/2.0 (https://commoncrawl.org/faq/). Honours robots.txt, and publishes its IP ranges at https://index.commoncrawl.org/ccbot.json.

Common Crawl is an open archive rather than an AI company, but its corpus is one of the most widely reused training datasets in existence, which is why it appears in almost every AI blocklist including Cloudflare's own managed robots.txt.

ByteDance

Bytespider: operated by ByteDance, sending a mobile Safari user agent with compatible; Bytespider; [email protected] appended.

ByteDance publishes no crawler documentation, no IP list and no robots.txt statement, and independent server log analyses repeatedly report Bytespider fetching robots.txt and then requesting disallowed paths anyway. Its compliance should be treated as unverified, which means the only reliable control is enforcement at the edge.

Amazon

Amazonbot: Amazon states the data may be used to train Amazon AI models. Honours robots.txt, rel="nofollow" and robots meta tags.
Amzn-SearchBot and Amzn-User: Amazon states these do not crawl content for generative AI model training. The first makes content eligible for search experiences such as Alexa, the second supports user actions that need current information.

Why robots.txt is not enough, and why you should still write one

Cloudflare's own documentation puts it without varnish: robots.txt compliance is voluntary, the file expresses your preferences, and it does not prevent crawlers from accessing your content at a technical level. Three separate failure modes follow from that.

It is declarative. There is no enforcement step. A crawler reads the file, decides, and proceeds. Nothing in the HTTP exchange makes a Disallow binding.
Some operators ignore it. The Perplexity investigation above is the documented case; the recurring Bytespider reports are the widely observed one. Cloudflare's AI Crawl Control now has a dedicated table counting robots.txt violations per crawler, which exists precisely because the problem is common enough to need reporting.
User-initiated agents are deliberately out of scope. OpenAI states robots.txt rules may not apply to ChatGPT-User; Perplexity states Perplexity-User generally ignores them. This is not evasion, it is a stated design position: a human asked for the page, so the operator treats the fetch as a human visit.

A fourth, quieter failure is spoofing. Any HTTP client can send User-agent: GPTBot or User-agent: Googlebot, and plenty of scrapers send the latter specifically because sites allowlist it. This is what makes a bare user agent rule dangerous in both directions: it misses the traffic you meant to stop, and it can be used to walk straight through an allowlist.

Write the file anyway. It costs nothing, compliant operators do stop, and it is the only place where Google-Extended and Applebot-Extended have any effect at all. It also carries legal weight now that the Content Signals Policy exists: the boilerplate Cloudflare prepends states that restrictions expressed via content signals are express reservations of rights under Article 4 of European Union Directive 2019/790 on copyright in the digital single market.

Content Signals adds three machine-readable preferences on top of the usual directives, set on a single line such as Content-signal: search=yes, ai-train=no:

search: building a search index and returning links and short excerpts. It explicitly excludes AI-generated search summaries.
ai-input: feeding your content into a model at answer time, which covers retrieval augmented generation and grounding.
ai-train: training or fine-tuning models.

A signal set to yes grants that use, no refuses it, and an omitted signal neither grants nor restricts. The full policy text is published at contentsignals.org.

What Cloudflare actually enforces, and what it costs

Cloudflare now ships several overlapping features on this subject. They do different jobs, and their pricing differs, so it is worth being precise.

AI bot policies: the enforcement switch, on every plan

Under Security Settings > Configure AI bot policies, each of the three behaviours (Search, Agent, Training) takes one of three actions:

Block (on all pages): blocks across the entire zone.
Block on pages with ads: uses Cloudflare's automated detection of pages that display advertising, and blocks only there.
Allow (do not block): no blocking.

Each setting blocks the verified bots classified with that behaviour, plus additional unverified bots that fall under the same classification. This is available to all customers, Free plan included, and it is the part of the stack that survives a crawler renaming itself, because it works on Cloudflare's classification rather than on the string in the header.

Two dates matter. The older one-click Block AI bots setting is deprecating on 15 September 2026; it blocked verified bots classified as crawling for AI training plus similar unverified bots, and excluded mixed-purpose bots used for both Training and Search. From the same date, Cloudflare applies new defaults to newly onboarded domains: Training and Agent blocked on pages that display ads, Search allowed. Mixed-purpose crawlers that combine Search and Training are blocked by every configuration that blocks training, including the legacy option. Customers could opt out of the new defaults before that date.

AI Crawl Control: the reporting side, also on every plan

AI Crawl Control (formerly AI Audit) is available on all plans and works with zero configuration. Its Crawlers tab lists each AI crawler seen on your zone with its operator, its category, the number of allowed and unsuccessful requests with a trend chart, the count of robots.txt violations, and a per-crawler Allow or Block action with a configurable block response. Its Directives tab tracks whether your robots.txt is reachable and returning a usable status code, whether it carries Content Signals, and which crawlers requested paths your file disallowed, with the offending directive line shown next to each violation.

One honest caveat on cost: on the Free plan, AI Crawl Control identifies crawlers from their user agent string alone, which detects well-known self-identifying crawlers and nothing else. Paid plans add Cloudflare's own bot detection signals for a more thorough identification. Pay per crawl, which lets you charge crawlers for access instead of blocking them, is in closed beta and not generally available.

Managed robots.txt: the declarative layer, generated for you

Turning on the managed robots.txt setting has Cloudflare generate and maintain the file, prepending its managed block before your own file if you already serve one. The managed block sets Content-signal: search=yes, ai-train=no, use=reference for all agents, then adds Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Cloudflare's documentation is explicit that this expresses preferences and that enforcement is a separate job for AI Crawl Control, and that the two are meant to be used together.

Custom security rules: the part you control precisely

Under Security > WAF > Custom rules, the field that matters is cf.client.bot. It is a boolean, available on every plan, carrying the same information as cf.bot_management.verified_bot, and it is true when Cloudflare has confirmed the identity of the client. Cloudflare verifies through reverse DNS validation, published IP lists, ASN blocks and, increasingly, cryptographic signatures. Because verification happens on evidence the client cannot forge, a spoofed User-agent: Googlebot from a random host will match the user agent string and fail cf.client.bot, which is exactly the discrimination you need.

A distinction that trips people up: verified means identified, not welcome. Googlebot is a verified bot. So is ClaudeBot. So is OAI-SearchBot. Writing a rule that allows everything where cf.client.bot is true does not block AI crawlers, it waves most of them through. cf.client.bot belongs in your expressions as a safety condition on block rules, not as your AI policy.

Two richer fields exist but are part of Bot Management, which is a paid add-on rather than a Free plan feature: cf.verified_bot_category, a string letting you segment verified traffic by type and purpose, and cf.bot_management.signed_agent, a boolean that is true when the request came from an agent self-identifying with Web Bot Auth. On Free and Pro you build with cf.client.bot plus explicit user agent tokens, and you lean on the AI bot policy presets for everything else.

Cryptographic verification: Web Bot Auth

Web Bot Auth is the mechanism replacing "trust this IP range" with "prove it". A crawler generates an Ed25519 key pair, publishes the public key as a JSON Web Key Set at /.well-known/http-message-signatures-directory over HTTPS, registers that directory with Cloudflare, and then signs each request using HTTP Message Signatures, attaching Signature, Signature-Input and Signature-Agent headers. Cloudflare validates the signature against the registered key.

Since 1 July 2026, signed agents count as verified bots and Cloudflare labels every verified bot as Direct (operated by a single narrow operator, on its own infrastructure, so only that operator can present as this bot) or Intermediary (an agentic service that many different end users drive). The distinction matters for policy, because an intermediary introduces transitive trust: you may trust the operator without trusting every end user behind it. Cloudflare is experimenting with forwarding end-user information via the RFC 7239 Forwarded header so site owners can apply preferences to the party actually responsible.

Practically, this is what makes the whole approach durable. Reverse DNS and IP allowlists degrade as operators move infrastructure; a signature does not. Expect more of the agent traffic reaching your site to be signed, and expect the useful policy question to shift from "which string is in the header" to "who signed this and on whose behalf".

Blocking AI crawlers without breaking Googlebot or Bingbot

This is the part the title promises, and it has two halves: understanding what Google's AI controls actually govern, and writing rules that cannot backfire.

Google-Extended and Googlebot are linked, and that is fine

The same fleet does both jobs. Googlebot fetches the page; Google-Extended only governs what Google may subsequently do with what Googlebot already has. That is why there is no separate user agent and why you cannot express the preference anywhere except robots.txt.

Disallowing Google-Extended means: your content is not used to train future Gemini models powering Gemini Apps and the Vertex AI API for Gemini, and it is not used for grounding in Gemini Apps or Grounding with Google Search on Vertex AI.

Disallowing Google-Extended does not mean: it does not remove you from the Google index, it does not lower your ranking, and it does not take you out of AI Overviews or AI Mode. Google states the point directly: Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal in Google Search. AI features are built into Search and driven by Googlebot, so robots.txt directives for Googlebot are the control that governs them.

If your actual goal is to stay out of AI Overviews and AI Mode, the controls are different ones: nosnippet, data-nosnippet, max-snippet and noindex, which limit what Search can display from your pages across all appearances, plus the Search Console setting Google rolled out worldwide on 31 August 2026 that lets a site opt out of appearing in and grounding generative AI Search features. Opting out means no traffic and no impressions from those features, and Google states it is not used as a ranking signal for search results outside them. Weigh that carefully: it is a real trade, not a free precaution.

Bing and Apple have their own separate controls

Microsoft's mechanism is meta tags rather than robots.txt tokens. Content marked NOARCHIVE is not included in Bing Chat answers and is not used to train Microsoft's generative foundation models. Content marked NOCACHE may appear in Bing Chat answers, but only URLs, titles and snippets may be used in training. Both still appear in ordinary search results. Apple's Applebot-Extended works the same way conceptually: it blocks training use, and pages that disallow it can still be included in search results.

In all three ecosystems the pattern is identical and worth internalising: the AI training opt-out is deliberately decoupled from search inclusion. You do not have to choose between being indexed and refusing to be training data.

Rules that cannot backfire

Custom rules are evaluated in order, which means a reordering months later can change behaviour. The robust habit is to make each block expression safe in isolation rather than relying on an exemption rule sitting above it:

Carry the verified-bot exclusion inline. Every block expression ends with and not cf.client.bot. A rule written that way cannot hit Googlebot even if someone moves it to the top of the list.
Match full tokens, never loose substrings. http.user_agent contains "bot" matches Googlebot, bingbot, Applebot, facebookexternalhit's relatives and most monitoring tools. It is the single most common way a site accidentally de-indexes itself, an incident we cover in detail in our guide on accidental Googlebot blocking and SEO drops.
Mind case sensitivity. String evaluation in the Cloudflare rules language is case-sensitive by default. contains "gptbot" will not match GPTBot. Either reproduce the token exactly, or normalise with lower(http.user_agent) contains "gptbot".
Never block by country or ASN to solve a crawler problem. It cuts search fetchers, link preview bots, uptime monitors and real customers behind those networks, and it trades a bandwidth problem for an availability problem.
Do not block /robots.txt itself. A broad rule that catches the file removes the only channel through which compliant crawlers learn your preferences. AI Crawl Control flags this in the Directives tab when the file starts returning errors.

Step by step on a production zone

Write the policy down first. One line each for search engines, AI answer engines, AI training and real-time agents: allow, block, or rate limit. Reversing a block after crawlers have stopped visiting takes far longer than getting the decision right at the start.
Measure the baseline. Open AI Crawl Control > Crawlers and note which crawlers are actually present, at what volume, and with how many robots.txt violations. Cross-check request volume in Security > Events. Without a before, you cannot demonstrate an after, and you cannot tell a working rule from a rule that quietly matches nothing.
Publish robots.txt. Add Disallow: / blocks for the training tokens you refuse, add a Content-signal line, and include Google-Extended and Applebot-Extended here since this is the only place they work. Or turn on the managed robots.txt setting and let Cloudflare maintain the list, keeping in mind that its default list blocks training broadly.
Set the AI bot policies. In Security Settings > Configure AI bot policies, set Training, Agent and Search to match the sentences you wrote in step one. For most sites that means Training blocked, Search allowed, Agent decided on its own merits. Watch Security > Events for the first few hours to see what the presets actually intercept on your traffic.
Add one custom rule for impersonators. Under Security > WAF > Custom rules, catch clients that send a known training agent string but fail Cloudflare verification:

Expression:
  (lower(http.user_agent) contains "gptbot"
   or lower(http.user_agent) contains "claudebot"
   or lower(http.user_agent) contains "ccbot"
   or lower(http.user_agent) contains "bytespider"
   or lower(http.user_agent) contains "amazonbot"
   or lower(http.user_agent) contains "meta-externalagent")
  and not cf.client.bot
Action: Block

Note what is not in that list: OAI-SearchBot, PerplexityBot and Claude-SearchBot are left out on purpose, because they index for answer engines that cite and link. Add them only if your policy line for AI answer engines said block. Note also that Google-Extended and Applebot-Extended are absent, and must stay absent: no request ever carries them.

Rate limit unidentified clients on content paths. New agents appear faster than any list can be maintained, and the fallback for the ones you have never heard of is volumetric rather than nominative:

Expression:
  not cf.client.bot and starts_with(http.request.uri.path, "/blog/")
Characteristic: IP address
Count: 30 requests over 1 minute
Action: Managed Challenge

Verified search crawlers never reach this rule because of the not cf.client.bot condition. An unidentified client bursting through your archive gets throttled instead of saturating the origin. Tune the threshold against the baseline from step two, keeping it comfortably above what a human reader behind a shared corporate address could produce.

Scope to paths when the concern is partial. If only part of your site is at stake, add the path condition to the block expression rather than applying it site-wide:

Expression:
  (lower(http.user_agent) contains "gptbot"
   or lower(http.user_agent) contains "claudebot")
  and starts_with(http.request.uri.path, "/archive/")
Action: Block

Confirm search crawlers still pass. Filter Security > Events on the Googlebot and bingbot user agents over several days. Every entry should be allowed, or explainable by a rule you deliberately wrote. Anything else is a regression to fix immediately.

The opposite choice: being discoverable by AI on purpose

Not every site should block. The default assumption that AI crawling is theft holds for some business models and not for others, and it is worth being honest about which one you are.

Opening up tends to make sense when discovery is your constraint rather than distribution: consultancies and service businesses, documentation and developer tooling, B2B software, niche expertise where the buyer's first question now goes to an assistant rather than a search box. In those cases a citation in an answer engine is a qualified referral you would otherwise have paid for, and there is no per-page-view revenue being cannibalised.

Blocking tends to make sense when the content itself is the product: publishers monetising page views or subscriptions, paywalled archives, proprietary datasets, marketplaces whose live pricing is the asset, and anyone with the leverage to negotiate a licence instead of giving the corpus away.

This site has made the first choice, deliberately. Our robots.txt welcomes AI crawlers on everything except the legal notice and post-purchase pages, and we publish a machine-readable summary at /llms.txt. The reasoning is simple: there is no advertising inventory here to protect, and a consulting practice is found by being cited.

If you reach the same conclusion, here is how to implement it properly rather than by simply doing nothing:

Check your AI bot policies rather than assuming. Set Search and Agent to Allow explicitly. If your domain was onboarded on or after 15 September 2026, the defaults now block Training and Agent on pages with ads, so "I never configured anything" no longer means "everything is allowed".
Leave managed robots.txt off, or accept that its default content disallows the whole training list. Write your own file instead, and use Content Signals to express a nuanced position, for example Content-signal: search=yes, ai-input=yes, ai-train=no if you want to be quoted and linked but not absorbed into model weights.
Keep /robots.txt returning 200. The Directives tab in AI Crawl Control exists partly because a robots.txt blocked by an overzealous security rule is a common and invisible failure. Check the status code there.
Audit your existing rules for collateral damage. A rate limiting rule written years ago against scrapers will also throttle OAI-SearchBot. Being discoverable requires the same deliberateness as blocking.
Publish an llms.txt. Proposed by Jeremy Howard in September 2024, it is a Markdown file at the site root giving an H1 title, a blockquote summary, and H2-delimited sections of links with a one-line description each, so that an assistant can navigate your site without parsing HTML. It is a proposal rather than a ratified standard, but OpenAI, Anthropic and Google all publish one for their developer documentation, and Chrome's Lighthouse now audits sites for its presence.
Do not confuse it with a sitemap. sitemap.xml tells a crawler which URLs exist; llms.txt tells a model which pages matter and why. Publish both.

Verifying the block, and catching false positives

Rules propagate globally within seconds, which is convenient and dangerous in equal measure: a bad expression goes worldwide just as fast as a good one. Verify in both directions, always.

Confirming the block works

Security > Events, filtered on the rule. Matches should appear within minutes on a site with any real crawler traffic. A rule matching nothing after a day is usually a case-sensitivity error or a token that was never sent in the first place.
AI Crawl Control > Crawlers. Compare allowed against unsuccessful requests per crawler, and watch the trend chart across the deployment. This is the clearest single view of whether the intended crawlers stopped.
Origin request volume against the baseline. After a week, compare with the figures from step two. Expect some rebound as blocked scrapers rotate user agents, which is exactly what the rate limiting rule is there to absorb.
Do not trust a curl test. Sending a spoofed User-agent: GPTBot from your laptop proves your impersonator rule fires, nothing more. Your machine fails the reverse DNS and signature checks that a genuine crawler passes, so a block is the correct and expected outcome and tells you nothing about the real crawler.

Catching false positives on legitimate traffic

The casualties of an over-broad AI rule are rarely humans, which is what makes them easy to miss: they are automated clients you depend on, failing quietly.

Search crawlers. In Google Search Console, run a live URL inspection and confirm a 200 response, then watch Settings > Crawl stats for a rise in client errors or a fall in crawl requests. Bing Webmaster Tools offers the equivalent URL inspection. A drop in indexed pages a fortnight after a rule change is the expensive way to find this out.
Link preview bots. Share a URL in Slack, Discord, WhatsApp and on your main social platform. A missing preview card means facebookexternalhit or a sibling is being caught.
Feed readers and monitoring. Uptime monitors alerting on 403, RSS subscriber counts falling, a status page going red for no reason: all classic symptoms.
Your own infrastructure. CI pipelines, headless browser tests, payment provider callbacks, accessibility scanners and preview deployments all look like automation from the edge.
Verify a suspicious IP rather than guessing. For Google, a reverse DNS lookup should resolve to googlebot.com, google.com or googleusercontent.com, and a forward lookup on that name must return the same IP; alternatively, match against Google's published CIDR lists. Anthropic and Common Crawl publish equivalent JSON IP lists.
Deploy in Log action first where the product allows it. Review what a new rule would have matched over a few hours, then promote it to challenge or block with evidence rather than hope.

If the symptoms you are chasing are generic automated noise rather than AI crawlers specifically, the replacement rules in our guide on Bot Fight Mode false positives and challenge loops share the same verified-bot foundation and are the right starting point.

Common mistakes

Writing a security rule on Google-Extended or Applebot-Extended. They are robots.txt tokens with no user agent behind them. The rule matches zero requests and creates a false sense of protection.
Assuming robots.txt enforces anything. It is a request, not a control. The enforcement point is the edge.
Blocking the answer engines while meaning to block the trainers. OAI-SearchBot is not GPTBot; Claude-SearchBot is not ClaudeBot. Blocking the wrong one of each pair costs you citations while leaving the training crawl untouched.
Treating cf.client.bot as an AI policy. Verified means identified, not welcome. Most AI crawlers are verified bots.
Loose substring matching. contains "bot" is the classic de-indexing accident.
Forgetting case sensitivity. A lowercase token in a case-sensitive comparison silently matches nothing.
Blocking entire countries or ASNs. It cuts search fetchers, preview bots, monitoring and real mobile users behind those networks.
Leaving "I'm Under Attack" mode on permanently. The JavaScript interstitial breaks every non-browser client: mobile applications, server-to-server APIs, cron jobs and payment callbacks. It is an emergency lever, not a configuration.
Maintaining a static list and never revisiting it. New agents appear constantly and existing ones get renamed. The behaviour presets survive that churn; a hand-written list does not.
Changing several things at once. Toggle one setting, observe, then the next. Otherwise no effect is attributable and you cannot roll back the one that hurt.
Never re-reading the decision. A policy set in 2024 predates AI search sending measurable referral traffic. It is worth re-deciding annually rather than inheriting it by default.

Frequently asked questions

Does Cloudflare block AI crawlers by default?

Not on an existing zone that you have never configured. Cloudflare gives every customer, including Free plans, a set of AI bot policies covering three behaviours: Search, Agent and Training. Each can be set to Allow, Block on all pages, or Block on pages with ads. From 15 September 2026 Cloudflare applies new defaults to newly onboarded domains: Training and Agent are blocked on pages that display ads, while Search stays allowed. Existing zones keep whatever they were already set to, so the only reliable answer for your own site is to open Security Settings and read the current values.

How do I block AI crawlers?

In three layers. Declare your preference in robots.txt so that compliant operators stop on their own. Set the Training preset to Block in the Cloudflare AI bot policies so that the enforcement happens at the edge for known crawlers even when they rotate user agents. Then add one custom security rule matching the training user agent tokens combined with not cf.client.bot, which removes clients that send a known agent string but fail Cloudflare verification. Only the second and third layers actually stop anything.

Is there an AI bot blocker built into Cloudflare, or do I need a third-party tool?

It is built in and needs no third-party tool. Configure AI bot policies, under Security Settings, is the blocker itself. AI Crawl Control, available on all plans with zero configuration, is the reporting side: it lists every AI crawler seen on your zone, the requests it made, and how often it ignored your robots.txt, with a per-crawler Allow or Block action. On the Free plan AI Crawl Control identifies crawlers by user agent string; paid plans add Cloudflare's own bot detection signals.

Will blocking AI crawlers hurt my Google rankings?

No, provided you block AI crawlers and not Googlebot. They are different clients. Google states that Google-Extended, the robots.txt token that governs AI training and grounding, does not impact a site's inclusion in Google Search and is not used as a ranking signal. The real risk is collateral damage: a rule written on an ASN, a country or a loose substring like bot can catch Googlebot by accident, which does cost rankings. Every block expression in this guide carries an explicit verified-bot exclusion for that reason.

What exactly does blocking Google-Extended do, and what does it not do?

Google-Extended has no user agent of its own. Google states it does not have a separate HTTP request user agent string and that crawling is done with existing Google user agent strings, which means it is a robots.txt token only and cannot be matched by a security rule at the edge. Disallowing it tells Google not to use your content to train future Gemini models and not to use it for grounding in Gemini Apps or Grounding with Google Search on Vertex AI. It does not remove you from the Google index, does not change your ranking, and does not remove you from AI Overviews or AI Mode, which are part of Search and are driven by Googlebot.

How do I stop bots from crawling my site entirely?

You can, but you almost never want to. A rule blocking every client that is not a browser also removes search engines, link preview bots for Slack, Discord and WhatsApp, RSS readers, uptime monitors, accessibility scanners, payment provider callbacks and your own deployment checks. A more useful goal is to cap what automation costs you: block the training crawlers you have decided against, rate limit unidentified clients on content-heavy paths, and leave the verified bots that bring you traffic or that you depend on operationally.

How do I block SEO bots such as Ahrefs, Semrush or Majestic without blocking search engines?

Match their user agent tokens explicitly rather than blocking broad categories, and keep the verified search crawlers out of the expression. Most commercial SEO crawlers self-identify honestly and honour robots.txt, so a Disallow plus a narrow block rule is usually enough. Remember that blocking them also removes your own site from those tools' link graphs, so competitor research and backlink monitoring stop working for you too.

How do I block Googlebot on purpose?

Blocking Googlebot at the edge is the wrong instrument for almost every reason people reach for it. If you want a page out of the index, serve it with a noindex robots meta tag and let Googlebot fetch it, because a blocked crawler cannot read the instruction to drop the page. If crawl volume is the problem, use Search Console crawl stats and fix the slow responses that make Google back off. Block Googlebot at the edge only for a staging or preview hostname, and scope the rule to that hostname alone. Our guide on accidental Googlebot blocking and SEO drops covers what happens when this goes wrong on a production zone.

How do I block AI crawlers on specific pages only?

Add a path condition to the block expression, for example a starts_with() test on http.request.uri.path, so the rule applies to your archive or members area and nothing else. Cloudflare also offers a ready-made version of the same idea: the Block on pages with ads option in the AI bot policies limits blocking to the pages Cloudflare detects as carrying advertising. If your site is a Cloudflare Pages project, the same rules apply once the custom domain is proxied through a zone you control.

I blocked AI crawlers and something legitimate broke. How do I fix it?

Open Security > Events, filter on the rule you added, and read the user agents it actually matched. The usual casualties are link preview bots, RSS readers, uptime monitors, your own CI and, when the expression used a loose substring, search crawlers. Narrow the expression rather than disabling the rule: add the missing verified-bot exclusion, replace a substring match with a full token, or scope the rule to the paths that needed protecting. If Googlebot is among the matches, remove the rule immediately and re-run a live URL inspection in Search Console.

Can I allow AI search engines while blocking AI training?

Yes, and this is the most common policy in practice. Several operators publish separate tokens for each purpose: OAI-SearchBot for ChatGPT search results versus GPTBot for model training, Claude-SearchBot versus ClaudeBot, PerplexityBot for indexing. Set the Training preset to Block and the Search preset to Allow in the Cloudflare AI bot policies, and keep only the training tokens in your custom rule. Note that Cloudflare blocks mixed-purpose crawlers, those used for both Search and Training, under any training block.

How do I block AI agents that act in real time for a user?

Set the Agent preset to Block in the AI bot policies. Agent covers automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher. Be deliberate here, because a human asked for that fetch, and several operators state that robots.txt does not apply to user-initiated requests. Blocking agents means a person using an assistant to read your page gets nothing back.

Is Cloudflare's anti-bot protection enough on its own?

For self-identifying AI crawlers, the AI bot policies plus a custom rule cover the realistic cases. Bot Fight Mode is a different product with a different purpose and is not an AI crawler control: it scores generic automation and cannot be tuned, which is why it produces false positives on legitimate clients, as detailed in our guide on Bot Fight Mode false positives. The signals that cannot be forged, cf.client.bot on every plan and the Bot Management fields on paid plans, are what make a rule reliable; a user agent string on its own is a claim, not evidence.

Should I make my site discoverable by AI instead of blocking it?

For a site whose business depends on being found and cited rather than on advertising revenue per page view, opening up to AI search crawlers is a defensible choice, and it is the one this site has made. Keep Search and Agent allowed, block Training if you prefer, declare it in robots.txt with Content Signals, and publish an llms.txt describing your pages in Markdown. Publishers who monetise page views, paywalled archives and sites whose data is the product should reach the opposite conclusion.

How do I know the block is actually working?

Open Security > Events and filter on the rule: matches appear within minutes, since rules propagate globally in seconds. Cross-check in AI Crawl Control, which reports allowed and unsuccessful requests per crawler and counts robots.txt violations. Then verify the other side: run a live URL inspection in Google Search Console and confirm a 200 response. Testing with curl and a spoofed user agent only proves that your impersonator rule fires, because your machine fails the reverse DNS check that a real crawler passes.

Sources

Everything factual above was checked against primary documentation on 12 September 2026: Cloudflare's pages on blocking AI bots, verified bots, AI Crawl Control, managed robots.txt, Web Bot Auth and Bot Management variables; Cloudflare's published investigation into undeclared Perplexity crawling; the crawler documentation published by OpenAI, Anthropic, Perplexity, Google Search Central, Apple, Amazon, Meta and Common Crawl; the Bing Webmaster blog on NOARCHIVE and NOCACHE; the Content Signals Policy; and the llms.txt proposal.

How to Block AI Crawlers in Cloudflare™ Without Blocking Googlebot