Cloudflare Just Blocked AI Crawlers by Default: What to Actually Do

On September 15, Cloudflare flipped the default from "let the bots in" to "make them ask." Here's who it hits, who it misses, and why the smart move this week is to not panic-flip anything at all.

By the Ghostwire Systems Team September 22, 2026·7 min read

Web & SEO · Hot Take

Key takeaways

On September 15, 2026, Cloudflare began blocking "mixed-use" AI crawlers by default on ad-supported pages — the biggest single change to how AI reads the open web this year.
The default applies to new customers, new sites on existing accounts, and all free-tier users. Existing paid setups keep their prior config unless you opt in.
Cloudflare also replaced Pay Per Crawl with Pay Per Use: publishers get paid when their content is actually used in an AI answer, not per fetch. First partners are Ceramic.ai and You.com.
The trap: you can't cleanly block AI training without risking Google Search. GoogleBot and Google-Extended still overlap.
Best move this week: audit first, block nothing site-wide, and treat Pay Per Use as upside — not a revenue plan.

Take a breath. The change is real, significant, and affects fewer people than the headlines suggest. Here's what shipped and what to do this week.

What actually changed on September 15

Cloudflare's Managed Robots rule for "mixed-use" AI crawlers flipped from off to on. Starting September 15, 2026, any site newly using Cloudflare — or already using it on the free tier — sits behind a default block for that class of bot on ad-supported pages.

The key word is mixed-use: crawlers that combine search indexing with AI training or answer-engine collection under the same identity, so a site owner has no clean way to say "index me for search but don't feed me to a model." Google is the poster child — GoogleBot indexes for Search, Google-Extended is the opt-out for Gemini and Vertex training, and the two share fetches. Matthew Prince's rationale: non-human traffic has now surpassed human traffic on Cloudflare's network. "Let everything in" doesn't hold up anymore.

Who is affected right now vs who is grandfathered

The default hits three buckets: new Cloudflare customers, any new site added to an existing account, and all free-tier users regardless of account age. Existing paid customers — Pro, Business, Enterprise — keep the crawler config they already had. If you want the new posture on a paid site, opt in from the dashboard.

Check before you assume. Cloudflare's dashboard shows which crawler categories are allowed, blocked, or set to "managed challenge" on each zone. Look before you tell anyone your posture.

Pay Per Use vs Pay Per Crawl (why per-USE is the real story)

The block got the headline, but the marketplace flip is the more interesting move. Cloudflare retired Pay Per Crawl and replaced it with Pay Per Use: publishers get paid when their content is actually used to produce an AI answer, not every time a bot slurps a page. Per-crawl paid you for raw material the model might throw away; per-use pays at the moment your content is actually worth something.

The launch partners are Ceramic.ai and You.com. Notable absences: OpenAI, Anthropic, Google, Perplexity. Until the big vendors sign on, Pay Per Use is a real mechanism with tiny volume. Treat it like a stock option, not a monetization channel.

Which crawler is which

Half the confusion here comes from people using "AI crawler" to mean six different things. Here's what's blocked, what isn't, and what you control at robots.txt.

Crawler	What it does	New Cloudflare default	Also honors robots.txt?
GoogleBot	Google Search indexing	Allowed	Yes
Google-Extended	Google's opt-out token for Gemini & Vertex training	Mixed-use — blocked	Yes (as a token, not a real bot)
Bingbot	Bing Search indexing	Allowed	Yes
GPTBot	OpenAI training crawl	Blocked	Yes
ChatGPT-User	Live user-agent fetches for ChatGPT answers	Blocked	Yes
ClaudeBot	Anthropic training crawl	Blocked	Yes
PerplexityBot	Perplexity answer engine	Blocked	Partially
CCBot	Common Crawl (used by many models)	Blocked	Yes

A lot of "AI" doesn't come from Google. Block Google-Extended and nothing else, and you've mostly just annoyed Google.

The Google-Extended trap

Here's the part nobody wants to say out loud. Google's public position is that Google-Extended is a clean AI opt-out that doesn't affect Search. In reality, GoogleBot and Google-Extended overlap on the same fetches, Google-Extended is a directive rather than a distinct user agent, and AI Overviews sit in a gray zone between Search and AI product. You cannot fully remove your content from Google's AI answers without risking Search visibility, and tooling that looks like it splits the two often misfires. If organic traffic is your business, test any block on a subset, watch Search Console for two weeks, then decide.

The AI crawler block isn't the story. The story is that the open web just admitted humans and machines have different rights to it — and nobody agrees on which is which.

What to do this week if you're on a free plan or a new site

Treat this as an audit prompt, not a policy change to celebrate or panic over. In roughly this order:

Check your bot analytics for the last 30 days. A site that GPTBot never visited doesn't need to think hard about GPTBot.
Compare Search Console traffic before and after Sept 15. If organic held, the block is doing no harm. If it dipped, investigate before it turns into a trend.
Check whether you appear in ChatGPT, Perplexity, and Google AI Overviews. Blocking answer-engine crawlers silently removes you from those surfaces — see our modern SEO strategy for 2027.
Decide per crawler, not "AI on/off." Blocking GPTBot for training while allowing ChatGPT-User for live answers is a coherent posture. "Block all AI" isn't a policy — it's a slogan.
Only then consider Pay Per Use. Opt in if you have valuable original content, and forget about it until the check gets interesting.

Don't blanket-block every AI user agent. Agentic browsers like Perplexity Comet and Claude for Chrome carry AI signatures but are driven by real customers — see our writeup on agentic browsers and your website.

What NOT to do

Don't blanket-block every AI-adjacent user agent. You'll take out agentic browsers, legitimate research bots, and probably a monitoring tool you forgot you installed.
Don't panic-monetize. Pay Per Use is not a business model yet. Enabling it is fine; expecting it to pay for your hosting is not.
Don't chase every AI referrer stat. Referrer data from AI answers is patchy and inconsistent. Trend lines beat single numbers.
Don't confuse "the default changed" with "the internet changed." Most working sites are on existing paid plans and see zero change today.

Where people go wrong (and when to call a pro)

The pattern we're already seeing: a business toggles "block all AI bots" and two weeks later can't figure out why Search Console impressions dropped and agentic-browser buyers can't check out. Or the opposite — they do nothing, don't notice their site inherited the new default, and quietly disappear from ChatGPT and Perplexity answers. The right move is a short audit: crawler posture, actual bot volume, AI-answer visibility, and a per-bot policy that matches the business.

It's a policy question dressed up as a technical one. If you're not sure which side of the default you're on, we audit this.

Frequently asked questions

If I'm on a paid Cloudflare plan today, does the Sept 15 change apply to me?

Not automatically. Existing paid customers keep the crawler configuration they already had. The new default block for mixed-use AI crawlers applies to new customers, any new site added to an existing account, and all free-tier users. If you want the new posture on a paid site, you have to opt in from the dashboard.

What actually is a "mixed-use" AI crawler?

Cloudflare's term for a bot that combines classic search indexing with AI training or agent-answer data collection under the same user agent, so you can't cleanly separate the two. Google's bundled GoogleBot plus Google-Extended is the poster child: they share fetches, and blocking one risks hitting the other. The Sept 15, 2026 default targets that mixed category, not pure search bots.

Will blocking Google-Extended hurt my Google search rankings?

Google says no in theory: Google-Extended is meant to opt out of Gemini and Vertex AI training without affecting Search. In practice the crawlers overlap, tooling that tries to split them can misfire, and AI Overviews in Search sit in a gray zone. If organic traffic is your business, test on a subset before you flip a site-wide block.

Is Pay Per Use going to make site owners real money in 2027?

For most small sites, no. Pay Per Use replaces Pay Per Crawl and pays publishers when their content is actually used in an AI answer, not just fetched. Ceramic.ai and You.com are the first partners. It's a better model than per-crawl, but volumes will be tiny until more AI vendors sign on. Treat it as upside, not a revenue plan.

Not sure which side of the default you're on?

Let's audit your crawler posture before it costs you traffic or answer-engine visibility.

Ghostwire Systems tunes bot policy, schema, and AI-answer visibility so your site works for humans, agents, and the vendors quietly deciding whether to cite you. Tell us what you're running.

Get a quote See our services