Cloudflare sets pay-per-crawl price for AI bots as new default

Cloudflare has announced that it will block known AI crawlers by default on all new domains unless explicit permission is granted or payment is made.

This change, termed “Content Independence Day” by Cloudflare CEO Matthew Prince, follows a movement where over one million websites had already opted to block AI bots since last fall.

Major publishers and platforms including the Associated Press, Time, The Atlantic, BuzzFeed, Reddit, Quora, and Universal Music Group have joined the initiative.

Prince explained that AI companies extract far more content than they return in traffic to publishers, citing a ratio of 1,500 pages crawled per visitor for OpenAI, compared to 18 to 1 for Google.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone,” he said.

Cloudflare is introducing a “pay-per-crawl” marketplace where publishers can set fees for AI bots seeking access to their content.

This system uses the HTTP 402 “payment required” code as a digital tollbooth, allowing publishers to allow free access, charge per crawl, or block AI crawlers entirely.

Currently in early access for leading content creators, Cloudflare plans to expand the system widely.

“When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Roger Lynch, CEO of Condé Nast.

Ed Newton-Rex, former VP of audio at Stability AI, supported the move, calling it “the way.”

Research from Originality.AI and the Reuters Institute shows that 48% of top global websites already block AI crawlers.

Cloudflare now makes blocking AI bots the default for new domains, while existing customers can adjust settings anytime.

“You cannot have a good sort of market without scarcity,” Prince stated, emphasising the need for collective action.