The next step for content creators in working with AI bots: Introducing AI Crawl Control
2025-08-28
Cloudflare launches AI Crawl Control (formerly AI Audit) and introduces easily customizable 402 HTTP responses....
Continue reading »
Content Independence Day: no AI crawl without compensation!
2025-07-01
It’s Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content....
The crawl before the fall… of referrals: understanding AI’s impact on content providers
2025-07-01
Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This helps site owners make decisions about which AI bots to allow or block. ...
Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
2025-07-01
Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf. ...
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
2025-07-01
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. ...
MORE POSTS
July 01, 2025 10:00 AM
From Googlebot to GPTBot: who’s crawling your site in 2025
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%....
- By
July 01, 2025 10:00 AM
Message Signatures are now part of our Verified Bots Program, simplifying bot authentication
Bots can start authenticating to Cloudflare using public key cryptography, preventing them from being spoofed and allowing origins to have confidence in their identity....
- By