AI Week 2025: Recap
2025-09-03
How do we embrace the power of AI without losing control? That was one of our big themes for AI Week 2025. Check out all of the products, partnerships, and features we announced....
Continue reading »
The age of agents: cryptographically recognizing agent traffic
2025-08-28
Cloudflare now lets websites and bot creators use Web Bot Auth to segment agents from verified bots, making it easier for customers to allow or disallow the many types of user and partner directed....
The next step for content creators in working with AI bots: Introducing AI Crawl Control
2025-08-28
Cloudflare launches AI Crawl Control (formerly AI Audit) and introduces easily customizable 402 HTTP responses....
Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
2025-08-04
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites....
Content Independence Day: no AI crawl without compensation!
2025-07-01
It’s Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content....
MORE POSTS
July 01, 2025 10:00 AM
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. ...
- By
July 01, 2025 10:00 AM
From Googlebot to GPTBot: who’s crawling your site in 2025
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%....
- By
July 01, 2025 10:00 AM
Message Signatures are now part of our Verified Bots Program, simplifying bot authentication
Bots can start authenticating to Cloudflare using public key cryptography, preventing them from being spoofed and allowing origins to have confidence in their identity....
- By
May 15, 2025 1:00 PM
Forget IPs: using cryptography to verify bot and agent traffic
Bots now browse like humans. We're proposing bots use cryptographic signatures so that website owners can verify their identity. Explanations and demonstration code can be found within the post....
- By
March 19, 2025 1:00 PM
Trapping misbehaving bots in an AI Labyrinth
How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives....
- By
December 23, 2024 2:01 PM
Grinch Bots strike again: defending your holidays from cyber threats
Cloudflare observed a 4x increase in bot-related traffic on Black Friday in 2024. 29% of all traffic on our network on Black Friday was Grinch Bots wreaking holiday havoc....
- By
September 27, 2024 1:00 PM
AI Everywhere with the WAF Rule Builder Assistant, Cloudflare Radar AI Insights, and updated AI bot protection
This year for Cloudflare’s birthday, we’ve extended our AI Assistant capabilities to help you build new WAF rules, added new AI bot & crawler traffic insights to Radar, and given customers new AI bot ...
- By
September 23, 2024 1:00 PM
Start auditing and controlling the AI models accessing your content
Cloudflare customers on any plan can now audit and control how AI models access the content on their site. ...
- By
July 03, 2024 1:00 PM
Declare your AIndependence: block AI bots, scrapers and crawlers with a single click
To help preserve a safe Internet for content creators, we’ve just launched a brand new “easy button” to block all AI bots. It’s available for all customers, including those on our free tier...
- By