This article was prepared using automated systems that process publicly available information. It may contain inaccuracies or omissions and is provided for informational purposes only. Nothing herein constitutes financial, investment, legal, or tax advice.
Cloudflare has announced it will block AI crawlers by default for all new domains, marking a significant shift in how publishers control content access. Over a million websites and major media groups support the move, which introduces a ‘pay-per-crawl’ marketplace for AI companies. This initiative aims to rebalance power between content creators and AI firms.
- Cloudflare now blocks AI crawlers by default for new domains, with over 1 million websites already participating.
- Publishers can set fees for AI crawlers via Cloudflare's 'pay-per-crawl' marketplace, using the HTTP 402 code.
- AI firms like OpenAI and Anthropic have scraping ratios as high as 60,000:1, prompting calls for a new economic model.
📎 Related coverage from: decrypt.co