Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default, September 15, 2026

From DX Today | No-Hype Podcast About AI & DX Rick Spair

Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default Today Cloudflare's new crawler defaults take effect, sorting automated visitors into three declared purposes, Search, Agent, and Training, and blocking the last two by default on ad supported pages. Chris and Laura dig into the mixed use crawler problem that puts Googlebot, Applebot, and Bingbot on notice, the crawl to referral ratios showing some AI companies taking thousands of pages for every visitor they send back, ...