Tag: robots.txt

SEO

Cloudflare Adds Disallow AI Training: Its Block Setting Now Stops Googlebot Too

Cloudflare changed what its Block setting does to mixed-use crawlers on September 15, and added a Training setting it says…

Sep 16, 2026 • 7 min read

SEO

Apple Updates Applebot Doc: Blocking AI Training Carries No Search Ranking Cost

Apple’s support documentation for Applebot now carries a sentence it didn’t have in late August: “Site rules for Applebot-Extended are…

Sep 7, 2026 • 5 min read

Web Analytics

Microsoft Clarity Adds AI Scrape-to-Referral Ratio: No Formula, No Benchmark

Every AI crawler request lands as a line in a server log. Until August 13, 2026, Microsoft Clarity counted those…

Aug 14, 2026 • 4 min read