Every AI crawler request lands as a line in a server log. Until August 13, 2026, Microsoft Clarity counted those requests and counted AI referral visits, with no single metric setting one against the other. The new AI Scrape-to-Referral Ratio card, added to the Bot Activity section of Clarity’s AI Visibility dashboard, sets the two sides side by side: how often a given AI operator’s crawler requests a page, against how many visits from that same operator followed. Microsoft says AI assistants “are becoming a new front door to the web, changing how people discover brands, products, and content.”
What the Card Adds
The ratio compares AI scrape activity against referral traffic in a single metric, per Microsoft’s announcement post. Bot Activity also ships overall scrape-to-referral performance, an operator-level breakdown ranking AI sources by traffic sent back, direct links into session recordings filtered by referral source, and engagement data on scrolling, conversions, form fills, purchases and sign-ups. Microsoft states: “With the new AI Scrape-to-Referral Ratio card and direct links into session recordings, you can evaluate the tradeoff between content extraction and traffic return.” The card exists, in Search Engine Land’s framing, to judge “whether it is worth having that AI bot crawl your site, and whether there is enough return and value to keep allowing that bot to scrape your website.”
What is the AI Scrape-to-Referral Ratio?
It is a Clarity dashboard metric, announced August 13, 2026, comparing how often an AI crawler requests a site’s pages against how many visits that same operator refers back. It reports an overall figure and a per-operator breakdown inside the Bot Activity section of AI Visibility, and depends on server-side log data from a connected CDN or server. Microsoft’s announcement does not give a formula for the ratio, and no target figure has been published for what counts as healthy.
One Published Example, Not a Benchmark
Microsoft’s post carries no numbers at all. Search Engine Land’s write-up of the report does: one worked example shows 41 referrals against a ratio of roughly 6,000 scrapes for every referral. That figure belongs to Search Engine Land’s look at the report, not to Microsoft, and neither source calls it typical. Whether 6,000-to-1 counts as acceptable “depends on the site, the business, and the purpose of the website,” per Search Engine Land, because Microsoft has not published a target number to compare against.
| What the ratio counts | What it leaves open |
|---|---|
| A request reaching the server | Whether the content was retrieved at all |
| Referral visits attributed to that AI source | Whether content was grounded, cited, or surfaced in an AI answer |
| Domains that map cleanly between bot and referral data | Domains that don’t map; those get context, not a ratio |
That distinction comes from Microsoft’s own Bot Activity documentation: “Bot activity represents requests made to your site and doesn’t indicate that content was retrieved, grounded, cited, or surfaced in AI generated responses. These metrics reflect observed bot behavior and may not translate into traffic, attribution, or downstream outcomes.” The same documentation describes bot activity as the earliest observable signal in the AI content lifecycle, ahead of grounding, citation and referral. A scrape is a request. It is not proof the content was used.
What It Costs to Turn On
The ratio only exists once server-side logs are flowing into Clarity. The data “is based on real server-side logs collected through supported CDN and server integrations,” wired up under Settings, AI Visibility, then a CDN or server connection. On WordPress, Microsoft says the integration “is enabled by default when using the latest Clarity WordPress plugin.” That connection is not necessarily free: Microsoft warns “Connecting to your server or CDN integration might result in additional costs depending on your provider, cloud platform, traffic volume, and regional configuration,” billed by the provider, not Microsoft. Sustained high-volume bot activity, per that documentation, “can introduce infrastructure overhead, degrade performance, and create operational risks” on top of whatever the connection bills.
A Number for a Trade That Was Already Pricing by Hand
The ratio formalizes a decision publishers have been making without a shared metric. We reported on TIME.com serving ad-bearing markdown to LLM crawlers, where TIME sees more bot traffic than human traffic on most days, and where the response logs show the crawler that answers a question and the crawler that trains a model treated as two separate commercial decisions. Clarity’s card does not change that math; it adds one input to it, how much a given AI operator requests against how much it sends back, limited to domains where the comparison holds up. Acting on that number is a policy call. A robots.txt file declares the policy; it does not enforce it. A site owner can still rewrite the crawler permissions a robots.txt file lists without waiting on Microsoft or the crawler operator.
