On October 26 Google added a section called “Only index content meant for Search” to its guide on preventing user-generated spam. It tells platforms to keep pages created for direct sharing, collaboration or utility, rather than for search, out of the index by default. Google’s changelog calls this link-shared content, and the guide offers two ways to handle it as if they were equals: noindex or a robots.txt block. Google’s own documentation says only one of them keeps a linked URL out of results.
What did Google change in its user-generated spam guide?
The Search Central changelog lists “Updated guidance on preventing user-generated spam” under October 26. The guide gained a section, “Only index content meant for Search”, whose rule reads: “When pages are created for direct sharing, collaboration, or utility rather than public search discovery, apply the noindex rule (using a robots meta tag or X-Robots-Tag HTTP response header) or block crawling in robots.txt by default.” The monitoring section now opens with verifying your site or subdomain in Search Console.
Two routes that Google’s docs do not treat the same
The guide presents the routes side by side, with an “or”. Google’s other pages do not. Search Engine Journal’s Matt Southern made this point, setting the new text against the robots.txt and noindex documentation, the “Control what you share” page and the “Indexed, though blocked by robots.txt” status in Search Console. His takeaway was to “check that any noindex can reach Googlebot.” We checked the same pages.
| Route | What Google’s docs say | Keeps a linked URL out? |
|---|---|---|
| noindex | Google “will drop that page entirely from Google Search results, regardless of whether other sites link to it.” | Yes, if the crawler can reach the page |
| robots.txt Disallow | “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites.” | Not reliably |
The robots.txt introduction says the file “is not a mechanism for keeping a web page out of Google” and that a disallowed URL may be indexed if it is linked from other places on the web, so “the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results.” The noindex page adds the catch: if the page is blocked by robots.txt, “the crawler will never see the noindex rule.” The “Control what you share” page, updated October 21, lists noindex for all content types and robots.txt only for images and video.
Search Console already has a label for the result. The Page Indexing report shows “Indexed, though blocked by robots.txt” and advises: “To avoid being indexed, remove the robots.txt block and use ‘noindex’.” As PPC Land put it, a share link exists to be pasted elsewhere, and a copy on a public page is the kind of link that can get a disallowed URL indexed. We read the guide’s “or” as shorthand, not as a promise that both routes work for a linked page. Google has not said which reading it intends.
October has now drawn Google’s user-content line from both ends
On October 8 Google published documentation for a push lane for user-generated content that is public, on stable URLs, attributable to a creator with a public profile and moderated; we covered it in Google’s UGC Fresh Data Program. On October 26 it told platforms to keep content “only accessed directly by people with a link” out of the index. The test in the guide is a platform’s own call: is the content “meant to be found through search engines” or not.
| Item | October 8, push lane | October 26, spam guide |
|---|---|---|
| Content | Public, attributable, moderated UGC | Shareable pages, generated outputs, interactive tools made for direct sharing |
| What Google asks | Apply; approved platforms send content to Google | noindex or robots.txt “by default” |
| Search Console | Application asks for an owner email | Monitoring starts with verifying the site or subdomain |
| Thresholds published | None for volume or user base | None for “spike” or “large share” |
Both pages stay vague where a platform would want a figure. The guide asks you to watch for “unexpected spikes in indexed pages, impressions, or queries that don’t match your platform’s purpose” and for “a small number of users generating a large share of indexable URLs”, and gives no threshold for either. The new section names no products or platforms, the guide gives no enforcement date, and the changelog files it as a guidance update, not a spam-policy change. It also does not say how a user could opt a shared page back into Search.
What to do if your product makes link-shared pages
This part is our advice, not Google’s. If your product produces share links, generated reports, calculator outputs or preview pages, decide per path whether it is meant for search. For the ones that are not, send X-Robots-Tag: noindex in the HTTP response and leave those paths out of robots.txt, so Googlebot can fetch the page and see the header.
- Open your live robots.txt and check that no Disallow rule covers the share, report or preview paths. Our robots.txt generator builds the file, and its FAQ explains that robots.txt controls crawling, not indexing.
- Test a real share URL with a header check and confirm the noindex arrives with a 200 response.
- In Search Console, verify the domain or subdomain that serves those pages, then watch the Page Indexing and Search performance reports for URLs you did not expect.
Pages meant to rank are a separate decision, and the guide does not describe that opt-in path. Keep them crawlable and indexable on purpose, and keep them off the share-link template.
