Free robots.txt Generator

Build a valid robots.txt with rules, presets, and a sitemap — copy or download it in seconds.

Robots Exclusion Protocol

robots.txt Generator

Build a valid robots.txt file with Allow/Disallow rules, crawl-delay, and sitemap — copy it in seconds

Crawler block
The crawler this block applies to. Use * for all bots, or a name like Googlebot.
Paths are relative to the domain root, e.g. /wp-admin/ or /private/.
Optional directives
Seconds a crawler should wait between requests. Honored by Bing/Yandex; Google ignores it.
Absolute URL of your XML sitemap. Add one per line by separating with commas.
Presets
Syntax valid

What is a robots.txt generator?

A robots.txt generator is a free tool that helps you create a robots.txt file — the plain-text file at the root of your domain that tells search-engine crawlers which URLs they may or may not request. The file follows the Robots Exclusion Protocol: each block names a User-agent and lists Allow and Disallow rules, with optional Crawl-delay and Sitemap directives. This online robots.txt generator lets you add rules, pick a preset (allow all, block all, WordPress defaults, or block AI bots), declare your sitemap, and copy a syntactically valid file in seconds. It runs 100% in your browser — nothing you type is ever sent to a server.

How to Create a robots.txt File: Step by Step

A robots.txt file controls crawler access at the path level. This guide walks you through generating a valid file and putting it in the right place.

1

Choose a User-agent

Enter the crawler the rules apply to. Use an asterisk (*) to target all bots, or a specific name such as Googlebot or Bingbot for a single crawler.

Tip: Most sites only need one block with User-agent: *. Add named blocks only when you want different rules for a specific bot.
2

Add Allow and Disallow Rules

Click "Add rule" to list the paths you want to control. Choose Disallow to block a path or Allow to permit one, then type the path beginning with a slash, for example /wp-admin/. Leaving a Disallow value empty allows everything.

3

Set Crawl-delay and Sitemap (Optional)

Add a crawl-delay in seconds if you want compliant crawlers to slow down — note that Google ignores this directive. Add the absolute URL of your XML sitemap so crawlers can discover all your pages.

4

Review the Validation Notices

The tool runs a lightweight syntax check as you type and flags common mistakes — such as a path that does not start with a slash, or a sitemap that is not an absolute URL. Fix any warnings before you publish.

Tip: robots.txt is case-sensitive and is matched as a prefix. /Admin and /admin are different paths.
5

Copy or Download the File

Click "Copy" to put the generated file on your clipboard, or "Download" to save it as robots.txt. Everything happens in your browser — nothing is uploaded.

6

Upload to Your Domain Root

Place the file at the root of your domain so it is reachable at https://www.example.com/robots.txt. It must live at the root — crawlers do not read a robots.txt in a subdirectory. Then test it in Google Search Console.

Frequently Asked Questions

What is a robots.txt file?
robots.txt is a plain-text file placed at the root of a domain that tells search-engine crawlers which URLs they are allowed to request. It follows the Robots Exclusion Protocol: each block names a User-agent and lists Allow and Disallow rules. It is a crawling instruction, not a security mechanism — well-behaved bots obey it, but it cannot force compliance.
Does robots.txt block a page from being indexed?
No. robots.txt controls crawling, not indexing. If a disallowed URL is linked from elsewhere, Google can still index it (often showing it without a description) because it knows the URL exists even though it cannot crawl the content. To keep a page out of the index, allow crawling and use a noindex meta robots tag or X-Robots-Tag header instead — Google must be able to crawl the page to see the noindex directive.
Where do I put the robots.txt file?
It must live at the root of your domain, reachable at https://www.example.com/robots.txt. Crawlers only read robots.txt from the root — a file in a subdirectory like /blog/robots.txt is ignored. Each subdomain needs its own robots.txt.
What is the difference between Allow all and Disallow all?
"Allow all" means crawlers may request every URL — this is achieved with an empty Disallow: line (or no Disallow at all). "Disallow all" blocks every URL with Disallow: /, which tells compliant crawlers to request nothing on the site. An empty Disallow value permits everything; a single slash blocks everything.
What is the difference between robots.txt and meta robots?
robots.txt is a single site-wide file that controls whether crawlers may request a URL at all. The meta robots tag is an HTML tag on an individual page that controls indexing and link-following (for example noindex, nofollow) after the page is crawled. Use robots.txt to manage crawl budget and block sections; use meta robots to keep a specific crawlable page out of search results.
Does robots.txt affect SEO?
Indirectly. A correct robots.txt helps search engines spend crawl budget on your important pages and avoids wasting it on duplicate or admin URLs. A misconfigured one can hurt you badly — accidentally disallowing your whole site, or blocking CSS and JavaScript that Google needs to render pages, can cause ranking and rendering problems. Always test changes in Google Search Console before publishing.

More free tools: browse the full catalog →