Build a robots.txt with crawl rules for search engines
A robots.txt generator builds the plain-text file that tells web crawlers which parts of a site they're allowed to visit — instead of you writing the syntax by hand.
Pick a preset, or set your own disallow/allow paths.
Add any custom bot rules and sitemap URLs.
Copy or download the result.
Yes — there's no paywall, no watermark, and no sign-up required.
It's a set of instructions for well-behaved crawlers — search engines, mostly — about which paths on
your site they should or shouldn't fetch. Each rule targets a user-agent (a specific bot, or *
for all of them) with Allow and Disallow paths.
Not strictly — search engines will crawl a site without one. It's useful when you specifically want to keep well-behaved crawlers out of certain paths, like an admin area, a staging section, or internal search result pages that don't need to be crawled.
Not for Google — Googlebot ignores the Crawl-delay directive entirely, no matter what value you set. Some other crawlers, like Bing's and Yandex's, do respect it. If your goal is specifically to slow down Google, this field won't do that.
Disallowing a path is advisory, not enforcement — well-behaved crawlers respect it, but it doesn't restrict access the way a password would, and it's publicly readable by anyone. A disallowed URL can still end up indexed if another page links to it, since robots.txt blocks crawling, not necessarily indexing. Don't rely on it to hide anything sensitive.
Beyond the default User-agent: * rule, add a specific bot's name (like Googlebot or
Bingbot) and give it its own set of disallowed paths — useful when you want different rules for
different crawlers instead of one blanket policy.
It has to sit in the root of your domain, reachable at exactly yourdomain.com/robots.txt —
it won't be found or respected in a subfolder.
No — this page has no upload step at all. Everything is built entirely in your browser using JavaScript, and nothing is ever sent anywhere.
Yes — there's no paywall, no watermark, and no sign-up required.
Not strictly. Search engines will crawl a site without one. It's useful when you specifically want to keep well-behaved crawlers out of certain paths, like an admin area or a staging section.
Not for Google — Googlebot ignores the Crawl-delay directive entirely. Some other crawlers, like Bing's and Yandex's, do respect it. Setting it won't slow down Google's crawling.
No, it's advisory only. Well-behaved crawlers respect it, but it doesn't restrict access the way a password would. A disallowed URL can still end up indexed if another page links to it — robots.txt blocks crawling, not necessarily indexing.
No. Everything is built entirely in your browser — nothing is uploaded or sent anywhere.
In the root of your domain, so it's reachable at yourdomain.com/robots.txt exactly — it won't be found in a subfolder.