Robots.txt Generator_
robots.txt controls crawling, not indexing. A page you block can still appear in search results — Google finds it through links elsewhere and lists it without a snippet, which is what Search Console means by Indexed, though blocked by robots.txt. Worse, blocking a page you want removed guarantees it stays: the crawler can no longer fetch it, so it never sees the noindex that would have taken it out.
The second thing no generator mentions is that Disallow is a prefix match — /admin also blocks /administrator — and that the rule which wins is the longest one, not the first. So this writes the file and then tells you which rule actually applies to a URL, and why.
Check a URL against these rules
- Input
- Rules built from the controls, a preset, or an existing robots.txt pasted straight into the box — it is the source of truth and stays editable.
- Output
- A complete robots.txt, plus a verdict for any URL you test against it naming the group, the winning rule and the reason it won.
- Processing
- Parsed and evaluated in this tab using the matching rules from RFC 9309 — most specific group, longest matching pattern, Allow winning a tie.
- Limits
- This reasons about the file. It cannot fetch your site, confirm the file is reachable at the root, or tell you what Googlebot has already indexed.
- Why only one group ever applies
- A crawler obeys the single most specific group whose user-agent token matches it, and ignores every other group in the file — including the * group. So a rule added to * that is meant to also cover Googlebot does nothing at all once a Googlebot group exists. This is the quietest way a robots.txt stops doing what its author intended.
Blocking a page is not removing it
Crawling and indexing are different things
A Disallow tells a crawler not to fetch a URL. It says nothing about whether that URL may appear in results, and Google is explicit that it may: if the address is linked from anywhere it can be indexed on the strength of the link alone, listed with the URL and no description. That listing is the warning Indexed, though blocked by robots.txt, and it is not a bug. It is the file doing exactly what it says.
Blocking a page keeps it indexed
This is the part that costs people months. To remove a page from the index you serve noindex — a meta tag or an X-Robots-Tag header — and the crawler has to fetch the page to see it. Add a Disallow at the same time and you have blocked the only route by which the instruction could arrive, so the old listing stays indefinitely. The correct order is: allow crawling, serve noindex, wait for it to be picked up, and only then block the path if you still want to save the crawl budget.
There is no Noindex directive here
A Noindex: line in robots.txt was an undocumented Google behaviour that never appeared in any standard. Google announced its removal in July 2019 and stopped honouring it that September; no other major crawler ever supported it. It is still widely copied from old advice, and it does nothing — the file parses, the line is ignored, and the page carries on being indexed.
Disallow matches a prefix, not a path
Disallow: /admin blocks /admin, and also /administrator, /admin-tools and /administration. Nothing in the syntax suggests a boundary, because there is not one — the value is compared as a string prefix against the path. A trailing slash confines it to the directory, and a trailing $ matches exactly there and nowhere further. Both are one character, and leaving them off is the most common way a robots.txt blocks more of a site than anyone realised.
The longest rule wins, not the first
Within the group that applies, every rule is tested and the one with the longest matching pattern decides the outcome; Allow wins when the lengths tie. Order is irrelevant, which is why the WordPress convention works — Allow: /wp-admin/admin-ajax.php is longer than Disallow: /wp-admin/ and therefore beats it, wherever it appears in the file. Reading top to bottom instead is what makes people conclude that an Allow below a Disallow is ignored.
It is a request, and it is public
Nothing enforces any of this. A crawler that intends to ignore the file simply does, and the file itself is readable by anyone at a fixed, guessable address — so a Disallow naming a sensitive path is an advertisement for it. Anything that must not be reached needs authentication. The file is worth writing for the crawlers that do behave, which is most of the ones that matter, and it is worth writing with the assumption that everyone will read it.
Build it, then check what it does
- 01Start from a preset, or paste a robots.txt you already have into the box to check it instead.
- 02Add rules with the controls above, or type in the box directly — it is the source of truth and everything reacts to it.
- 03Test a URL. The verdict names the group that applied, the rule that won and why it won, which is the question the file itself cannot answer.
- 04Read the notes. They fire on the prefix trap, on rules that a specific crawler will never see, and on directives that no longer do anything.
- 05Save it as robots.txt at the root of the host — not in a subdirectory, and not shared between subdomains.
A page that will not leave the index
It was blocked months ago and it is still listed. Blocking is why — the noindex on it has never been fetched.
Disallow: /old-page
+ <meta name="robots"
content="noindex">Remove the Disallow. Let it be crawled, let the noindex be seen, then block it.
A rule that blocked more than intended
Traffic to a whole section disappeared, and the robots.txt looks like it names one directory.
Disallow: /news
/news, /newsletter, /news-archive, /newsroom A trailing slash fixes it.
A rule Googlebot never sees
A Disallow added to the wildcard group has no effect on Google, because Google has its own group further down the file.
User-agent: * Disallow: /private/ User-agent: Googlebot Disallow: /tmp/
Only /tmp/. One group applies. The * group is ignored completely.
Keeping AI crawlers out
Each one needs its own group. Adding the tokens to a shared user-agent line looks tidier and does not work the way people expect.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Google-Extended covers AI training and is separate from Googlebot, which is search.
What each line does, and what it does not
| Line | Effect | Caveat |
|---|---|---|
User-agent: * | Opens a group for any crawler with no group of its own | A crawler with its own group ignores this one entirely |
Disallow: /path | Asks that URLs starting with those characters not be fetched | Prefix match — also blocks /pathological |
Disallow: /path/ | Confines it to the directory | Still crawling only. The URLs can remain indexed |
Disallow: /path$ | Matches exactly that path | $ is a Google extension, honoured by Bing too |
Disallow: | Empty value — no restriction at all | Means the opposite of Disallow: /, one character apart |
Allow: /path | Carves an exception out of a broader Disallow | Only wins if it is longer, or equal and therefore tied |
Sitemap: https://… | Points crawlers at a sitemap | Must be an absolute URL, and belongs to no group |
Crawl-delay: 10 | Asks for a pause between requests | Google ignores it. Bing and Yandex honour it |
Noindex: /path | Nothing whatsoever | Removed by Google in September 2019 and never standard |
The two lines most often confused are Disallow with an empty value and Disallow: / — one permits everything and the other forbids everything, and they differ by a single character.
Writing a robots.txt that behaves
- End directory rules with a slash. Without one the value is a prefix and will block neighbouring paths that happen to start the same way.
- Use noindex for anything that must leave the index, and leave it crawlable until it has gone. Blocking and deindexing are opposite instructions.
- Repeat rules inside every specific group. A crawler obeys one group only, so anything in the wildcard group is invisible to a crawler that has its own.
- Add the Sitemap line. It is the only directive here that helps discovery instead of restricting it, and it is independent of every group.
- Test the URLs you care about rather than reading the file. Longest-match resolution is not obvious by eye once there is more than one rule.
- Never put a secret path in it. The file is public at a fixed address, so listing something is closer to publishing it than to hiding it.
- Keep one file per host and protocol. https://example.com/robots.txt does not govern the http site, a subdomain, or a different port.
Where robots.txt goes wrong
Blocking a page prevents it being removed
A crawler that cannot fetch a page cannot see the noindex on it. Disallow plus noindex is a contradiction, and the one that wins is the block — so the listing stays, sometimes for years.
Disallow: / on a live site is catastrophic and quiet
It is one character away from the empty Disallow that means the opposite, it produces no error anywhere, and the effect appears gradually as pages drop out. It is the usual result of a staging file being deployed to production.
A specific group silently cancels the wildcard
Adding a Googlebot group means Googlebot stops reading the * group completely, including every rule that was already there. Nothing warns about this, and the file still looks correct.
Blocked resources change how pages are rendered
Disallowing CSS or JavaScript directories stops the renderer fetching them, so Google evaluates the page without its styles or its scripts. This was a common way to make a responsive site look broken to a crawler.
The file is an index of what you would rather hide
It is public, at a predictable URL, and it is one of the first things any scanner reads. Naming an admin path in it does not protect that path — it advertises it.
Format, matching and limits
- Standard
- RFC 9309, published in 2022, which finally standardised what had been a convention since 1994. Google, Bing and the major crawlers follow it.
- Location
- Exactly /robots.txt at the root of a host. It governs one scheme, host and port — a subdomain, the http version and a non-default port each need their own.
- Group selection
- The most specific matching user-agent token wins, compared case-insensitively as a prefix of the crawler name. Only that group applies; * is the fallback used when nothing else matches.
- Rule selection
- Within the group, the longest matching path pattern decides, and Allow wins a tie of equal length. Order in the file is irrelevant.
- Wildcards
- * matches any sequence and $ anchors to the end of the path. Both are extensions rather than part of the original convention, and both are honoured by Google and Bing.
- Size
- Google reads at least 500 kibibytes and ignores everything past it, so a very long file can silently lose its later rules.
- Missing file
- A 404 means everything is allowed. A 5xx is treated by Google as everything being disallowed, which makes a broken robots.txt endpoint more dangerous than an absent one.
- Network
- None from tool code. A test sweep calls every function this page uses with
fetchandXMLHttpRequestreplaced by stubs that throw, so a stray request fails the build instead of shipping. Disconnect from the network and the page still works.
Questions about robots.txt
Does robots.txt stop a page appearing in Google?
No. It asks crawlers not to fetch the page, and a URL that is linked from anywhere can still be indexed on the strength of those links — listed with no title or description. Search Console reports this as "Indexed, though blocked by robots.txt". Use a noindex meta tag or X-Robots-Tag header to keep a page out of the index.
Why is a page indexed though it is blocked by robots.txt?
Because blocking prevents crawling, not indexing. Google learned the URL from a link, could not fetch the page to see any noindex on it, and so listed the address alone. The fix is to unblock the page, serve a noindex, wait for it to be recrawled, and only then consider blocking the path again.
Can I use noindex in robots.txt?
No. A Noindex: line was an undocumented Google behaviour that never appeared in any standard, and Google stopped honouring it in September 2019. It is still widely copied from old guides and it does nothing at all.
What does Disallow: / mean?
It asks every crawler in that group not to fetch anything on the host. It is one character away from an empty Disallow, which means the opposite — no restriction at all — and confusing the two on a live site removes it from search results over the following weeks.
Does Disallow: /admin block only that directory?
No. The value is matched as a prefix, so it also blocks /administrator, /admin-tools and anything else beginning with those characters. Write /admin/ to confine it to the directory, or /admin$ to match exactly that path.
Which rule wins when Allow and Disallow both match?
The longest pattern, and Allow if they are the same length. Position in the file makes no difference. This is why Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ — it is longer, not because it comes after.
Do I need a separate group for each crawler?
Only if you want different rules for it, and if you do then that group has to be complete. A crawler obeys the single most specific group that matches and ignores every other one, so rules left in the * group will not apply to it.
How do I block AI crawlers?
Give each one its own group with Disallow: / — GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended are the common ones. Note that Google-Extended governs AI training and is separate from Googlebot, so blocking it does not affect search. All of it depends on the crawler choosing to obey.
Does Google respect Crawl-delay?
No. Google sets its own crawl rate and offers no way to raise or lower it from robots.txt. Bing and Yandex do honour the directive, so it is worth including if those matter to you — it simply will not do what it is usually added for.
Where does the file have to go?
At /robots.txt on the root of each host. It applies to one scheme, host and port, so a subdomain needs its own file, and the https and http versions are separate as far as crawlers are concerned. A file in a subdirectory is never read.
Is the file I build here uploaded anywhere?
No. The work is JavaScript running in this tab. Every function it calls is covered by a test that stubs fetch and XMLHttpRequest to throw, so a request that slipped in would break the build rather than reach a server — and you can confirm it for yourself by disconnecting and carrying on.