Technical SEO · RFC 9309
Robots.txt tester.
Paste a file, enter crawler product tokens and URL paths, then see the selected group and exact longest matching rule for every combination.
Rules and matrix
Rule report
Run the matrix to see allow/disallow decisions and winning lines.
How decisions are made
RFC 9309 selects case-insensitive matching product-token groups and combines repeated groups for the same token. A specific token takes precedence over the wildcard group. Rules are matched from the first path octet; the matching rule with the greatest UTF-8 byte length wins, and an Allow rule wins an equal-length conflict. * matches zero or more characters and $ anchors the end.
Robots.txt is not access control. It is a public request to cooperative crawlers. A blocked URL may still be discovered or indexed, and private content needs authentication or another real security control. Results here model RFC 9309 core rules; crawler-specific extensions may differ.
Worked example
The default file blocks /private/, re-allows /private/press/, blocks any URL path ending in .pdf, and gives GPTBot its own group with Disallow: /. Running Googlebot and GPTBot against six paths gives:
- Googlebot, /private/: disallowed by line 2, a 9-octet rule.
- Googlebot, /private/press/release: allowed by line 3. Both
Disallow: /private/(9 octets) andAllow: /private/press/(15 octets) match, and the longer one wins. - Googlebot, /guide.pdf: disallowed by
/*.pdf$(7 octets). /guide.pdf?download=1 is allowed, because the$anchor means the rule only matches a path that ends in .pdf. - GPTBot, every path except /robots.txt: disallowed by line 7. GPTBot has its own group, so the wildcard group's rules do not apply to it. /robots.txt is always allowed.
That is 12 combinations. The summary line reads 5 allowed and 7 blocked: Googlebot is allowed 4 of the 6 paths and GPTBot only /robots.txt.
How the sources apply
The matching follows RFC 9309, the Robots Exclusion Protocol standard: pick the group whose product token matches the crawler (otherwise the * group), collect that group's rules, and choose the matching rule with the most octets, with Allow winning a tie. Crawlers are required to parse at least 500 kibibytes of a robots.txt file, and Google documents the same 500 KiB limit. Standards last reviewed October 26, 2026.
Sources
- RFC 9309: Robots Exclusion Protocol
- Google Search Central: how Google interprets the robots.txt specification
- Google Search Central: create and submit a robots.txt file
Sources last reviewed October 26, 2026.
Robots.txt tester FAQ
Which rule wins when Allow and Disallow both match?
The rule with the longest matching path wins, measured in UTF-8 octets. If an Allow and a Disallow are the same length, Allow wins.
Does a Disallow line remove a page from Google?
No. Robots.txt asks cooperative crawlers not to fetch a URL, and a blocked URL can still be indexed if other pages link to it. To keep a page out of results, use a noindex rule on a page that can be crawled, or require authentication.
What does the dollar sign do?
It anchors the end of the path. Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?download=1.
Why does a specific crawler ignore the * group?
A crawler uses the most specific group that names its product token. If GPTBot has its own group, the wildcard group does not apply to it, even for paths the wildcard group allows.
Does the tester fetch my live robots.txt?
No. Paste the file text. Nothing is requested or uploaded, so it also works for staging files.