Technical SEO · RFC 9309

Robots.txt tester.

Paste a file, enter crawler product tokens and URL paths, then see the selected group and exact longest matching rule for every combination.

Entirely browser-localWildcard + $ anchor supportSources checked 30 September 2026

Advertisement

Rules and matrix

Paste only. This tool never requests your site or sends the file anywhere.

Rule report

Run the matrix to see allow/disallow decisions and winning lines.

Advertisement

How decisions are made

RFC 9309 selects case-insensitive matching product-token groups and combines repeated groups for the same token. A specific token takes precedence over the wildcard group. Rules are matched from the first path octet; the matching rule with the greatest UTF-8 byte length wins, and an Allow rule wins an equal-length conflict. * matches zero or more characters and $ anchors the end.

Robots.txt is not access control. It is a public request to cooperative crawlers. A blocked URL may still be discovered or indexed, and private content needs authentication or another real security control. Results here model RFC 9309 core rules; crawler-specific extensions may differ.

Read RFC 9309 Google’s robots.txt interpretation

Worked example

The default file blocks /private/, re-allows /private/press/, blocks any URL path ending in .pdf, and gives GPTBot its own group with Disallow: /. Running Googlebot and GPTBot against six paths gives:

That is 12 combinations. The summary line reads 5 allowed and 7 blocked: Googlebot is allowed 4 of the 6 paths and GPTBot only /robots.txt.

How the sources apply

The matching follows RFC 9309, the Robots Exclusion Protocol standard: pick the group whose product token matches the crawler (otherwise the * group), collect that group's rules, and choose the matching rule with the most octets, with Allow winning a tie. Crawlers are required to parse at least 500 kibibytes of a robots.txt file, and Google documents the same 500 KiB limit. Standards last reviewed October 26, 2026.

Sources

Sources last reviewed October 26, 2026.

Advertisement

Robots.txt tester FAQ

Which rule wins when Allow and Disallow both match?

The rule with the longest matching path wins, measured in UTF-8 octets. If an Allow and a Disallow are the same length, Allow wins.

Does a Disallow line remove a page from Google?

No. Robots.txt asks cooperative crawlers not to fetch a URL, and a blocked URL can still be indexed if other pages link to it. To keep a page out of results, use a noindex rule on a page that can be crawled, or require authentication.

What does the dollar sign do?

It anchors the end of the path. Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?download=1.

Why does a specific crawler ignore the * group?

A crawler uses the most specific group that names its product token. If GPTBot has its own group, the wildcard group does not apply to it, even for paths the wildcard group allows.

Does the tester fetch my live robots.txt?

No. Paste the file text. Nothing is requested or uploaded, so it also works for staging files.