Technical SEO · RFC 9309
Robots.txt tester.
Paste a file, enter crawler product tokens and URL paths, then see the selected group and exact longest matching rule for every combination.
Rules and matrix
Rule report
Run the matrix to see allow/disallow decisions and winning lines.
How decisions are made
RFC 9309 selects case-insensitive matching product-token groups and combines repeated groups for the same token. A specific token takes precedence over the wildcard group. Rules are matched from the first path octet; the matching rule with the greatest UTF-8 byte length wins, and an Allow rule wins an equal-length conflict. * matches zero or more characters and $ anchors the end.
Robots.txt is not access control. It is a public request to cooperative crawlers. A blocked URL may still be discovered or indexed, and private content needs authentication or another real security control. Results here model RFC 9309 core rules; crawler-specific extensions may differ.