SEO Tools
Robots.txt generator.
Configure user-agent rules, disallowed paths, crawl-delay, and sitemap reference. Output a ready-to-deploy robots.txt file.
Inputs
Result
Configure rules and press Generate.
What robots.txt does, and what it does not
A robots.txt file tells crawlers which URLs on a site they may request. RFC 9309, the Robots Exclusion Protocol standard published in September 2022, is explicit that these rules are not a form of access authorization. Google's documentation adds that robots.txt is used mainly to avoid overloading your site with requests and is not a mechanism for keeping a web page out of Google; to do that, block indexing with noindex or password-protect the page. So a disallowed URL can still appear in search results if other sites link to it, and a private directory listed in robots.txt is actually advertised to anyone who reads the file. Use robots.txt to steer well-behaved crawlers away from low-value areas, and use authentication for anything confidential.
What this generator writes
The output always begins with two comment lines. Then:
- "Allow all" is ticked by default. With it ticked the group is
User-agent: *followed by an emptyDisallow:, which means nothing is blocked. The block checkboxes and custom paths are ignored while it is ticked, so untick it first when you want to block something. - With it unticked the group is
User-agent: *followed by oneDisallow:line for each ticked common block (/admin/,/cgi-bin/,/tmp/,/private/) and each line you add under custom paths. Blank lines and lines starting with#in that box are skipped. The tool writes each path exactly as typed, so begin each one with/. - Crawl delay writes a
Crawl-delay:line when the number is above zero (see the FAQ below on who reads it). - Sitemap URL writes a
Sitemap:line with the address you give.
The result can be copied or downloaded as robots.txt.
Worked example
To block an admin area and a staging folder but keep everything else open, untick "Allow all", tick "Block /admin/", add /staging/ under custom paths and enter https://www.example.com/sitemap.xml. The output is:
# Robots.txt generated by TNAADO Tools
# https://tools.tnaado.ca/seo/robots-txt-generator
User-agent: *
Disallow: /admin/
Disallow: /staging/
Sitemap: https://www.example.com/sitemap.xmlSave it as robots.txt and upload it to the root of the site, so it is reachable at https://www.example.com/robots.txt. Google's guide describes the process as four steps: create the file, add rules, upload it to the root, and test it. The URL of the file is case-sensitive, and Google's documentation says a robots.txt file is not valid for other subdomains, protocols or port numbers than the one it is served from, so a blog on its own subdomain needs its own file.
Rules and syntax worth knowing
- Fields Google supports:
user-agent,allow,disallowandsitemap. Google's documentation says other fields, such as crawl-delay, are not supported. - Paths are case-sensitive and must begin with
/.Disallow: /Admin/does not block/admin/. - Wildcards: Google supports
*(zero or more of any character) and$(end of the URL), for exampleDisallow: /*.pdf$. This generator's fields do not stop you typing them in the custom box. - Which rule wins: crawlers use the most specific rule, based on the length of the path. For conflicts, Google uses the least restrictive rule. With
Allow: /pandDisallow: /, the URL/pageis allowed, because the allow rule is longer. - Size and caching: Google enforces a size limit of 500 kibibytes and generally caches the file for up to 24 hours, so changes are not instant.
- The sitemap line follows sitemaps.org, and Google says Google, Bing and other major search engines support it. Give the complete URL.
Common mistakes
Disallow: /by accident. It blocks the whole site. It is the most damaging single line a robots.txt file can contain, and it often arrives by copying a staging file to production. Read the output before you upload it.- Using robots.txt to remove a page from search. Blocking the crawl also stops the crawler from seeing a noindex tag on that page. To de-index, leave the page crawlable and use noindex, or remove it.
- Expecting Crawl-delay to slow Google. Google does not support it. Other crawlers may treat it differently, so check each crawler's own documentation before relying on it.
- Listing secrets. The file is public.
- Forgetting to test after upload. Fetch
/robots.txtin a browser and confirm that it loads as plain text with a 200 response.
What this tool does not do
It generates one group for all crawlers (User-agent: *). It does not build separate groups for named bots, check that your sitemap exists, or read your live file. To check whether a particular URL is blocked by a file you have written, use the Robots.txt Tester; to build the sitemap the file points to, use the Sitemap Generator, and to control indexing of a single page use the Robots Meta Tag Checker.
Frequently asked questions
What is a robots.txt file?
It is a plain text file at the root of a site that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309. The rules are requests to well-behaved crawlers and are not access control, so they do not protect private content.
Where do I put the robots.txt file?
At the root of the host, so it is reachable at https://yourdomain/robots.txt. Google's documentation says the file is valid only for the host, protocol and port number it is served from, so each subdomain needs its own file.
Does Google obey Crawl-delay?
No. Google's documentation lists user-agent, allow, disallow and sitemap as the supported fields and says other fields, such as crawl-delay, are not supported. The generator can write the line for crawlers that read it, but it will not change how fast Googlebot crawls.
Will Disallow remove a page from Google?
No. Google says robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still be listed if other pages link to it. Use a noindex directive on a crawlable page, or password-protect the page.
Why are my Block checkboxes not showing up in the output?
Because Allow all is ticked by default. While it is ticked the tool writes an empty Disallow and ignores the block options and custom paths. Untick Allow all first, then choose what to block.
Should I add my sitemap to robots.txt?
It is a good idea. Google's documentation says Google, Bing and other major search engines support the sitemap field, as defined by sitemaps.org. Use the full URL, including https.
How long until a change to robots.txt takes effect?
Not instantly. Google's documentation says it generally caches the contents of robots.txt for up to 24 hours, and may cache it longer in some situations, so allow at least a day before expecting new rules to apply.
Sources
- RFC 9309: Robots Exclusion ProtocolThe standard; rules are not a form of access authorization; longest-match rule selection. Retrieved 2026-10.
- Google Search Central: introduction to robots.txtPurpose of robots.txt and that it does not keep a page out of Google (use noindex or a password). Retrieved 2026-10.
- Google Search Central: create a robots.txt fileThe four steps: create, add rules, upload to the root, test. Retrieved 2026-10.
- Google Search Central: how Google interprets robots.txtSupported fields, wildcards, rule precedence, 500 KiB limit, 24-hour caching, host scope. Retrieved 2026-10.
- Google Search Central: robots meta tag and X-Robots-TagA noindex rule cannot be found if the URL is disallowed from crawling. Retrieved 2026-10.
- sitemaps.org: Sitemaps protocolThe Sitemap field in robots.txt and how sitemaps are discovered. Retrieved 2026-10.
Each source was opened and checked against the statements above in October 2026. Standards and vendor documentation change; follow the links for the current text.