AIO Copilot

Robots.txt Validator

Paste a robots.txt file to check every line: the syntax, fields crawlers recognise, rules before any User-agent and Disallow rules that block the whole site. It checks syntax, not whether a given URL is blocked.

How to use the robots.txt validator

Paste the live file, read the flags, then test real URLs in Search Console.

  1. Copy your live file

    Open yourdomain.com/robots.txt in a browser, copy the whole file and paste it in. Each host needs its own file, so check subdomains separately.

  2. Run the validation

    Select Validate. Every line is parsed in your browser and nothing is uploaded. Blank lines are skipped, and anything after a # is read as a comment, including at the end of a rule.

  3. Read the results

    Errors are lines crawlers cannot use, such as a rule before any User-agent line. Warnings are legal lines that may not do what you expect, such as Disallow: / or a missing Sitemap line.

  4. Test URLs with Google

    This tool checks syntax; it does not match URLs against your rules. Use URL Inspection in Search Console to see whether Google can crawl a given page, and the robots.txt report to see the file Google last fetched.

What does a robots.txt file do?

A robots.txt file tells crawlers which URLs on a host they may request. It sits at the root, such as https://example.com/robots.txt, and Google describes its main use as a way "to manage crawler traffic to your site", adding that it "is not a mechanism for keeping a web page out of Google." The format dates from 1994 and became an internet standard, RFC 9309, in September 2022.

Our glossary entry on robots.txt has the short definition. This page covers the rules Google applies when it reads the file.

How does Google read robots.txt?

Google's robots.txt specification sets out how Google reads the file:

  • Four fields count: user-agent, allow, disallow and sitemap. "Other fields such as crawl-delay aren't supported," and Google ignores invalid lines.
  • Field names are case-insensitive, but paths are case-sensitive, and a rule path must start with /.
  • Google reads the first 500 KiB of the file and ignores the rest.
  • When rules conflict, the rule with the longest matching path wins. On a tie Google uses the least restrictive rule, as RFC 9309 also recommends.
  • * matches any run of characters and $ marks the end of the URL.
  • A file covers only the host, protocol and port it is served from, so shop.example.com needs its own.
  • Google generally caches the file for up to 24 hours, so an edit is not read at once.

Why does Googlebot ignore my User-agent: * rules?

Because a crawler obeys one group only. Google says "only one group is valid for a particular crawler", the one naming the most specific user agent that matches, and "other groups are ignored." The sample file above shows the effect: Googlebot has its own group, so it skips the * group entirely, including Disallow: /admin/. If a named group should share the wildcard rules, repeat them inside it. Several User-agent lines in a row form a single group that shares the rules beneath them.

Does robots.txt stop a page from being indexed?

No. Disallow stops crawling, not indexing. Google says it "might still find and index a disallowed URL if it is linked from other places on the web," and that such a result "won't have a description." The Page indexing report lists these as "Indexed, though blocked by robots.txt". To keep a page out of results, use noindex in a robots meta tag or an X-Robots-Tag header, and leave the page crawlable: Google's noindex documentation says the page "must not be blocked by a robots.txt file" or the rule is never seen. Nor is the file a lock. Google notes that well-behaved crawlers follow it but "other crawlers might not", so private URLs need a password instead.

Which robots.txt mistakes do the most damage?

Disallow: / under User-agent: * blocks every compliant crawler from the whole site, which makes it the first line to check after any launch or migration that involved a staging site. Next comes blocking the CSS, JavaScript or images that pages need: Google says that if their absence makes a page harder for its crawler to understand, "don't block them." The validator flags Disallow rules on /css, /js, /images, /assets, /static, /media and /fonts for that reason.

The server's response matters too. Google treats a 4xx for robots.txt, other than 429, as if no file existed. A 5xx makes it stop crawling the site for the first 12 hours while it retries, then use a cached copy for up to 30 days. Crawl-delay does nothing for Google, which retired its crawl rate limiter on 8 January 2024 and now advises returning 500, 503 or 429 in an emergency, and for no more than a day or two. Bing does read it: its 2012 explanation says Bingbot treats the value as a window "from 1 to 30 seconds" in which it crawls the site once. The Host line has dropped out of use as well: Yandex's robots.txt help now lists only User-agent, Disallow, Allow, Sitemap and Clean-param.

What this validator checks, and what to use alongside it

It checks syntax and flags risky lines. It does not fetch your live file, enforce the 500 KiB limit, check that paths start with a slash, or test whether a URL is allowed. Consecutive User-agent lines are shown as one group sharing the rules beneath them, which is how crawlers read them under RFC 9309, and a # comment at the end of a rule is ignored before the path is checked. A Sitemap line must be a fully qualified URL, and the validator warns when it is not; the sitemap validator checks the file it points to.

For a live answer, use Search Console. URL Inspection tells you whether Google can crawl a given URL, and the robots.txt report shows the files Google found for the top 20 hosts on a Domain property, when each was last checked, and any parsing errors, with a button to request a recrawl after a fix. Robots.txt opens the crawling section of our technical SEO audit checklist, because one wrong line can stop Googlebot fetching the whole site.

robots.txt questions.

Location, crawl-delay, indexing, conflicting rules and how to test a URL.

Where does a robots.txt file have to be?

At the top level of each host, for example https://example.com/robots.txt. Google applies a file only to the host, protocol and port it is served from, so a subdomain or an HTTP version needs its own. The file must be UTF-8 plain text, and Google reads only the first 500 KiB.

Does Google support Crawl-delay?

No. Google's robots.txt documentation lists crawl-delay as unsupported, and Google retired the crawl rate limiter in Search Console on 8 January 2024. To slow Googlebot in an emergency, Google says to return 500, 503 or 429 status codes for no longer than one or two days. Bingbot does honor Crawl-delay, reading it as a window of 1 to 30 seconds in which it crawls the site once.

Can robots.txt remove a page from Google?

No. Google says a disallowed URL can still be indexed if other pages link to it, and the result then appears without a description. Use a noindex meta tag or X-Robots-Tag header instead, and leave that page crawlable, because Google has to fetch it to see the noindex.

What happens if a site has no robots.txt?

Crawlers treat a missing file as permission to crawl everything. Google handles any 4xx response for robots.txt, except 429, as if no file exists. A 5xx is different: Google stops crawling the site for the first 12 hours while it retries, then falls back on a cached copy for up to 30 days.

Which rule wins when Allow and Disallow overlap?

The most specific one, meaning the rule with the longest matching path. If an Allow and a Disallow rule are equally specific, Google uses the less restrictive Allow, which is also what RFC 9309 recommends. So Disallow: /shop/ with Allow: /shop/sale/ blocks the shop except the sale section.

How do I check whether a specific URL is blocked?

Use URL Inspection in Search Console, which reports whether Google can crawl that URL under your live robots.txt. The robots.txt report shows the file Google last fetched, its fetch status and any parsing errors, and lets you request a recrawl after a fix. Google also publishes its robots.txt parser as an open-source library for local testing.

Robots.txt is one line of a technical audit.

Sitemaps, canonical tags, redirects and indexing all decide whether crawlers reach the pages you care about. Our technical SEO audit checklist covers each of them in the order problems block rankings, with a free test for each check.

Read the checklist

What to do next.