Fetch a live robots.txt or paste one in. See which search and AI crawlers can read the site, test any URL against any bot, and catch the syntax errors that make a rule do nothing.
Free, fetch or paste
Every verdict cites a line
Search and AI crawlers
What it checks
What the tester checks for you.
Three things people get wrong in a file that looks fine.
1
Who is allowed in
Every crawler is resolved against the file the way it actually resolves: its own group if it has one, otherwise the wildcard group, otherwise nothing at all.
2
Whether one path is crawlable
Enter a URL path and a bot, and see the single rule that decides it. Longest matching pattern wins and Allow beats Disallow on a tie, which is the rule most people get backwards.
3
What is silently broken
Misspelled directives, rules above the first User-agent, paths written as full URLs, missing leading slashes and duplicate groups all parse without complaint and quietly do nothing.
Honest limit
What robots.txt does not do.
What robots.txt does not do
It controls crawling, not indexing. A page blocked here can still appear in search results, without a description, if other sites link to it — Google cannot read the page to see your noindex tag precisely because you blocked it. Nor is robots.txt a security measure: it is a public file that lists the paths you would rather people did not visit, which is the opposite of hiding them.
Use a noindex meta tag to keep a page out of the index, and real authentication to keep it private.
FAQ
Questions about robots.txt.
What is a robots.txt tester?
A robots.txt tester reads a site’s robots.txt file and shows how crawlers will interpret it, rather than leaving you to work it out by eye. It reports which bots may access the site, lets you check whether a specific URL path is crawlable by a specific crawler, and flags syntax that will be ignored. It is the fastest way to find out that a rule you wrote is not doing what you assumed.
Does blocking a page in robots.txt remove it from Google?
No, and this is the most common misunderstanding about the file. robots.txt prevents crawling, not indexing, so a blocked URL can still appear in results — usually with no description — when other pages link to it. To remove a page from the index you must let Google crawl it and serve a noindex meta tag, which is impossible if robots.txt is blocking access.
What happens if a site has no robots.txt at all?
Everything is permitted. A missing robots.txt returns a 404 and crawlers treat that as unrestricted access to the whole site, which is a perfectly valid state for most sites. It only becomes a problem if you actually needed to restrict something, or if the file is missing because of a deployment error rather than a decision.
Why is my Disallow rule being ignored?
The usual causes are all silent. A rule placed above the first User-agent line belongs to no group, a misspelled directive such as "Disalow" is discarded, a path written as a full URL rather than a path never matches, and a second group for a user-agent that already has one is commonly skipped. This tester flags all four with the line number.
Can I use robots.txt to hide private pages?
No. robots.txt is publicly readable at a predictable address, so listing a path there advertises its existence to anyone curious enough to look. Well-behaved crawlers obey it and everyone else ignores it, which means anything genuinely sensitive needs authentication rather than a Disallow line.
How do Allow and Disallow interact when both match?
The longest matching pattern wins, and if two patterns are the same length the Allow takes precedence. That is why "Disallow: /admin" combined with "Allow: /admin/public" leaves the public subfolder crawlable while the rest of /admin stays blocked. This tester shows which single rule decided each result.
What happens next
01Answer a few quick questions about your business. About two minutes.
02Pick a time that suits you for a free 30-minute call with a founder.
03We study your site before the call, so you leave with a plan, not a pitch.
