robots.txt tester
Load your live robots.txt, test any URL against any crawler, and try edits safely before you publish them.
| Path | Googlebot | Decided by |
|---|---|---|
| / | Allowed | No matching rule · group * |
| /admin/settings | Blocked | Line 2: Disallow: /admin · group * |
| /admin/help | Allowed | Line 4: Allow: /admin/help · group * |
| /blog/my-first-post | Allowed | No matching rule · group * |
Sitemaps listed: https://example.com/sitemap.xml
A clear answer in seconds.
- A pass, warning or fail for every check
- The evidence behind each result
- What to change, in plain English
- ✓/blog/ · GooglebotAllowed
- ✕/admin/ · GooglebotBlocked
- ✓/blog/ · GPTBotAllowed
- •Rule usedDisallow: /admin/
How it works
- Enter an addressYour site, a competitor's, or any public page.
- We check it liveOur server fetches it the way search engines and bots do.
- Read what to fixEach problem comes with the reason and the fix.
Why it matters
One wrong Disallow line can hide a whole section of your site from Google. This tester applies the same rules search engines use: the most specific matching rule wins, and Allow wins a tie.
Each result shows the exact line that decided it, so you know what to change. Edits here are only in your browser until you copy them to your site.
Questions
Does Disallow remove a page from Google?
No. It stops crawling, not indexing: a blocked page can still appear in results if other sites link to it. Use a noindex tag on a crawlable page to keep it out.
Which rule wins when two match?
The one with the longest path. If they are the same length, Allow wins. A bot that has its own User-agent group ignores the * group entirely.
More free tools
- Crawling & indexingPaidRaw vs rendered HTMLCompare what crawlers that skip JavaScript see with what a browser shows.See the tool
- ✓TitleSame in both
- !Main textOnly after JavaScript
- !Links38 raw · 112 rendered
- Crawling & indexingFreerobots.txt generatorDecide which search and AI crawlers may visit, and download the file.Try it freerobots.txt
User-agent: *Allow: /Disallow: /admin/
- Crawling & indexingFreehreflang checkerValidate language tags and confirm every alternate links back.Try it free
- ✓en-IN → /en-in/Links back
- ✓hi-IN → /hi/Links back
- ✕en-US → /us/Missing return link
Want the full picture?
This tool checks one thing. A free Pagelyt scan checks speed, SEO, AI readiness, security and more, and tells you what to fix first.