Can AI assistants read your website?
Check 14 AI crawlers at once: what your robots.txt tells each one, and how your server really answers when it visits.
A clear answer in seconds.
- A pass, warning or fail for every check
- The evidence behind each result
- What to change, in plain English
- ✓GPTBotAllowed
- ✓ClaudeBotAllowed
- ✕PerplexityBotBlocked in robots.txt
- !CCBotTurned away by server
How it works
- Enter an addressYour site, a competitor's, or any public page.
- We check it liveOur server fetches it the way search engines and bots do.
- Read what to fixEach problem comes with the reason and the fix.
Why it matters
AI assistants can only recommend pages they are allowed to read. A single line in robots.txt, or a CDN setting such as "block AI bots", can quietly keep you out of ChatGPT, Claude and Perplexity answers.
This checker reads your robots.txt the way the bots do (RFC 9309), then requests your page with each bot's name to see whether your server or firewall turns it away. Training bots and answer bots are listed separately, so you can block training while staying visible in answers.
Questions
Should I block AI bots?
Blocking training bots (GPTBot, ClaudeBot, CCBot) keeps your content out of future models. Blocking search and user bots (OAI-SearchBot, Claude-User, PerplexityBot) also removes you from AI answers and citations, which most businesses want to avoid.
Why does the server test differ from robots.txt?
robots.txt is a request; firewalls enforce rules. Some CDNs block AI user agents outright. Our test uses the bot's name but not its network, so a firewall that verifies real bots may treat the genuine crawler differently.
More free tools
- AI visibilityFreellms.txt generatorA short guide that helps AI assistants understand your site.Try it freellms.txt
# Sharma Dental Clinic> Family dental care in Pune since 2009. ## Services
- Crawling & indexingFreerobots.txt testerTest whether a URL is allowed or blocked for Googlebot, Bingbot or any AI crawler.Try it free
- ✓/blog/ · GooglebotAllowed
- ✕/admin/ · GooglebotBlocked
- ✓/blog/ · GPTBotAllowed
- Crawling & indexingPaidRaw vs rendered HTMLCompare what crawlers that skip JavaScript see with what a browser shows.See the tool
- ✓TitleSame in both
- !Main textOnly after JavaScript
- !Links38 raw · 112 rendered
Want the full picture?
This tool checks one thing. A free Pagelyt scan checks speed, SEO, AI readiness, security and more, and tells you what to fix first.