How to let AI assistants read your site
If robots.txt blocks AI crawlers, assistants such as ChatGPT, Claude and Perplexity cannot cite you.
Why it matters
More people now ask an assistant instead of searching. If its crawler is blocked, the assistant cannot read your pages, so it recommends whoever it can read instead.
Steps
- Open your robots.txt file.
- Decide which AI crawlers you want to allow. Allowing them is what makes you citable.
- Remove the Disallow: / rules for those user agents.
- Reload /robots.txt to confirm the change is live.
Check it worked
Run the free AI bot checker on your site. Each crawler you chose to allow should show as allowed.
Not sure if this affects your site?
Run a free scan to see exactly which of these your website needs.