PagelytBot
If you found PagelytBot in your server logs, someone used Pagelyt to check a website. This page explains what it fetched and why.
How to recognise it.
Requests from Pagelyt carry this user agent:
PagelytBot/0.1 (+https://pagelyt.dev/bot)Requests come from Cloudflare's network, so the IP addresses belong to Cloudflare and can change.
When it visits.
PagelytBot does not crawl the web on its own. It only visits a website when a person asks for a report on it, when an owner has scheduled rescans of their own site, or when a paid account compares its site with a competitor. Each report makes a small, fixed number of requests and then stops.
Your pages
The homepage, then up to 30 pages of the same site, taken from your sitemap first and then from internal links.
Standard files
robots.txt, sitemap.xml, llms.txt and security.txt, to report on whether they exist and are set up correctly.
A sample of links and images
A handful of links and images from the page, to find broken links and oversized files.
Exposure checks
A few well-known paths such as /.env and /.git/config, only to warn you if private files are publicly readable.
Other requests in a report.
Some checks can't use the PagelytBot name, because they test how your site treats other visitors:
- The AI crawler checker requests your homepage once with a normal browser user agent and once with each AI crawler's name, to show whether your server treats them differently.
- Speed scores come from Google PageSpeed Insights, which loads the page from Google's servers.
- Uptime checks, which owners can turn on for their own sites, send one homepage request at a time as
PagelytUptime/1.0. - Screenshots and PDFs are taken with a headless Chrome browser run by Cloudflare.
How to block it.
Reports are started by people for a specific site, so PagelytBot reads your robots.txt to report on it but doesn't use it to skip pages. To stop it, block requests whose user agent contains PagelytBot (and PagelytUptime) in your firewall or server settings. Pagelyt will then show an error instead of a report.
Something wrong, or too many requests? Email customer_care@pagelyt.dev or get in touch, and include the site and the time from your logs.