FAQSign inAnalyse a website

robots.txt generator

Build a clean robots.txt: keep private paths out of crawlers, choose which AI bots may visit, and point to your sitemap.

Crawling & indexingGeneratorFree
robots.txt generatorRuns in your browser
Block these AI crawlers

Leave all unticked to stay visible to AI assistants. Blocking them means they can’t cite you.

One per line
robots.txt6 lines
User-agent: *
Disallow: /admin
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

Checking …

One moment while we look up this website.

What you’ll get

A ready-to-use file, no code needed.

  • A live preview as you fill it in
  • Copy or download in one click
  • Hand it to your developer, or paste it in yourself
Example result

How it works

  1. Fill in the detailsSigned in, we pre-fill them from your latest report.
  2. Preview the resultSee exactly what will be published as you type.
  3. Copy or downloadPaste it into your site, or send it to your developer.

Why it matters

  • robots.txt is the first file every crawler reads. A short, correct one keeps admin and checkout pages out of search and tells crawlers where your sitemap is.

  • Test the result with the robots.txt tester before you upload it.

Questions

Can robots.txt keep a page secret?

No. The file is public and only polite bots follow it. Protect private pages with a login instead.

Do I need a robots.txt at all?

Without one, crawlers assume everything is allowed, which is fine for many sites. Adding one is still the easiest way to list your sitemap.

More free tools

13 tools. One free account.Check AI access, crawling, how you look in Google, security and more. Nothing to install.Browse all tools

Want the full picture?

This tool checks one thing. A free Pagelyt scan checks speed, SEO, AI readiness, security and more, and tells you what to fix first.

Analyse My WebsiteBook a consultation