Free tool

robots.txt Generator and Tester

Build a robots.txt file, then test whether a URL is allowed or blocked for a given crawler.

Build the file

A path starts with a slash. * matches anything; $ marks the end of a URL.

Use this to open one path inside a blocked one.

Also block AI crawlers

These crawlers can choose to obey robots.txt, but it is a request, not a lock.

Your robots.txt

Save it as robots.txt at the root of your site, so it's reachable at /robots.txt.

Test a URL against it

Blockedby Disallow: /admin/

How to use it

Choose what to block, add your sitemap, and copy the file to the root of your site. Then use the tester underneath to check whether a given URL is allowed or blocked for a crawler, and which rule decides it.

How the rules work

  • A crawler uses the most specific group that names it, and falls back to *.
  • Within the group, the longest matching path wins. If an Allow and a Disallow match equally, Allow wins.
  • * matches any run of characters, and $ marks the end of a URL.
  • An empty Disallow: allows everything.

Things to get right

  • robots.txt blocks crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of search, use a noindex tag and let it be crawled.
  • Don't block files a page needs to render, such as CSS and JavaScript.
  • Remove a staging Disallow: / before launch.
  • It isn't security. The file is public.
  • Blocking AI crawlers is a request that well-behaved crawlers honour, not a lock.

Where does the file go?

At the root of the host: https://example.com/robots.txt. Each subdomain needs its own.

Learn more