Sign in

SEO tools

AI Bot Access Checker

Test up to 100 URLs with selected crawler user agents. Compare status, robots.txt access and word counts, then export your results.

0 / 100

One URL per line. Missing https:// is added for you.

Each URL is fetched once per crawler and checked against robots.txt.

Add at least one URL to start.

Reading the results

What each result means

Every URL is requested once per crawler, with that crawler's user agent, and matched against the site's robots.txt.

  • Allowed or BlockedWhether the site's robots.txt lets that user agent crawl the URL. When a site has no robots.txt, every crawler is allowed.
  • HTTP statusWhat the server answered when the page was requested as that crawler. A 403 or 503 for only some bots usually points at a firewall or CDN rule, not robots.txt.
  • Words or redirectHow much text each crawler was served, or where a redirect sent it. Far fewer words for one bot than for the others means it is getting a different or empty page.
  • Blocked a crawler you want?Remove or narrow the Disallow rule for that user agent in your robots.txt, then run the check again to confirm.