Robots.txt Checker
Check if your website is accessible to search engines and AI crawlers like Googlebot, ChatGPT, Claude, and Perplexity. Analyze your robots.txt, sitemap, and llms.txt.
Frequently Asked Questions
Does this match how Google reads robots.txt?
It applies the same rules Google documents: the most specific (longest) matching path wins regardless of order, and on a tie the least restrictive rule wins, so Allow beats Disallow. Google also ignores anything past the first 500 KiB of the file, so keep it small.
Will blocking a page in robots.txt remove it from search results?
No, and this is the most common mistake. Disallow stops crawling, not indexing — a blocked URL can still appear in results from external links, just without a description. To actually remove a page, allow the crawl and serve a noindex meta tag or X-Robots-Tag header.
Can I block AI crawlers like GPTBot in robots.txt?
Yes. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot all honour robots.txt, so a per-user-agent Disallow works. Note that blocking them also removes you from the answers those assistants give, which is increasingly a source of referral traffic.
Is the Robots.txt Checker free to use?
Yes. Every tool on this site is completely free, with no account, no signup, and no paid tier to upgrade to.
Do I need to install a browser extension or any software?
No. The Robots.txt Checker runs online, in the browser you already have. It works the same on desktop and mobile, and there is nothing to download, install, or add to Chrome.
Related Tools
Related Reading
Why Crawler Accessibility Matters
Your robots.txt file controls which bots can access your site. With AI-powered search becoming more prevalent, ensuring your content is accessible to both traditional search engines and AI crawlers is crucial. Here's what we check:
robots.txt Analysis
We check if your robots.txt file blocks common AI crawlers like GPTBot (OpenAI), Claude-Web (Anthropic), Google-Extended, and others. Blocking these bots means AI services can't use your content in their responses.
Sitemap Detection
A properly configured sitemap.xml helps all crawlers, including AI bots, discover and index your content efficiently. We verify its presence and if it's referenced in robots.txt.
llms.txt Support
The emerging llms.txt standard allows you to provide specific instructions to AI systems about how to handle your content. We check if your site has adopted this new standard.
Common AI Bots We Check:
- GPTBot - OpenAI/ChatGPT
- Claude-Web - Anthropic/Claude
- Google-Extended - Google AI
- PerplexityBot - Perplexity AI
- CCBot - Common Crawl
- Bytespider - ByteDance/TikTok
Need Help With Your SEO?
Our team can help configure your site for optimal search engine and AI accessibility while maintaining control over your content. Book a free discovery call to discuss your needs.