Can AI find your site?

An AI Crawl Checker tests whether AI crawlers — GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), Googlebot, CCBot, and a dozen more — can reach your website. It requests your page live with each bot's real user-agent, evaluates your robots.txt per bot, and returns a pass/fail matrix with the exact reason each blocked bot fails. If AI bots can't reach you, you can't show up in AI answers.

Breadth vs. depth. The AI Crawl Checker (this tool) is wide, shallow, and instant: it tests every major AI bot against one URL and answers a single question — which bots can reach you, and what's blocking the ones that can't. The AI Visibility audit is one bot but deep: 25+ checks across crawlability, content structure, content quality, performance, security, and accessibility. Run this tool first to make sure access is open, then run the full Visibility audit to make sure everything else is configured to win in AI.

The full matrix: GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, and Claude-User (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Googlebot and Google-Extended (Google); CCBot (Common Crawl); Bytespider (ByteDance); Amazonbot (Amazon); Applebot-Extended (Apple); and Meta-ExternalAgent (Meta). Each bot that sends real traffic is probed live with its published user-agent string; control tokens like Google-Extended and Applebot-Extended are evaluated against your robots.txt. We keep the roster updated as new AI crawlers appear.

It's a simple headline: how many of the tested AI bots can reach your page — for example, "12 of 15 AI bots can reach you." A bot counts as blocked if your robots.txt disallows it, or if a live request with its user-agent is rejected (403/429) while a normal browser gets through. For every blocked bot, the report shows the specific failure mode and the fix.

Firewalls and CDNs. Services like Cloudflare ship bot-protection settings that return 403 or 429 to AI crawler user-agents regardless of what robots.txt says. The checker catches this by comparing each bot's response against a normal browser request: if the browser succeeds and the bot fails, the block lives in your WAF/CDN configuration, not in robots.txt — and the report tells you which setting to change.

Five failure modes cover almost every case: 1) a robots.txt disallow rule for the bot (or a wildcard rule that catches it), 2) a WAF/CDN block that rejects the bot's user-agent, 3) rate-limiting that throttles crawler traffic, 4) render-blocking — the content only exists after JavaScript runs, which AI crawlers don't execute, and 5) a noindex directive in a meta tag or X-Robots-Tag header. The report identifies which one applies to each blocked bot and gives the targeted fix.

Yes. Enter a URL and you get the full report — access score, the complete bot matrix, per-bot failure diagnosis, and a copy-paste robots.txt fix. For continuous monitoring of which AI bots actually visit your site, check out Sona's Agent Analytics at sona.com.

This tool tells you which AI bots are allowed to reach your site. Sona Agent Analytics tells you which ones actually do — tracking how AI agents discover, crawl, and reference your brand in real time, with alerts when your AI visibility changes. It's the difference between permission and reality. Learn more at sona.com.