Can AI crawl your site?

An AI Crawl Checker tests whether AI crawlers, GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), Googlebot, CCBot and a dozen more, can reach your website. It requests your page live with each bot’s real user-agent, evaluates your robots.txt per bot, and returns a pass/fail matrix with the exact reason each blocked bot fails. If AI bots cannot reach you, you cannot show up in AI answers.

Breadth versus depth. The AI Crawl Checker is wide, shallow and instant: it tests every major AI bot against one URL and answers a single question, which bots can reach you and what is blocking the ones that cannot. The AI Visibility audit is one bot but deep: 25+ checks across crawlability, content structure, content quality, performance, security and accessibility. Run this tool first to make sure access is open, then run the full audit.

The full matrix is 45 bots: GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, Claude-User and Claude-Code (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Googlebot, GoogleOther, Google-Extended, Google-CloudVertexBot, Google-NotebookLM, Google-Agent, Gemini-Deep-Research and GoogleAgent-URLContext (Google); Bingbot (Microsoft); Applebot and Applebot-Extended (Apple); CCBot (Common Crawl); Bytespider and TikTokSpider (ByteDance); Amazonbot, Amzn-SearchBot, Amzn-User and bedrockbot (Amazon); DuckAssistBot (DuckDuckGo); Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer and FacebookBot (Meta); MistralAI-User (Mistral); AI2Bot (Allen Institute); PetalBot and PanguBot (Huawei); cohere-training-data-crawler (Cohere); Diffbot; YouBot (You.com); ExaBot (Exa); Kimi-User (Moonshot AI); TongyiBot (Alibaba); ChatGLM-Spider (Zhipu AI); and Webzio-Extended (Webz.io). Bots that send real traffic are probed live with their published user-agents; control tokens like Google-Extended are evaluated against your robots.txt; and fetchers documented to ignore robots.txt are judged by live probe only. Operators with no honored user-agent, like xAI’s Grok, DeepSeek, OpenAI’s Operator, Devin, Manus and Tavily, are flagged as a separate stealth tier rather than given a misleading pass/fail row.

It is a simple headline: how many of the tested AI bots can reach your page, for example "41 of 45 AI bots can reach you." A bot counts as blocked if your robots.txt disallows it, or if a live request with its user-agent is rejected (403 or 429) while a normal browser gets through. For every blocked bot, the report shows the specific failure mode and the fix.

Seven failure modes cover almost every case: 1) a robots.txt disallow rule for the bot (or a wildcard rule that catches it), 2) a WAF/CDN block that rejects the bot's user-agent, 3) a soft block, the bot gets HTTP 200 but the body is a challenge page or a login/block-page redirect, 4) cloaking, the bot receives a stripped page with a fraction of the content a browser gets, 5) rate-limiting that throttles crawler traffic, 6) render-blocking, the content only exists after JavaScript runs, which AI crawlers don't execute, and 7) a noindex directive in a meta tag or X-Robots-Tag header. There's also pay-per-crawl (HTTP 402), which the report labels as deliberate monetization rather than a failure. The report identifies which one applies to each blocked bot and gives the targeted fix.

Because modern bot protection rarely sends an honest 403. Cloudflare, DataDome, and similar services often return HTTP 200 with a challenge page ("Just a moment...") that contains none of your content, or serve bots a stripped version of the page. Status-code-only checkers mark these as passing. This tool reads the body each bot actually receives and compares it against what a normal browser gets on the same URL: challenge-page signatures, block-page redirects, and content that shrinks to under a quarter of the browser version all get flagged, with the vendor-specific fix.

Yes. Beyond per-bot access, every report checks the discovery files crawlers read before your pages: robots.txt health (5xx responses, malformed rules, crawl-delay traps), sitemap presence and reachability, and llms.txt, whether it exists, conforms to the spec, and whether its links resolve. It also samples one URL per path template from your sitemap and re-tests representative bots against each, so a /blog that passes doesn't hide a /docs that blocks. If you do not have an llms.txt yet, the free llms.txt generator crawls your site and produces an editable, validated file ready to deploy.

Two newer layers the report covers beyond raw access. Policy signals are the emerging standards that declare how AI systems may use your content: Content-Signal directives in robots.txt, RSL licensing (the License directive backed by Reddit, Yahoo, and 50+ publishers), noai directives, ai.txt, TDM-Rep, and the nosnippet/max-snippet preview controls that quietly remove pages from Google's AI Overviews. Agent readiness covers the conventions that make your site cheaper for AI agents to consume: serving markdown via content negotiation (Accept: text/markdown), llms-full.txt, an MCP discovery card at /.well-known/mcp.json, and agents.json. None are required, the report shows where you stand on each.

Firewalls and CDNs. Services like Cloudflare ship bot-protection settings that return 403 or 429 to AI crawler user-agents regardless of what robots.txt says. The checker catches this by comparing each bot’s response against a normal browser request: if the browser succeeds and the bot fails, the block lives in your WAF or CDN configuration, not in robots.txt. The report fingerprints the exact vendor doing the blocking, Cloudflare, Akamai, DataDome, Imperva, HUMAN (PerimeterX), AWS WAF, Sucuri and others, and generates a paste-ready allowlist rule for that specific vendor.

Yes, use the free Bot IP Verifier at /verify-bot. User-agent strings are trivially faked (studies find 12-18% of "Googlebot" traffic is spoofed), so the verifier checks a visitor IP against the bot operator's published IP range feeds (OpenAI, Google, Microsoft, Perplexity) or documented reverse-DNS hostnames with forward confirmation (Apple, Amazon, DuckDuckGo, Huawei). You get a clear verdict: genuine, spoofed, or, for operators like Anthropic that publish no verification data, honestly unverifiable.

Yes. Enter a URL and you get the full report: access score, the complete bot matrix, per-bot failure diagnosis, a copy-paste robots.txt fix, and a prioritized action plan you can have emailed to you or share with your team via a permanent report link. For continuous monitoring of which AI bots actually visit your site, check out Sona Agent Analytics.

Agent Analytics is part of the paid Sona platform. This free tool tells you which AI bots are allowed to reach your site; Agent Analytics tells you which ones actually do, and what that traffic is worth. It logs every AI crawler visit by bot, page and hour with verified bot signatures that separate real crawlers from spoofed user-agents, then connects that activity to outcomes: how many human visitors each AI engine refers back after crawling you, and which of those referrals turn into pipeline and closed-won revenue. That lets you prioritize the pages AI reads most by the revenue they drive. It is the difference between permission and reality.