NewAI Optimization — prioritized fixes for AI search visibilityLearn more
Free Tool

AI Crawler Diagnostic Tool — Is Your Site Accessible to AI?

This tool checks whether AI crawlers like GPTBot and ClaudeBot can access your site. If they're blocked, AI engines can't see or cite you.
Free, no signup ✦ Usually ready in seconds

How it works

  1. Enter your site

    We fetch your robots.txt and check response headers for Cloudflare WAF signals that often block AI bots by default.

  2. Check ChatGPT, Claude, Perplexity & more

    ChatGPT, Google, Perplexity, Claude, Microsoft Copilot, Meta AI, and Apple Intelligence — each gets allow, block, or could-not-verify from robots.txt.

  3. Get a verdict

    See how many crawlers are blocked and what that means for AI visibility — then fix robots.txt if needed.

Example result

What a mixed crawl-access report looks like — not a live scan of your site.
    • ChatGPT can reach your site
    • Allowed in robots.txt
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction
    • Google can reach your site
    • Allowed in robots.txt
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction

    AI answers and Gemini share this one access point.

    • Perplexity can't reach your site
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction
    • Blocked in your robots.txt
    • Claude can reach your site
    • Allowed in robots.txt
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction
    • Copilot can't reach your site
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction
    • Blocked in your robots.txt
    • Meta AI can reach your site
    • Allowed in robots.txt
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction
    • Apple can reach your site
    • Allowed in robots.txt
    • No Cloudflare WAF challenge detected
    • No X-Robots-Tag or meta robots restriction

Why it matters

A site can rank well on Google and still be invisible to ChatGPT, Claude, or Perplexity — if AI crawlers are blocked in robots.txt or by a Cloudflare WAF default.

Blocking AI bots is one of the most common, quiet reasons a brand never shows up in AI answers. Checking access is the fastest way to know whether the door is open.

How to fix

If crawlers are blocked, update your robots.txt: add an explicit Allow: / under each bot you want in, or remove overly broad Disallow rules — then re-check here. Or use the free robots.txt Generator to produce a ready file.

Where the file lives depends on your builder. WordPress: your SEO plugin — Yoast under Tools → File editor, Rank Math under General Settings → Edit robots.txt. Shopify: Online Store → Themes → Edit code → robots.txt.liquid. Wix: Marketing & SEO → SEO Tools → Robots.txt editor. Squarespace: Settings → Crawlers. Webflow: Site settings → SEO → Indexing. On a self-hosted site the file sits in the web root, next to your index file.

FAQ

Last updated

Related tools

See all
More free tools for AI visibility.

AI Schema Markup Generator

Structured data · JSON-LD

Generate valid JSON-LD structured data so AI and search engines understand your business.

AI FAQ Generator

Content · FAQPage

Generate FAQ Q&A plus FAQPage JSON-LD markup ready to paste on your site.

llm.txt Generator

llms.txt · Guidance

Generate an llms.txt file that helps AI tools understand what matters on your site.

llm.txt Checker

llms.txt · Audit

Check whether your site has a valid llm.txt / llms.txt file and what’s inside.

AI Prompt Research

Prompts · Demand

Discover real prompts people ask AI about your niche — and where you could show up.

AI Crawler Diagnostic Tool — Check GPTBot & ClaudeBot Access | Beamtrace — Beamtrace