robots.txt is a plain-text file at the root of your domain (yourdomain.com/robots.txt) that tells crawlers which parts of the site they may request. Each block names a User-agent and then Allow or Disallow rules for it. It is a set of instructions, not a security control — well-behaved crawlers follow it, and search engines and AI assistants both read it before fetching your pages.

Free Tool

Free robots.txt Generator (with AI Crawler Support)

Generate a valid robots.txt file, including rules to allow or block AI crawlers like GPTBot and ClaudeBot.

You have a website

Analyze the live robots.txt and generate AI crawler rules on top of it.

You have no website

Build a complete robots.txt from scratch — then upload it when you launch.

Free, no signup ✦ Pulls your current robots.txt before generating changes

How it works

Start from your live site — or build manually
Enter your website and we fetch the current robots.txt when one exists, so you start from the real file. No site yet? Choose Build manually and generate a complete file to upload when you launch.
Pick an AI crawler strategy
Start with Allow All, switch to Allow Search, block every crawler, or fine-tune each supported bot in Custom.
Copy or download the result
Review the generated file — and the diff when you started from an existing robots.txt — then copy or download it.

Why it matters for AI crawlers

Default robots.txt files quietly cut off AI crawlers. A single broad User-agent: * with a Disallow rule, or a hosting default written before AI assistants existed, is enough to keep GPTBot, ClaudeBot and PerplexityBot out — while your Google rankings stay untouched and nothing looks broken.

Editing the file by hand is easy to get wrong, especially when you want AI-specific rules without touching your broader search setup. This generator layers explicit per-crawler groups on top of the file you already have, so a later change to the catch-all group can’t silently flip AI access back off.

How to use it: where the file goes

robots.txt only works from the root of the domain — https://yourdomain.com/robots.txt, never a subfolder. Most site builders manage the file for you rather than letting you upload it directly:

Self-hosted site
Drop robots.txt in the web root, next to your index file, so it answers at https://yourdomain.com/robots.txt.
WordPress
Edit it through your SEO plugin — Yoast under Tools → File editor, Rank Math under General Settings → Edit robots.txt.
Shopify
Online Store → Themes → Edit code, then edit the robots.txt.liquid template and add the AI crawler groups.
Wix, Squarespace and Webflow
Wix: Marketing & SEO → SEO Tools → Robots.txt editor. Squarespace: Settings → Crawlers. Webflow: Site settings → SEO → Indexing.

After you publish the updated file, run the free AI Crawler Diagnostic to confirm the bots you care about can actually reach it.

FAQ

What is robots.txt?

How to allow AI crawlers?

Add an explicit group for each AI crawler token you want in, with Allow: /. The official tokens are GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot, Claude-User and Claude-SearchBot (Anthropic), PerplexityBot and Perplexity-User (Perplexity), Google-Extended (Gemini and AI Overviews training use), bingbot (Copilot), meta-externalagent (Meta AI) and Applebot-Extended (Apple Intelligence). A specific group always wins over a broader User-agent: * group, so an explicit Allow keeps AI access stable even if the catch-all group changes later. This generator writes all of those groups for you.

robots.txt vs llms.txt?

robots.txt controls access — whether a crawler is allowed to fetch a URL at all. llms.txt is a newer, unofficial convention that does the opposite job: it points AI tools at the content on your site that matters most and summarises it for them. They do not replace each other. If AI crawlers are blocked in robots.txt, an llms.txt file changes nothing, because nothing can be fetched in the first place.

Where do I upload it?

robots.txt must sit at the root of the domain — https://yourdomain.com/robots.txt — not in a subfolder. On a self-hosted site, drop the file in the web root next to your index file. Most site builders manage it for you instead: WordPress via an SEO plugin such as Yoast or Rank Math, Shopify by editing the robots.txt.liquid template, Wix and Squarespace in their SEO or crawler settings, and Webflow in Site settings → SEO → Indexing. After publishing, open yourdomain.com/robots.txt in a browser to confirm the new file is live.

Last updated August 21, 2026

Related tools

See all

Verify the live result after you publish the updated file.

Free Tool

Free robots.txt Generator (with AI Crawler Support)

Generate a valid robots.txt file, including rules to allow or block AI crawlers like GPTBot and ClaudeBot.

You have a website

Analyze the live robots.txt and generate AI crawler rules on top of it.

You have no website

Build a complete robots.txt from scratch — then upload it when you launch.

Free, no signup ✦ Pulls your current robots.txt before generating changes

How it works

Start from your live site — or build manually
Enter your website and we fetch the current robots.txt when one exists, so you start from the real file. No site yet? Choose Build manually and generate a complete file to upload when you launch.
Pick an AI crawler strategy
Start with Allow All, switch to Allow Search, block every crawler, or fine-tune each supported bot in Custom.
Copy or download the result
Review the generated file — and the diff when you started from an existing robots.txt — then copy or download it.

Why it matters for AI crawlers

Default robots.txt files quietly cut off AI crawlers. A single broad User-agent: * with a Disallow rule, or a hosting default written before AI assistants existed, is enough to keep GPTBot, ClaudeBot and PerplexityBot out — while your Google rankings stay untouched and nothing looks broken.

Editing the file by hand is easy to get wrong, especially when you want AI-specific rules without touching your broader search setup. This generator layers explicit per-crawler groups on top of the file you already have, so a later change to the catch-all group can’t silently flip AI access back off.

How to use it: where the file goes

robots.txt only works from the root of the domain — https://yourdomain.com/robots.txt, never a subfolder. Most site builders manage the file for you rather than letting you upload it directly:

Self-hosted site
Drop robots.txt in the web root, next to your index file, so it answers at https://yourdomain.com/robots.txt.
WordPress
Edit it through your SEO plugin — Yoast under Tools → File editor, Rank Math under General Settings → Edit robots.txt.
Shopify
Online Store → Themes → Edit code, then edit the robots.txt.liquid template and add the AI crawler groups.
Wix, Squarespace and Webflow
Wix: Marketing & SEO → SEO Tools → Robots.txt editor. Squarespace: Settings → Crawlers. Webflow: Site settings → SEO → Indexing.

After you publish the updated file, run the free AI Crawler Diagnostic to confirm the bots you care about can actually reach it.

FAQ

What is robots.txt?

How to allow AI crawlers?

Add an explicit group for each AI crawler token you want in, with Allow: /. The official tokens are GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot, Claude-User and Claude-SearchBot (Anthropic), PerplexityBot and Perplexity-User (Perplexity), Google-Extended (Gemini and AI Overviews training use), bingbot (Copilot), meta-externalagent (Meta AI) and Applebot-Extended (Apple Intelligence). A specific group always wins over a broader User-agent: * group, so an explicit Allow keeps AI access stable even if the catch-all group changes later. This generator writes all of those groups for you.

robots.txt vs llms.txt?

robots.txt controls access — whether a crawler is allowed to fetch a URL at all. llms.txt is a newer, unofficial convention that does the opposite job: it points AI tools at the content on your site that matters most and summarises it for them. They do not replace each other. If AI crawlers are blocked in robots.txt, an llms.txt file changes nothing, because nothing can be fetched in the first place.

Where do I upload it?

robots.txt must sit at the root of the domain — https://yourdomain.com/robots.txt — not in a subfolder. On a self-hosted site, drop the file in the web root next to your index file. Most site builders manage it for you instead: WordPress via an SEO plugin such as Yoast or Rank Math, Shopify by editing the robots.txt.liquid template, Wix and Squarespace in their SEO or crawler settings, and Webflow in Site settings → SEO → Indexing. After publishing, open yourdomain.com/robots.txt in a browser to confirm the new file is live.

Last updated August 21, 2026

Related tools

See all

Verify the live result after you publish the updated file.

Free robots.txt Generator (with AI Crawler Support)

How it works

Why it matters for AI crawlers

How to use it: where the file goes

FAQ

What is robots.txt?

How to allow AI crawlers?

robots.txt vs llms.txt?

Where do I upload it?

Related tools

AI Crawler Diagnostic

Want the full AI visibility picture?

Free robots.txt Generator (with AI Crawler Support)

How it works

Why it matters for AI crawlers

How to use it: where the file goes

FAQ

What is robots.txt?

How to allow AI crawlers?

robots.txt vs llms.txt?

Where do I upload it?

Related tools

AI Crawler Diagnostic

Want the full AI visibility picture?

Free robots.txt Generator (with AI Crawler Support)

How it works

Why it matters for AI crawlers

How to use it: where the file goes

FAQ

What is robots.txt?

How to allow AI crawlers?

robots.txt vs llms.txt?

Where do I upload it?

Related tools

AI Crawler Diagnostic

Want the full AI visibility picture?