llms.txt: Does It Actually Improve Visibility?

Over the past year, llms.txt has been promoted as a simple way to boost AI visibility: publish a small text file at your site's root, and AI models will supposedly discover, understand, and cite your content more easily. The evidence, however, is far less clear. Google says it ignores the file for Search, while no major AI platform has confirmed using it to decide what to cite or recommend.

This guide explains what llms.txt is, how it works, who (if anyone) uses it, whether it affects SEO or AI visibility, and whether it's worth adding to your website.

What is llms.txt & what is it used for

llms.txt is a Markdown file that gives language models a curated, plain-text overview of a site's key content, a proposed web standard introduced by Jeremy Howard, co-founder of Answer.AI and fast.ai.

Is it needed for AI crawlers?

Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety. Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.

Howard framed the file for use at inference time — the moment a model is answering a question and needs clean context — and explicitly not as a training opt-out or a ranking signal.

llms.txt is often referred to as "robots.txt for AI," but that framing is inaccurate. robots.txt controls crawler access by allowing or disallowing bots. llms.txt controls nothing. It simply suggests what a model might want to read.

The spec positions the file to coexist with the standards you already have: sitemaps list every page for search engines, robots.txt governs access, and llms.txt offers a curated overview for LLMs. It's worth remembering that this is a community proposal with no backing from the W3C, IETF, or ISO — it's a convention, not a ratified standard.

llms.txt format

The spec defines a precise Markdown structure in a fixed order – it’s deliberately simple:

• An H1 with the project or site name as the only required element.
• An optional blockquote with a short summary of the site.
• Zero or more Markdown sections (without headings) for additional detail.
• Zero or more H2-delimited link lists, each entry formatted as a link with an optional note.

One H2 has a special meaning: a section titled "Optional" flags links that a model can skip when it needs a shorter context. Below is a stripped-down llms.txt file example:

# Acme Analytics
> Acme Analytics is a web analytics platform for small e-commerce stores.
## Docs
[Getting started](https://acme.example/docs/start): Setup and first report
[API reference](https://acme.example/docs/api): Endpoints and authentication
## Optional
[Changelog](https://acme.example/changelog): Release history

llms.txt types

There are two files in practice, and they serve different jobs. The distinction matters when you decide what to publish, so here's how they compare alongside the related .md proposal:

If you're wondering how to see the llms.txt file of a website, it's the same trick you'd use to check any root-level file: append /llms.txt to the domain and load it in your browser. That's also the fastest way to see how other companies structure theirs.

Who actually reads it

As of mid-2026, no major AI platform — OpenAI, Google, Anthropic, Perplexity, or Microsoft — has publicly confirmed that it reads or uses llms.txt for training, retrieval, or ranking. Their crawler documentation consistently points site owners to robots.txt for controlling bots like GPTBot and never mentions llms.txt as an input.

Nobody reads it, but…

There's a tempting counterargument: several of those same companies publish their own llms.txt files. But they publish them only on their developer documentation sites, for consumption by coding assistants — not as evidence that their consumer products read anyone else's files.

Server-log and citation studies back up the "nobody's reading it" picture with hard numbers:

Do AI bots use the file at all?

John Mueller, a Google Search Advocate, summed up the skeptical view in a widely shared comment, comparing the file to a signal search engines abandoned long ago:

AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag – this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)

A few trackers *have *reported occasional AI-bot fetches of the file, and Google added an llms.txt check to Chrome's Lighthouse "Agentic Browsing" audit — which is the source of that self-contradiction we opened with. However, an occasional fetch and an agent-readiness audit are not the same as a consumer AI platform using the file to decide what to cite or recommend.

Does it help SEO

The cleanest way to answer whether an llms.txt file will help your SEO is to split the question into three, because they get tangled together constantly and the answers aren't identical.

The three aspects

1. Traditional Google ranking. No effect. It isn't a ranking signal, and Google's documentation says the file is ignored.
2. AI Overviews and AI-search visibility. No demonstrated effect. Google says its generative features run on the same core ranking systems as regular Search and need only normal SEO.
3. LLM training and retrieval. No confirmed consumption by any major model.

Google has put the ranking question to rest in its own words. Its guidance on optimizing for generative AI features states that you don't need new machine-readable files to appear in Search, "as Google Search itself doesn't use them," adding that maintaining an llms.txt file "won't harm (nor help) your visibility or rankings."

Independent testing

A 300,000-domain analysis found no correlation between having the file and how often a domain was cited by LLMs – removing the variable from its model actually improved accuracy.

A separate before-and-after test of ten sites showed eight sites with no change, two that grew for reasons traceable to other work, such as PR and new content, rather than the file, and one that declined.

There is a single self-reported case in which a site's llms.txt was cited by Google's AI Mode shortly after being submitted through Search Console — but it's a sample of one, not reproducible proof.

Should you create one

None of the above makes llms.txt useless — it makes it specific. The file has a real, non-hype audience, and it's worth being honest about who's in it and who isn't.

The target audience

It's genuinely useful if you publish developer documentation, API references, or a product aimed at technical and agentic users.

AI coding assistants like Cursor, Claude Code, GitHub Copilot, and Windsurf, along with RAG pipelines that read docs, do consume these files. That's not a coincidence: nearly every high-profile adopter is a developer-tool or docs company, and adoption itself surged after Mintlify began auto-generating the file for hosted docs, not because any AI search engine started demanding it.

llms.txt file for AI optimization

It's not a shortcut if your goal is consumer AI-search visibility. There's no evidence it moves citations or rankings, and the time is better spent on the things that demonstrably do: strong content, topical authority, and clean structured data.

If that's your aim, our guidance on getting cited by AI and on broader AI visibility fundamentals is a better starting point.

How to create llms.txt

If you do want to create one, it's simple. You can write it by hand, auto-generate it through docs platforms like Mintlify, Fern, or GitBook, build one with Firecrawl's generator, or use a WordPress option — the standalone Website LLMs.txt plugin, or native support in Yoast SEO and Rank Math.

A few best practices keep it from backfiring: keep the file fresh, since a stale llms.txt is worse than none; noindex any auto-generated .md page copies to avoid duplicate-content issues; and think twice before publishing a full llms-full.txt, which hands competitors and scrapers a clean, complete copy of your best content in one download.

If the real question underneath is whether AI systems are mentioning your brand, that's something you can measure directly rather than infer from an unproven file. Beamtrace tracks how your brand appears in AI answers on ChatGPT today, with Perplexity, Gemini, Claude, and other platforms coming soon — which turns "I hope this helps" into an actual before-and-after signal.

Conclusion

llms.txt is a well-intentioned proposal that solves a real technical problem for a specific audience — and it got repackaged into an AI-visibility lever it was never designed to be. Built to give models clean context at inference time, it’s used by coding assistants and RAG pipelines that read developer docs, not by consumer chatbots.

The evidence, taken together, is consistent: no major platform confirms its use, and no study has found it affecting citations or rankings. For a developer-docs or API audience, the file is a cheap and legitimate addition. For everyone chasing visibility in AI answers, the honest levers haven't changed — quality content, real authority, sound, well-structured data, and knowing whether AI is actually mentioning you rather than just guessing.

Frequently asked questions

Should I create an llms.txt file?

If you run developer docs, an API reference, or a product used by coding assistants and agents, yes — it's cheap, and those tools read it. If your goal is ranking or being cited in consumer AI answers, there's no evidence it helps, so it's optional at best.

Will an llms.txt file help my SEO?

No. Google has said it ignores the file, and it neither helps nor harms your rankings. Independent studies found no link between having the file and how often a site is cited by AI.

How do I see the llms.txt file of a website?

Add /llms.txt to the end of the domain — for example, example.com/llms.txt — and open it in your browser. If the site has one, it loads as plain text.

Can llms.txt get my content indexed or cited by LLMs?

There's no confirmed mechanism for this. No major AI platform has said it uses the file to index, retrieve, or cite content, and log studies show AI bots rarely request it.

How is llms.txt different from robots.txt?

robots.txt controls access — it tells crawlers what they may or may not fetch. llms.txt controls nothing; it only suggests which pages a model might find most useful. They solve different problems and can coexist.

Key references

• Jeremy Howard, Answer.AI, /llms.txt specification – https://llmstxt.org/
• Google Search Central – https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
• SE Ranking – https://seranking.com/blog/llms-txt/
• Ahrefs – https://ahrefs.com/blog/what-is-llms-txt/
• Search Engine Land – https://searchengineland.com/does-llms-txt-matter-467740