Last Updated: March 22, 2026
- llms.txt is still an experimental, niche idea, and major AI providers do not rely on it for crawling, training, or ranking content.
- Yoast SEO keeps llms.txt support in its plugin, but it functions more as a low‑stakes experiment than a core SEO tactic.
- Control over AI crawlers today comes mainly from robots.txt, AI‑specific user agents, meta directives like noai, and paywalls or contracts, not llms.txt.
- If you have limited time or resources, you get far more impact from strong content, structured data, and clear AI access rules than from maintaining llms.txt.
llms.txt started as a simple idea: give AI crawlers a clean list of your best content, in a machine friendly format, so they can use it in their answers without wading through your whole site.
Two years later, it sits in a strange spot: still talked about, supported by tools like Yoast, but mostly ignored by the big AI players that shape traffic and visibility.
Updated context: where llms.txt actually stands now
When llms.txt first appeared, people treated it like the next robots.txt for AI, a way to future proof a site against changing search behavior.
That never really happened, at least not at scale.
Today, llms.txt is still a community idea, not an official standard from OpenAI, Google, Microsoft, or Anthropic.
There are GitHub repos and blog posts that describe suggested formats, but nothing like a formal spec that major providers commit to following.
llms.txt is best seen as an experiment for curating content to AI crawlers, not a control mechanism that companies are contractually or technically bound to respect.
That matters for your strategy.
If you treat it as a ranking lever or a protection layer, you will be disappointed pretty fast.
What actually happened to the llms.txt idea
The pattern is familiar: someone proposes a simple file, people write early blog posts, a few tools add support, and then the market either adopts it or quietly moves on.
With llms.txt, the market mostly moved on.
Here is the reality right now:
- No major AI provider treats llms.txt as a primary signal for crawling or training.
- No big search engine or LLM vendor has shipped official documentation that requires or heavily recommends it.
- Most AI access and training questions are handled through robots.txt, AI‑specific user agents, meta tags, and legal agreements.
Is llms.txt dead? I would not go that far.
Some AI search startups and a handful of tools still experiment with it, but it is clearly not the center of the AI + SEO story you should care about right now.
How llms.txt works in practice (and what has changed)
The basic idea has not changed much: llms.txt sits in your site’s root folder and lists URLs that you consider high value for AI systems.
Those URLs usually point to cleaner, lighter versions of your content, often in markdown or other text‑heavy formats.
The old assumption was that LLM crawlers would read llms.txt, follow those links, and use that text to train models or power AI search answers.
That assumption never turned into a mainstream reality.
Do links have to be markdown only?
Early articles insisted that llms.txt should only point to .md files, so that AI systems could avoid HTML clutter.
In practice, people quickly started mixing formats.
Right now, the pattern you see in the wild looks more like this:
- Markdown for docs, guides, and technical content.
- Plain HTML URLs for articles that are already fairly clean.
- Sometimes JSON or JSON‑LD for structured product or knowledge data.
Because there is no official spec, there is also no strict validator.
You are not blocked from listing HTML pages or even API endpoints if that makes sense for your setup.
A more realistic llms.txt example
If you want to experiment, a modern file might look closer to this:
/llms.txt # Blog guides (HTML) https://yourdomain.com/blog/seo-content-strategy/ https://yourdomain.com/blog/ai-search-guide/ # Docs (Markdown) https://yourdomain.com/docs/getting-started.md https://yourdomain.com/docs/api-reference.md # Optional metadata lines (experimental, not standard) https://yourdomain.com/docs/pricing-faq.md priority=0.8 lang=en lastmod=2026-02-10
Some people add loose metadata like priority or language, hoping AI systems will parse it.
No major provider has promised to respect these hints, so treat them as informal, not dependable.
Folder structure and serving markdown
You do not need a special server rule to serve .md files, but clean paths help.
A simple structure could look like this:
/var/www/yourdomain/ ├── public_html/ │ ├── index.html │ ├── llms.txt │ ├── blog/ │ └── docs/ │ ├── getting-started.md │ └── api-reference.md
If your server is picky about .md, you can map it to text/plain or text/markdown.
For Nginx, something like this is usually enough:
location ~* .md$ {
add_header Content-Type text/markdown;
}
And for Apache, you can add a simple directive in your config or .htaccess:
AddType text/markdown .md
Again, this is not mandatory, but it keeps things clean and predictable for any tool that chooses to read those files.
How Yoast SEO actually handles llms.txt now
Yoast added llms.txt support in the 21.x range of the plugin and has kept it in both the free and premium versions.
They frame it as part of preparing for AI search, not as a ranking feature.
Current Yoast behavior looks roughly like this:
- It generates llms.txt automatically and updates it on a schedule.
- It pulls from your most important posts and pages, with a bias toward cornerstone and recently updated content.
- It does not auto‑generate separate markdown files for you; it usually points to your existing URLs.
There are some practical limits that matter.
For example, multilingual setups can get messy because Yoast has to choose which language versions to surface, and there is no shared convention for marking language in llms.txt.
You should treat Yoast’s llms.txt feature as a convenience toggle, not as a main feature that will move your rankings or traffic by itself.
That applies even more if you are on a small site or a site that does not already have strong organic reach.
Do major AI providers actually care about llms.txt?
This is where the original hype and the current reality feel far apart.
Most of the large players still route decisions through robots.txt, user‑agent rules, and their own internal content pipelines, not llms.txt.
Current behavior of big AI bots
Here is a simplified look at what different providers do right now.
| Provider / Bot | Primary control signals | llms.txt support |
|---|---|---|
| OpenAI (GPTBot, oai-search) | robots.txt user agent rules, meta robots, noai / noimageai | No official support; treats it as optional content at best |
| Google (Googlebot, Google-Extended, AI Overviews) | robots.txt, meta robots, site structure, structured data, licensing deals | No formal use; no public doc that references llms.txt |
| Microsoft (Bingbot, Bing AI, Copilot) | robots.txt, meta tags, content deals for training corpora | No public statement of support |
| Anthropic (Claude, ClaudeBot) | robots.txt, meta directives, opt‑out policies | No mention of llms.txt in official docs |
| Perplexity, You.com, other AI search tools | robots.txt, user agents, some opt‑out headers and meta tags | Occasional experiments, but no shared, stable standard |
In other words, nobody that controls serious traffic says: “If you want us to see your content, you must use llms.txt.”
Some smaller players might read it, but they often treat it as a best‑effort hint, not a contract.
Google’s position compared to the early days
The old version of this article speculated about what Google might do.
We do not have to guess anymore.
Google’s current behavior is pretty clear:
- They rely on the long‑standing robots.txt standard for crawl access.
- They use Google‑Extended and site‑level preferences to control some AI training uses.
- They continue to lean on structured data, content quality, and E‑E‑A‑T for rankings and AI Overviews.
There is no official documentation from Google that elevates llms.txt, or anything like it, to a first class signal.
So if you are hoping that Yoast’s llms.txt toggle will nudge Google AI Overviews in your favor, that is not how things work today.
Where llms.txt fits into legal and consent questions
The last few years brought lawsuits, public debates, and changing policies around AI training data.
That shifted attention to consent and opt‑out signals, not to extra recommendation files.
Right now, consent and control for AI training usually depend on:
- robots.txt rules for AI‑specific user agents, like GPTBot or ClaudeBot.
- Meta tags like noai and noimageai, where respected.
- Paywalls and access restrictions that keep content out of public scrapes.
- Direct licensing deals or platform agreements between publishers and AI vendors.
llms.txt is not widely treated as a consent or contract file; vendors look at your robots.txt and headers long before they care about a list of “preferred” URLs.
This is one of the reasons I would not put sensitive, proprietary, or premium full text into clean markdown and list it prominently in llms.txt.
You might only make it easier for less reputable crawlers to copy it.
Which content should never go into llms.txt
I think this is where many early guides were too casual.
If you remove the hype and just look at risk, some content should stay out.
- Paid course material and gated ebooks.
- Detailed internal documentation that is not meant for public consumption.
- High value research reports you sell or gate tightly.
- Any content you already worry about being scraped or cloned.
You want AI systems to see your public, marketing, and support content in a controlled way.
You do not want to hand them a perfect, machine ready dump of material that underpins your business model.
robots.txt, meta tags, and llms.txt: how they actually fit together
The original comparison to robots.txt was a bit too neat, like these files were cousins.
In reality, robots.txt still sets the rules, and llms.txt, at best, offers some optional hints.
| Feature | robots.txt | Meta tags / headers | llms.txt |
|---|---|---|---|
| Main role | Allow or block crawling | Control indexing, snippets, AI reuse on specific pages | Highlight content you want AI tools to notice |
| Scope | Site or folder level | Page level | List of URLs |
| Support | Universal among search engines | Widely used for SEO, partly used for AI | Experimental, niche |
| Enforceability | Strong social and technical norms | Respected by major crawlers | No formal guarantee |
Modern robots.txt patterns for AI crawlers
A big change since early discussions is how many AI‑specific user agents now exist.
You can see them in real robots.txt files on large sites.
For example, a site that wants to block some AI crawlers but still allow basic search might use something like:
User-agent: * Disallow: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Googlebot Allow: / User-agent: Google-Extended Disallow: /
This is where your real leverage sits.
If you care about how AI tools touch your content, you start here, not with llms.txt.
Meta directives and AI specific tags
Page level tags give you a finer brush.
Some sites use meta robots and emerging AI tags like this:
<meta name="robots" content="index,follow"> <meta name="ai" content="noai"> <meta name="ai-image" content="noimageai">
Support is mixed, and some providers map these to their own internal rules.
Still, if your goal is control, these directives get more traction than llms.txt.
Think of llms.txt as optional “nice to have” hints, only after you have your robots.txt and meta directives in good shape.
A simple combined setup that makes sense
Here is one way to tie everything together without overcomplicating things.
- Use robots.txt to block AI crawlers you do not trust and allow ones you accept.
- Use meta robots and noai tags on pages that need special handling.
- Expose a small, curated list of evergreen guides in llms.txt, pointing to content you are happy to be used in AI answers.
For example, your root might contain:
- /robots.txt controlling basic crawl and AI access.
- /llms.txt listing 20 to 50 key URLs.
Nothing fancy, no full content dumps, just a hint for any tool that chooses to care.
Impact on AI Overviews and chat style search
You might wonder if llms.txt changes how often you appear in AI Overviews or chat answers.
Right now, the honest answer is: not in any measurable way that most sites can see.
AI Overviews and chat engines lean on:
- Existing search indexes and classic ranking signals.
- Structured data like FAQ, HowTo, and Product schema.
- Content clarity, freshness, and author signals.
llms.txt rarely shows up in public case studies from big brands.
When companies talk about AI visibility wins, they focus on schema, strong answers, and technical hygiene, not special text files for LLMs.
Should you use llms.txt at all?
The real question now is not “what if this becomes huge?” but “does this still earn a spot in my 2026 SEO and AI roadmap?”
For most sites, the answer is: maybe, but only as a light, low priority experiment.
Where llms.txt can make some sense
There are a few situations where I think llms.txt is reasonable.
Not mandatory, but reasonable.
Large documentation or knowledge base sites
If you run a big docs site for a SaaS product or API, you often have hundreds of pages, nested navigation, and many versions.
Listing your core getting started, integration, and troubleshooting guides in llms.txt creates a simple, flat map any crawler can read.
This works best when:
- Your docs are already public and you are fine with AI tools using them.
- You maintain a small list, not your entire library.
- You pair it with clear robots.txt and meta rules.
Support heavy sites with FAQs and how to content
Think of sites with a lot of help content: hosting companies, ecommerce platforms, or big SaaS tools.
A compact llms.txt that points to your clearest FAQs and troubleshooting steps might help some AI search tools give better answers.
Will you see a traffic spike from that alone?
Unlikely, but it can keep your setup tidy while the ecosystem slowly matures.
AI‑friendly publishers that want to experiment
Some publishers actually like being used heavily by AI tools, because it feeds brand reach and partnership deals.
If you are in that camp, llms.txt is a harmless way to say “start here” without changing your main templates.
If you already manage complex SEO setups and enjoy controlled experiments, adding llms.txt through Yoast is a tiny extra step with low risk.
Where llms.txt is usually not worth the trouble
There are also plenty of cases where I think you should skip it.
Not forever, but for now.
- Small local sites that mainly rely on Google Maps and branded searches.
- Service businesses with a tight set of core pages and a small blog.
- Any site where dev or content resources are already stretched thin.
In those cases, your energy is better spent on:
- Improving content quality and search intent match.
- Fixing Core Web Vitals and crawl issues.
- Adding or improving structured data.
- Clarifying AI access rules in robots.txt.
A quick decision checklist
Here is a simple way to decide, without overthinking it.
- Do you already have solid robots.txt, meta robots, and schema in place?
- Do you have at least 20 to 50 truly strong, evergreen pages worth curating?
- Are you comfortable with AI tools reading and reusing those pages?
- Can you maintain a short list without it turning into a chore?
If you answer yes to most of these, flip on the Yoast llms.txt feature or create a simple manual file.
If not, skip it for now and come back later, or skip it entirely.
Real pros and cons, not hypothetical ones
We can move away from theory here and look at what site owners have seen so far.
| Pros | Cons |
|---|---|
| Very low effort if generated by a plugin like Yoast | Ignored by most major AI providers as a primary signal |
| Creates a clear list of high value URLs that is easy for any crawler to read | Can expose clean content to scrapers if used carelessly |
| Helpful for your own housekeeping and content audits | No clear, consistent evidence of traffic or ranking gains |
| Simple way to test how some AI tools handle curated lists | Another moving part to maintain if done manually |
Notice what is missing from the pros: there is no reliable, repeatable story of “we added llms.txt and our AI visibility exploded.”
If someone promises that, I would want to see real data, not just a chart from one week.
Spam, abuse, and lessons from the last two years
The earlier version of this post guessed that people might abuse llms.txt with thin or misleading content.
That did happen in small pockets, but it did not reshape anything.
Patterns that showed up:
- Auto generated “AI only” summaries that did not match the real pages.
- Keyword stuffed markdown files meant only for LLMs.
- Sites that listed junk or doorway style pages in llms.txt hoping to influence AI search.
AI tools adapted the way search engines always do: they dial down trust in obviously spammy sources, or just ignore low quality signals.
That is one more reason to keep llms.txt consistent with what your users see on the canonical URLs.
How to monitor AI crawlers and llms.txt activity
One frustration with AI traffic is how invisible it can feel.
Still, you can pick up some clues if you know where to look.
Common AI user agents to watch for
User agents change, but a few patterns show up often.
You might see names like these in your logs:
- GPTBot
- ClaudeBot
- Google-Extended
- PerplexityBot
- CCBot or other generic crawlers used in AI datasets
You should always confirm current user agent strings from each vendor’s official pages, because copycats exist.
Never assume every “AI” or “GPT” string is legitimate.
Simple log checks for llms.txt and markdown hits
If you have shell access, you can start with basic grep queries.
For Apache or Nginx access logs, something like this is common:
# Requests to llms.txt grep "llms.txt" access.log # Requests to markdown files grep ".md" access.log # Requests from GPTBot grep "GPTBot" access.log
If you use logrotate, remember to check older files too.
Otherwise you might think nothing is happening when traffic is just buried in yesterday’s file.
In tools like Matomo or GA4, you can:
- Set up a segment or filter for URLs that contain “/docs” or “.md”.
- Track views of /llms.txt as a separate page or event.
You will not get a perfect map of AI training behavior from this, but you can at least see whether anyone is touching the URLs you list.
Where llms.txt sits in your 2026 SEO and AI plan
At this point, I would place llms.txt as a tier three tactic.
Useful to test, not worth serious investment, and never the first lever you pull.
If you have not already nailed your content quality, structured data, crawl control, and internal linking, you are working on the wrong things by obsessing over llms.txt.
A simple priority stack for most sites looks like this:
- Solid technical SEO: crawlability, performance, proper use of robots.txt.
- Search intent driven content that answers real questions clearly.
- Structured data for key content types like FAQs, products, and how‑tos.
- Clear AI access rules for the crawlers you care about.
- Experiments like llms.txt, custom feeds, or AI specific landing pages.
You can see where llms.txt lands there.
Useful once the foundation is solid, largely irrelevant before that.
My honest take on using Yoast’s llms.txt feature
I would keep this simple.
If Yoast is already part of your stack, turning on llms.txt is almost free from a time standpoint, and that is enough reason to try it.
Just do not build complex markdown pipelines or dedicate serious developer time around it until a major AI vendor clearly states they support it in production.
Watch your logs, watch your analytics, and be ready to delete or adjust your llms.txt file if it becomes noisy or risky.
If you are not using Yoast, I would only hand‑build llms.txt if you genuinely enjoy experiments and have a clear, limited use case like a docs hub.
Otherwise, you are better off investing that energy into content and technical fixes that you already know search engines and users value.
AI search and LLM behavior will keep evolving.
Keep llms.txt in the back of your mind, but keep your main focus on the parts of SEO and content that have proven themselves over time.
Need a quick summary of this article? Choose your favorite AI tool below:
