Back to journal
AI Visibility

What Is llms.txt? Does Your Website Need One?

llms.txt is a proposed plain-text index file for AI tools. For most business websites it is optional, and the evidence says it will not improve AI search visibility. Here is what it is, what Google has said, what the data shows, and where your time is better spent.

UK
Umair Khan
Head of AI Visibility
11 min read
November 18, 2026

Over the past year, we've watched "add an llms.txt file" climb onto nearly every AI SEO checklist. Clients ask us about it, and so do other business owners who've been told it's the missing piece of their AI strategy.

So here is our straight answer. llms.txt is a proposed plain-text index file for AI tools. For most business websites, it is optional, and the evidence says it will not improve your AI search visibility. Below, we explain what it is, what Google has said, what the largest data we could find shows, and where we think your time is better spent.

Humanoid robot standing at a store counter, looking over the products on display
llms.txt was proposed as a curated index to help AI tools find the useful parts of a website.

What is llms.txt?

llms.txt is a Markdown file placed at the root of a website (yourdomain.com/llms.txt) that summarizes the site and links to its most important pages, so AI tools can orient themselves without crawling everything.

Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed it in 2024 at llmstxt.org. The idea was practical: language models have limited context windows, and websites are full of navigation, scripts, and clutter. A clean, curated index could help an AI tool find the useful parts quickly.

A simple example, using a made-up business:

# Example Plumbing Co.

> Family-owned plumbing company serving Annapolis, MD. Water heaters, drain cleaning, and emergency repairs.

## Services
- [Water heater replacement](https://example.com/water-heaters): Costs, timelines, and what's included
- [Drain cleaning](https://example.com/drain-cleaning): Methods and when each applies

## About
- [Our team and service area](https://example.com/about): Licensing, neighborhoods served

Two things it is not:

  • It is not robots.txt. It doesn't allow or block anything. It controls nothing.
  • It is not a sitemap. A sitemap lists URLs for search engine crawlers. An llms.txt file is a curated, human-readable summary written for language models.

Some sites also publish an llms-full.txt with the complete text of their documentation in one file. That is a separate, even less standardized practice.

What has Google said about llms.txt?

This is where most of the confusion comes from, so we'll be precise.

Google's guide to optimizing for generative AI features, published in May 2026, includes a "mythbusting" section on what site owners don't need to do. It says you don't need special machine-readable files, AI text files, or markup to appear in AI Overviews and AI Mode. In Google's framing, creating one neither helps nor hurts your Search visibility.

Then the picture got muddier. According to Ahrefs' June 2026 study, the Chrome team shipped an llms.txt check inside Lighthouse's experimental agentic browsing audits just days after Google's guide. Asked about the apparent contradiction, Google's John Mueller explained that the file isn't meant for search. He described it as a "temporary crutch" for AI coding tools reading developer documentation.

Our reading is that Google Search doesn't use the file, and a different team at Google is experimenting with how AI agents browse websites. Those are separate questions, and only the first one affects your rankings today.

What does the data show?

Opinions are cheap on this topic, so we looked for measurements.

Ahrefs (June 15, 2026). Ahrefs analyzed server logs and live traffic for 137,210 domains that received traffic in May 2026. The headline findings:

  • About 28% of those domains published an llms.txt file. Ahrefs notes its customers skew technical, so that is likely an upper bound for the wider web.
  • 97% of those files received zero requests that month. Nothing fetched them at all.
  • Of the requests that did arrive, 96% came from bots, and 77% of the bots weren't AI tools. Many were SEO audit tools, tech profilers, and scanners built to check for llms.txt.
  • AI retrieval bots, the kind that fetch pages to answer live questions in tools like Perplexity and ChatGPT, accounted for just 1.1% of requests.
  • No AI bot requested llms.txt files that didn't exist. Sites without one weren't missing a knock at the door.
  • The most active identifiable AI visitors were training crawlers and agentic tools, including coding agents, rather than AI search bots.

Ahrefs sells SEO software and says its sample is limited, and it stresses that a fetch isn't proof a bot read or used the file. Its numbers are a ceiling on actual use. It measured requests, not whether the file improved anyone's citations.

SE Ranking (November 2025). Coverage of an SE Ranking study of roughly 300,000 domains, summarized by 365i, found no measurable relationship between having an llms.txt file and how often a site was cited by AI systems. Adoption at that point was around 10%.

The two studies measure different things. Both point the same way.

Does your website need an llms.txt file?

For most business websites, no. Here is how we'd think about it by site type.

Your siteOur view
Local service business (plumber, roofer, dentist)Skip it. Your effort is better spent on your Google Business Profile, reviews, and service pages.
Professional services or B2B companyOptional. It won't hurt, but it's unlikely to move results.
E-commerce storeLow priority. Product feeds, structured product data, and reviews matter far more.
Developer documentation, APIs, or software toolsWorth considering. This is the audience the file was designed for, and the strongest use case in the data.
Site where AI agents act on behalf of customersWorth a small experiment, treated as speculative.

The common thread: if your customers find you by searching, asking Google, or asking an AI assistant for a local recommendation, llms.txt isn't what decides whether you're named.

What is the case for and against publishing one?

Reasons some sites publish it

  • It is cheap and quick to create, and some website platforms generate it automatically.
  • Developer-documentation sites have a real audience in coding agents.
  • If AI agents do come to mediate more search in the future, an index file could become more useful. That's speculation, and we wouldn't plan around it.

Reasons we're cautious

  • The base rate is poor. Nearly all published files go unread.
  • No major AI platform has committed to using it for search visibility.
  • It's another file to keep accurate. A stale index that points AI tools at outdated pages works against you.
  • Security researchers are already probing llms.txt as a possible route for prompt injection, since agents may trust the file's contents. Ahrefs flags this too. Treat the file like code if you publish one.

How should you create an llms.txt file if you decide to?

If you've weighed the above and still want a file, here's our practical approach:

  • Write it by hand or review any automated version. Include a clear title, a one- or two-sentence description of the business, and links to your most important real pages with brief, factual descriptions.
  • Stick to plain facts and links. Avoid anything that reads like instructions to a machine.
  • Only link to pages you control, and keep them live.
  • Restrict who can edit it, and review it when your site changes. Version control helps.
  • Place it at the root: yourdomain.com/llms.txt.
  • Check your logs after a few months. If nothing ever requests it, you've learned something useful, and nothing is lost.
  • Don't expect rankings or citations from it.

Note - Our recommendation is to treat it as a low-priority experiment, never a strategy.

What should you focus on instead?

If AI search visibility is the goal, the evidence points toward fundamentals. At Rizing Metrics, we put time in this order:

  • Make sure crawlers can reach your pages. Check robots.txt, your CDN, and any firewall or bot-protection settings. Nothing else matters if Googlebot and the AI search crawlers are blocked. Our technical SEO reviews start here.
  • Publish content only your business can write. Google's guide stresses unique, non-commodity content: real experience, original data, specific examples. Our SEO content services are built around this.
  • Keep your business information accurate and consistent. Name, address, phone, hours, and services should match across your site, Google Business Profile, and major directories. For local companies, our local SEO work focuses on this.
  • Earn reviews and credible mentions. Third-party confirmation of who you are carries weight with both search engines and AI tools. Our link building and digital PR approach is built on earning these honestly.
  • Answer real questions clearly. Lead with the answer, then add detail. This is the practical heart of answer engine optimization.
  • Measure what you can. Track Search Console data, referral traffic, branded search, and leads. We cover this in our guide to Google AI Overviews optimization. Run manual prompt tests in the AI tools your customers use, and treat any third-party AI visibility score as an estimate.

How does llms.txt compare with robots.txt and sitemaps?

FilePurposeWho reads itDo you need it?
robots.txtTells crawlers which parts of your site they may accessSearch engines and most reputable botsYes, for nearly every site
XML sitemapLists URLs you want search engines to findSearch enginesYes, for most sites
llms.txtProposed curated index of your site for AI toolsMostly coding agents and tools, per current dataOptional

One more point on robots.txt, since it's what actually affects AI access. Perplexity's crawler documentation recommends allowing PerplexityBot if you want to appear in its search results, and notes that its Perplexity-User fetcher generally ignores robots.txt because a person triggered the request. Access settings change what AI tools can reach in a way llms.txt doesn't.

What are the common llms.txt mistakes to avoid?

  • Treating it as an AI ranking factor. Neither Google's documentation nor the large studies support this.
  • Paying for "llms.txt optimization" as a standalone service. Be wary of anyone selling it as a shortcut to AI citations.
  • Publishing an unreviewed auto-generated file. Check what it says and links to.
  • Letting it go stale. Outdated links and descriptions create inaccuracies.
  • Stuffing it with promotional claims or instruction-style text. It adds risk and no benefit.
  • Confusing it with robots.txt. It can't block or allow anything.
  • Neglecting the basics while fiddling with files. That's the costliest mistake of all.

How do we help businesses with AI search visibility?

At Rizing Metrics, we believe AI visibility comes from the same foundation as strong SEO: a site that can be crawled, content that shows real expertise, and business information people and machines can trust. Our approach focuses on fixing that foundation before experimenting with the newer layers.

If you're unsure whether your site is ready, a Website SEO Audit is where we'd start. It shows whether crawlers can reach your pages and what's holding back your visibility. From there, our AI Visibility Optimization and Organic SEO services help you build on it.

Want a clear read on where you stand? Try our free AI audit, or contact us to talk through a plan for your business.

Frequently asked questions

What is llms.txt?

llms.txt is a proposed Markdown file placed at a website's root that summarizes the site and links to its key pages for AI tools. Jeremy Howard proposed it in 2024. It controls nothing and blocks nothing.

Does llms.txt help SEO?

No evidence says it does. Google says its Search systems don't need such files and that adding one neither helps nor hurts. An SE Ranking study of about 300,000 domains found no measurable effect on AI citations.

Does llms.txt help me appear in ChatGPT, Perplexity, or Google AI Overviews?

There's no confirmation that it does. Ahrefs' 2026 analysis found 97% of files received zero requests, and AI retrieval bots made up only about 1% of requests. No AI bot requested files that didn't exist.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which areas they may access. llms.txt is an optional summary file for AI tools. It can't allow or block anything.

Do I need an llms.txt file?

Most business websites don't. It makes more sense for developer documentation, APIs, and software tools, where coding agents are a real audience.

Can an llms.txt file hurt my website?

It's unlikely to affect rankings, but it carries risks. A stale file spreads outdated information, and security researchers have probed these files for prompt injection. Keep it accurate and tightly controlled.

What should I put in an llms.txt file?

A clear title, a short factual description of your business, and links to your most important live pages with brief descriptions. Avoid promotional claims and instruction-like text.

What is llms-full.txt?

A companion file some sites use to put the full text of their documentation in one place. It's even less standardized than llms.txt, and the same cautions apply.

Do AI crawlers respect robots.txt?

Mostly, with exceptions. Perplexity says PerplexityBot follows it, while Perplexity-User generally ignores it for user-initiated fetches. Check each platform's documentation before assuming.

What should I do instead to improve AI search visibility?

Make sure crawlers can reach your site, publish original and useful content, keep business information consistent everywhere, earn reviews and credible mentions, and measure results. These are the same fundamentals that drive strong organic search visibility.

Can an SEO agency help with AI search visibility?

An agency can strengthen the foundations: technical health, content, local signals, and authority. No one can guarantee AI citations, and you should be skeptical of anyone who promises them.

Free 15-minute audit

Ready to fix your AI visibility?

Get a free AI Visibility Audit - we'll show you exactly where ChatGPT and Gemini cite you (or don't).

See how our AI visibility optimization service builds the foundations AI tools rely on