Tornado SEO

llms.txt

By Thibault · Updated

Definition

llms.txt is a Markdown file placed at the root of a website (/llms.txt) that gives AI models a short summary of the site and a curated list of links to its most useful pages.

Why does llms.txt matter?

llms.txt gives AI models a clean way into a website, with no menus, scripts or ads in the way. Jeremy Howard proposed it on September 3, 2024, to solve a simple problem. An AI model can only read a limited amount of text at once, and an HTML page full of navigation uses up that space. The file sends the model straight to the pages that answer questions best, in a format it reads without effort.

How does an llms.txt file work?

An llms.txt file is plain Markdown with its elements in a fixed order. The proposal published on llmstxt.org describes this structure:

  1. An H1 with the name of the site or project. This is the only required section.
  2. A blockquote with a short summary of the site.
  3. Optional paragraphs or lists with more detail, without headings.
  4. H2 sections that list links in the form [name](url), each one optionally followed by a colon and a note.
  5. A section titled "Optional" for secondary links that an AI can skip when it needs less context.

The proposal also recommends a clean Markdown version of each page at the same address, ending in .md (page.html.md or page.md). The file can sit at the root or in a subfolder, and it covers every URL under its path.

Does Google use llms.txt?

Google Search does not use llms.txt. Its guide to generative AI features says you don't need new machine-readable files or Markdown to appear in Google Search. It adds that an llms.txt file will neither harm nor help your visibility or rankings. John Mueller had already compared it to the keywords meta tag. Chrome is the exception: Lighthouse has an experimental Agentic Browsing audit that flags a server error when it fetches the file. A missing file is marked as not applicable, because the file is optional for now.

What does an llms.txt file look like?

Take an online shoe store. The file opens with "# Acme Shoes", followed by "> Running and hiking shoes, shipped in Europe." Next comes a "## Guides" section that links to the size guide and the returns policy, then a "## Products" section with the main categories. It ends with an "## Optional" section that links to the blog archives. That is about fifteen lines, and an AI can read all of it in one pass.

Thibault's take My advice: create one if you have 30 minutes, but don't expect traffic from it. Google says plainly that it neither helps nor harms you. What matters more is accessibility. If your content depends on JavaScript that bots don't render, AI models can't read it, with or without an llms.txt. Check your raw HTML first. After that, if you publish documentation, an API or a product catalog that agents will read, llms.txt is a useful map.

Frequently asked questions

Is llms.txt an official standard?

No. llms.txt is a community proposal that Jeremy Howard published in September 2024. Google Search does not use it, though Chrome Lighthouse checks for it in an experimental audit.

Where do I put an llms.txt file?

Put it at the root of your site, at yoursite.com/llms.txt. The proposal also allows it in a subfolder, where it covers the URLs under that path.

Does llms.txt replace robots.txt?

No. robots.txt tells bots what they are allowed to crawl, and llms.txt only suggests what is worth reading. To block an AI bot, you still use robots.txt.

Sources

  1. llmstxt.org/
  2. developers.google.com/search/docs/fundamentals/ai-optimization-guide
  3. developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
  4. searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask

Stay ahead of every search update

Google, ChatGPT, Gemini, Claude: what changed, why it matters, and what to do about it. Every morning, in 3 minutes, free.

What the...?