Why publish an llms.txt file?
The format was proposed in September 2024 by Jeremy Howard (Answer.AI). The idea: a modern web page is loaded with navigation, scripts and banners, while a language model needs clean text and a clear map of what matters. llms.txt fills that role, a bit like a table of contents written for AI.
To be precise about its reach: so far, the major engines have not publicly confirmed using it to decide which sources to cite. It is, however, read by some AI tools, coding assistants and agents that explore a site, and it costs almost nothing to maintain.
It is also a useful exercise: describing in a few lines what your company does and which pages are authoritative forces you to clarify your message, which benefits all your other pages.
How do you write a good llms.txt?
- 1Create a text file served at https://your-domain/llms.txt, encoded in UTF-8.
- 2Start with a level-1 heading containing the site or company name.
- 3Add a blockquote (a line starting with >) that sums up in two or three sentences what you do and for whom.
- 4Then group your links into sections (## Product, ## Docs, ## Blog...), each link formatted as [Title](URL) followed by a short description.
- 5List only public, up-to-date, useful pages: pricing, offers, documentation, reference articles.
- 6Update it whenever your offer or site structure changes significantly.
What mistakes should you avoid with llms.txt?
- Pasting your whole sitemap into it: it is a selection, not an inventory.
- Presenting it as a guarantee of being cited by ChatGPT or Google.
- Confusing it with robots.txt: llms.txt neither allows nor blocks anything.
- Leaving dead links or outdated pages after a redesign.