What Is llms.txt? An Example File
I published a concise guide to my estate for visitors who read by machinery, and it leads them directly into the swamp.
The short answer
- What it is
- llms.txt is a proposed Markdown file at the root of a site. It gives AI tools a short summary of the site and a list of its most useful pages.
- Why it matters
- A concise file can help tools that choose to read it, such as coding assistants reading product docs. It is not a formal standard, and it does not control access or rankings.
- How a crawler should report it
- Report whether /llms.txt answers 200 with a text content type, and whether it has an H1 title. List any linked URLs that are broken, and treat absence as a notice and not an error.
- Live examples
https://crawltestlab.com/llms.txt(a deliberately flawed file)
A Plain Markdown Guide Written for AI Tools
What is llms.txt? It is a text file, placed at /llms.txt, that describes a website in a form language models can read easily. Jeremy Howard of Answer.AI proposed it in September 2024. The idea is simple. HTML pages are full of menus, scripts, and ads, while a short Markdown file holds just the key points. Sites with large documentation sets were among the first to adopt it.
The proposed format is short. It opens with an H1 that holds the site name, followed by a blockquote summary. Next come optional notes and H2 sections that list links, each with a brief description. Unlike robots.txt, this file gives no orders. It cannot block a crawler or grant permission for anything. A section named Optional marks links that a tool may skip when it has little room.
Be realistic about the benefits. Google has said you do not need new machine-readable files or special AI text files to appear in its AI features. There is no general proof that the file raises visibility in AI answers. Its best use is practical. A developer tool can point at it and find your docs quickly. To check a file, request it, confirm it returns status 200, and test the links.
If you decide to publish one, keep it short and keep it current. Link to pages that answer real questions, and write one plain line about each. Do not paste the entire site into it, unless you offer that as a separate file. Never include anything that you would not want to publish, since the file is public. Generate it from your docs if you can, so that the links do not go stale.