llms.txt, schema and structure for AI search

What llms.txt is supposed to be

The idea dates back to 2024 and is quickly explained: a text file in the root directory of the domain, similar to robots.txt, in which a company lists its most important content in plain text and Markdown. The thinking behind it: language models shouldn't have to wade through navigation, cookie banners and layout code, but get a clean summary directly.

That sounds reasonable. The problem is how binding it is. There is no authority that has defined it, no commitment from any provider to read the file, and to this day no reliable evidence that any of the major systems actually evaluates it.

It is a proposal with backing in the professional community, but without takers. That may change, it just isn't a basis for an investment today.

The shortcut that isn't one

The idea dates back to 2024 and is quickly explained: a text file in the root directory of the domain, similar to robots.txt, in which a company lists its most important content in plain text and Markdown. The thinking behind it: language models shouldn't have to wade through navigation, cookie banners and layout code, but get a clean summary directly.

That sounds reasonable. The problem is how binding it is. There is no authority that has defined it, no commitment from any provider to read the file, and to this day no reliable evidence that any of the major systems actually evaluates it.

It is a proposal with backing in the professional community, but without takers. That may change, it just isn't a basis for an investment today.

What structured data does

Schema.org is the counterpart: established since 2011, jointly supported by Google, Microsoft, Yahoo and Yandex, documented and verifiable in practice. The markup sits in the source code as JSON-LD and tells machines what a piece of content is: an article, a company, a collection of questions and answers, a review, a product.

For search, this has been the basis for rich results for years: review stars, additional details, event data. For generative systems, it is something different and more important: an unambiguous statement about who is behind a piece of content, when it was created and what it refers to.

This is exactly where it is decided whether a system classifies content as citable. A text without a recognizable author, without a date and without a connection to an organization is, when in doubt, a second-choice source.

The five markup types that are enough

You don't need the entire vocabulary. Five types cover what a service provider needs.

Organization, once, centrally: name, logo, address, founding date, social media profiles. This is the anchor everything else refers to.

WebSite: the domain itself, with language and a reference to the organization.

BlogPosting or Article: per article, with headline, description, image, date and author as a Person.

BreadcrumbList: the path. It shows machines where a page sits in the structure and takes just a few lines.

FAQPage: for pages with real questions and answers. Important: the answers must be visible on the page, otherwise the markup is not permitted.

Everything else is fine-tuning. If you have these five right, you are further ahead than most competitors.

The mistake that cancels out everything before it

The most common cause of invisibility is neither a missing file nor missing markup. It is more mundane: the content isn't in the HTML.

Many modern sites load their text via JavaScript. In the browser, everything looks right. In the delivered document, there is nothing in that spot. Search engines can partly compensate because they render pages. The tools generative systems use to fetch pages usually don't.

The test takes ten seconds: open the page, view the source code, search for a sentence from the middle of the text. If it isn't found, the content doesn't exist for these systems, no matter how well it is marked up.

This especially affects expandable sections, tabs and lazy-loaded lists.

The order of implementation

First, the HTML test for all important page types. If something is missing here, no further measure has any effect.

Then add Organization and WebSite once, centrally. Effort: one hour. Effect: the foundation for everything else.

Then Article and BreadcrumbList on the content pages, ideally via the template, so new articles include them automatically.

Then FAQPage, where questions are actually answered.

Last, if there is time, llms.txt. The file can be written in twenty minutes and does no harm. It is a bet that the proposal will catch on, a cheap bet, but a bet nonetheless.

If you reverse the order and start with llms.txt, you end up with a file nobody reads and a website machines can't classify.

How to check the implementation

For structured data, there are two free testing tools from Google: the Rich Results Test and the Schema Markup Validator. The first tells you whether markup qualifies for rich results, the second whether it is formally correct. Both should run without errors.

A third test is more important and rarely done: fetch the page with a simple request without JavaScript and look at what arrives. That shows what a language model actually gets to see.

After implementation, it is worth checking Search Console. Detected markup shows up there with a delay of a few days to weeks, along with errors and warnings. Warnings are usually not critical, errors mean the markup is ignored.

After that, a check once a quarter is enough.