What is actually inside llms.txt

Most of these files do not exist

Before anything can be read, the file has to exist. This corpus is the result of an experiment rather than an incidental find: for CC-MAIN-2026-30 the paths /llms.txt and /llms-full.txt were added to the crawl's seed list on purpose, and what follows is the outcome of 6,563,125 attempted URLs.

The seeds were drawn two ways. Most are a random sample of hosts Common Crawl had recently fetched without problems — 5,167,831 URLs for /llms.txt, and a deliberately narrower 1,761,228 for /llms-full.txt, since a site with no /llms.txt rarely publishes the longer file. The rest are hosts already known from the two preceding crawls (CC-MAIN-2026-21 and CC-MAIN-2026-25) to serve one of them: 27,394 with a /llms.txt and 15,763 with a /llms-full.txt. Not every seed is fetched within a single crawl, which is why the totals below fall short of the sample. Because the first group is a random sample, the 200-plus-text rate for /llms.txt reads as an adoption rate for hosts Common Crawl can fetch. The /llms-full.txt rate does not, because its sample was enriched with hosts already known to serve the file.

The query below reads the URL index after the fact, to recover the outcome of each attempt. It is not what selected the population. Quoted verbatim, never re-run here.

select crawl, url, url_host_name, fetch_status, content_mime_detected
  from ccindex
 where (url_path = '/llms.txt'
        or url_path = '/llms-full.txt')
   and url_query is null
   and crawl like 'CC-MAIN-2026-30';

69.8% of attempts 404 and another 7.77% redirect, which is unsurprising: the crawler tries the path on every host it knows. What is surprising is the 85 distinct status codes it received, and how little of the 200 population is actually a document — only 45.63% of successful responses carry a text/plain or text/markdown body.

The content types tell you what the other 200s are. A site that serves its single-page app for every unknown path answers text/html; one whose rewrite rules send /llms.txt to /robots.txt answers text/x-robots. Both are a 200 that an agent would have to parse and discard.

Split by filename, llms-full.txt is both rarer and much more often misconfigured: it is attempted on 1,667,437 URLs but only 7.01% of its 200s carry a text body, against 52.50% for llms.txt.

One reconciliation note, because the two populations are not quite the same. The index query above matches the path exactly, while the WARC extraction that produced the dataset matched LIKE '%/llms.txt' and so also captured files deeper in a site — 10,996 of them (1.84%), things like /branding/vesence/llms.txt or one file per product. Set those aside and the 598,298 extracted records leave 587,302 at a root path, exactly the 587,302 the index counts.

The corpus

Those 598,298 surviving responses are the corpus: every URL ending in llms.txt or llms-full.txt that returned HTTP 200 with a text/plain or text/markdown body, extracted from the WARC archives. This report asks what is written in them. Published adoption studies count who has the file and who fetches it; none of them open it.

14,191 of those responses (2.37%, across 13,167 hosts) have an empty body: a 200, the right content type, and nothing in it. They are a finding about deployment, not documents to analyse — every empty body is identical, so leaving them in would manufacture a template cluster and inflate the share of files with no links, no title and no generator. They are counted here and excluded from every figure that follows, which is computed over the 584,107 responses that contain something.

Three findings organise everything below. First, this is plugin output, not curation: 68.3% of documents are template-generated, and the ten largest template clusters alone account for 38.7% of the whole corpus. Second, the file has quietly become a policy and service-discovery channel — 6.6% carry usage or rights language and 44.9% advertise an agent endpoint, neither of which the specification mentions. Third, it is unguarded: 2.54% are parked domains pitching themselves to agents, 1.20% hit a spam lexicon, and a small but non-zero share carry instructions written at the reading model.

The shape is followed; the substance is not

The llms.txt specification is small enough to test mechanically: an H1 title (the only required element), a > blockquote summary, optional heading-free prose, then ## sections of - [name](url): notes bullets, optionally including an ## Optional section that a short-context reader may skip.

99.2% of documents are markdown at all. The rest — 0.71% — are JSON error blobs, HTML error pages, robots.txt files served under the wrong name, and YAML policy documents, every one of them returned as HTTP 200 with a text/plain or text/markdown content type. (The 14,191 entirely empty responses are excluded here and counted in the overview.)

The surface form travels well. 49.90% of documents carry the full spec shape — H1, summary blockquote, H2 sections, link bullets — and that is not an artefact of one generator: 54.17% of human-authored documents reach it against 48.00% of templated ones. Whatever else is wrong with the llms.txt web, people did read the spec.

What does not travel is the part that carries the value. The specification's purpose is to hand an agent a curated, annotated list of links, and that is precisely where adherence collapses: only 32.94% of documents annotate any link with the : notes the format allows, the ## Optional section that lets a short-context reader skip material appears in 9.71%, and 22.6% of documents contain no link at all (22.2% of the markdown ones). The shape is followed; the substance is not.

Plugin output, not curation

Detection runs in two layers: an explicit self-declaration with a version string ("Generated by Yoast SEO v28.0, this is an llms.txt file"), and, for the silent generators, a language-independent structural fingerprint — for the site builders, the help-centre URL they embed rather than the prose around it, so that localised templates are still detected.

The largest single producer is wix at 41.34% of the corpus. Together with the WordPress SEO plugins, a handful of vendors decide what most of the llms.txt web says — and they interpreted the same informal spec in mutually incompatible ways: some emit a curated index, some dump the sitemap, some emit no links at all and advertise an API instead.

Template detection needs no statistical model. Dropping the parts that are site-specific by construction — the title, the summary blockquote, the link bullets — and then erasing URLs, e-mail addresses, numbers and every non-ASCII run leaves the vendor's boilerplate and section structure. Exact matching on that skeleton is enough: one template collapses to a single string across thousands of sites. Translated boilerplate forms one cluster per language, which is what the generator fingerprints unify; skeleton clustering is what catches template families whose producer is unknown.

68.3% of the corpus is templated by this measure; the ten biggest clusters alone are 38.7%. Every rate elsewhere in this report is therefore reported against the human-authored remainder wherever the distinction matters.

One template family deserves its own line. 44.9% of documents mention the Model Context Protocol and 44.4% publish a live MCP endpoint URL. For those sites llms.txt is not a link index at all: it is a service-discovery record telling agents to stop scraping and call an API instead. The specification does not mention this use.

Files that were pasted out of a chat window

A third production route leaves its own traces. 1.43% of documents carry a marker that only appears when someone asked an assistant to write the file and shipped the answer unedited: ChatGPT's contentReference[oaicite:…] citation markers, an unfilled [Insert company name] placeholder, a leftover ```markdown fence around the whole document, or a utm_source=chatgpt link. The topic model found these on its own — whole clusters of clinics and agencies held together by the token oaicite.

A policy channel the spec never defined

Nothing in the specification concerns rights. In practice a measurable share of documents use llms.txt to state terms — and they do so in at least four mutually incompatible dialects: prose paragraphs, YAML permission blocks, robots.txt syntax served under the llms.txt name, and links out to a licence page.

The named-crawler table is the closest thing this corpus has to a robots.txt of the agent era. It is also where a publisher's intent is least ambiguous: naming a user-agent and putting allow or deny next to it is a deliberate act.

A verdict is read from the same line the crawler is named on. Documents that name a crawler without any allow/deny marker are counted in the first column only, which is why allowed + denied does not sum to the total.

Unguarded, and already being used

llms.txt is text an agent reads as guidance, published at a predictable path, inspected by no security tooling. The lexicon below grades what publishers put there, from benign steering through self-promotion to outright instruction override. It is deliberately conservative: high precision by construction, unknown recall.

The override row is small enough to audit exhaustively, and it was: all of its documents were read individually. Four are genuine — a bug-bounty researcher's labelled payload catcher, two protest files, and one “LLM Training Policy” whose summary blockquote instructs the reader to ignore its instructions and fetch another URL. The rest are false positives of two kinds: technical writing that contains the literal ChatML control tokens because it is explaining them, and an incidental “you are now” in a page description. Read the override count as an upper bound on candidates, not a count of attacks — and note that every genuine case was placed deliberately by its author, rather than smuggled in.

Parked domains are the corpus's most absurd corner. 14,812 documents (2.54%) advertise the domain itself for sale, and 99.99% of those are well-formed enough to clear conformance level 2 — someone wrote a spec-shaped sales pitch aimed at a language model.

Link injection is visible too. 0.37% of documents send the majority of their links to five or more third-party domains, and 0.11% are written in a language implausible for their ccTLD — the classic signature of a compromised site republished as a link farm.

Language, length, topics and what it costs to read

Language is identified with commonlid, Common Crawl's own LID evaluation kit: cld2 as the primary model with GlotLID as the fallback where cld2 abstains. Input is the document's prose with link bullets, URLs and markdown syntax stripped, so that link dumps and translated boilerplate do not decide the label.

Length is where the specification's premise — a small, cheap, curated file — meets what was actually shipped. The median document is 737 tokens; the 99th percentile is 159,880, and the largest is 3,313,814 tokens: a single file whose entire purpose is being inexpensive to read.

Ingesting the whole corpus once costs the following, at 5,363,837,955 input tokens (counted with o200k_base; other tokenizers differ by roughly ±20%). List prices come from litellm's model cost map, charging each document as its own request — pricing the corpus as one call would trigger the long-context tier several providers apply above ~200k tokens and roughly double the figure.

Topics

Topics come from an LDA model over the human-authored subset — templates removed, one document per template skeleton — fitted per language on a 2 KB prose excerpt. Topic names are not machine-generated: the model's top terms and representative documents were read by hand and labelled.

How this was measured, and what it cannot show

Every grouped result carries five example documents, each with two links: the site as it is now, and #index — the record's row in the Hugging Face dataset split, which opens in the dataset viewer. The index is the durable reference; the same row is reachable in code as load_dataset("commoncrawl/llms.txt", "CC-MAIN-2026-30")["train"][index], and returns the exact bytes a figure was computed from long after the live page has changed or gone.

Source: the Hugging Face dataset commoncrawl/llms.txt, config CC-MAIN-2026-30 — WARC response records for */llms.txt and */llms-full.txt with HTTP 200 and a text/plain or text/markdown body, 584,107 documents. Analysis code: analyze.py in the llms-txt-experiments repository; every number here is produced by analyze.py aggregate from a single streaming pass over the corpus.

Limitations

• One crawl snapshot. Nothing here supports a claim about growth or change over time.
• The corpus is HTTP 200 plus text/plain|markdown only. Sites that serve llms.txt as text/html, redirect, or 404 are absent — in this crawl those are the majority of all /llms.txt URL attempts.
• The seeds are a random sample only within the set of hosts Common Crawl fetches successfully, which is not the same population as the web. The /llms.txt success rate is therefore an adoption estimate for that population and not for the web at large, and the /llms-full.txt rate is not an adoption estimate at all, its sample having been enriched with hosts already known to serve the file.
• Instruction severity, policy directives and spam categories are lexicons, not classifiers. They are tuned for precision and were validated by reading sampled matches; recall is unknown and every rate here is a lower bound.
• A spam-lexicon hit means the vocabulary is present at least twice, not that the site is spam — though in practice most hits are hijacked legitimate sites rather than purpose-built ones. Successive spot checks removed bare slot (a fifth of its matches were appointment slots, a Danish castle and a football manager), word-bounded judi (it was matching "judicial"), dropped bare xxx (it matched the Roman numeral in "XXXI Velada Musical"), and required two occurrences rather than one, which excluded a car service advertising "casino trips". Measured precision after those changes is about seven in eight.
• Policy and abuse lexicons scan the first 20 KB and last 4 KB of each document, so text buried in the middle of a multi-megabyte dump can be missed.
• Token counts use one tokenizer (o200k_base).
• 14,191 responses (2.37%) had an empty body and are excluded from every figure except the intake table in the overview. They are all byte-identical, so including them would create a spurious template cluster and drag down the rates for links, titles and generators.
• A document counts as templated when its normalised skeleton is shared by at least 50 documents, or when a generator was positively identified.
• LDA topic labels were assigned by a human reading top terms and example documents — reproducible, but a judgement call.