What Is llms.txt Used For? Data From Opus 5 and GPT-5.6 Sessions

llms.txt is most useful when an agent has to build something.

In sessions where Claude Code or Codex was asked to build against a named vendor, 36.3% opened an llms.txt. In vendor-selection sessions—tasks such as choosing an auth provider—only 0.5% did.

The agents were similar. The job was different.

1. Build tasks create the need for an index

A buy prompt can often be answered from search results and existing model knowledge. The agent needs to compare positioning, features, and reputation, but it does not yet need the exact shape of an API.

A build prompt forces the agent into the documentation. It needs setup instructions, authentication details, reference pages, examples, and error behavior. That is when a compact map of the documentation becomes useful.

Captured in Gauge Agents, which runs coding agents against real product tasks.

2. It points agents to the authoritative source

Agents use llms.txt to learn where information lives. They commonly move from the index to per-page Markdown, llms-full.txt, an OpenAPI specification, a typed SDK, or shipped source code.

The index answers “where should I look?” For exact API shapes, agents still prefer the closest available source of truth.

3. Real traffic shows what happens next

Gauge AI Traffic shows the same pattern outside the instrumented sessions. After requesting llms.txt, Claude Code continued to another page on the same host 84% of the time. In 14% of visits, it fetched at least ten distinct follow-up paths.

The generic curl traffic that includes Codex went deeper still: 94% continued past the index, and 39% reached at least ten follow-up paths. The paths include focused documentation visits, moves to llms-full.txt, and broad research across Markdown pages and product documentation.

For coding agents, llms.txt usually acts as the beginning of the documentation visit.

4. It can still influence a buying decision

Buy sessions rarely opened llms.txt. When an agent did encounter one, however, it sometimes treated the file's presence and quality as evidence that a product was ready for agent-driven development.

Captured in Gauge Agents, which runs coding agents against real product tasks.

A thin or stale file can send the opposite signal. Agents sometimes treated missing pages as evidence that the underlying capability did not exist.

Captured in Gauge Agents, which runs coding agents against real product tasks.

What should an llms.txt file do?

The observed uses suggest four priorities:

• Put it at the root: yourdomain.com/llms.txt, where agents expect to find it.
• Make it a curated map to core concepts, setup, authentication, API references, examples, and troubleshooting.
• Link directly to authoritative contracts and complete documentation rather than asking the index to carry implementation details.
• Keep it current and complete. Omissions can be mistaken for missing product capabilities.

How we measured it

The build result comes from Gauge Agent Experience sessions where Claude Code using Opus 5 or Codex using GPT-5.6 was asked to build against a named vendor. The buy result comes from labelled vendor-comparison sessions. Both measure whether the agent opened an llms.txt during the task.

The follow-up-path analysis comes from Gauge AI Traffic. Requests are grouped into 10-minute window sessions on the same host. The “Codex & Other” row is generic curl traffic, which includes coding agents that do not identify themselves.