Does Codex Read llms.txt? You Have to Look in the Right Place

Yes, Codex reads llms.txt. But it usually doesn't identify itself.

In a sample of several hundred coding sessions, Codex opened llms.txt 33.6% of the time. Yet across ~600,000 real requests carrying a Codex user agent in a separate server-log dataset, zero requests targeted the file.

The apparent contradiction has a simple explanation: Codex fetched llms.txt with shell curl, so the server recorded the request as curl rather than Codex.

The generic curl traffic in the server logs also had a distinctive shape. Compared with its base rate across all requests, curl was only 1.2 times as common on robots.txt— but 152 times as common on llms.txt and 106 times as common on Markdown files. This is strong evidence that a large portion of agent traffic will look just like curl, typically from a Macbook.

Codex reads it through the shell

Codex does not have a general purpose WebFetch tool like Claude Code does. Its equivalent would be page_open, but that tool is only used to open URLs that appeared in search results during the conversation.

When fetching a page at a known URL (like llms.txt), Codex uses a shell command rather than an identifying fetch tool.

A typical request looked like this:

curl https://docs.example.com/llms.txt

The server can see that request, but the default user agent only says curl. It does not say whether a developer, script, or coding agent ran the command.

In our sessions, we observe only 0.3% of these shell commands supplying a user-agent or other header flag.

Gauge's AI Traffic feature reads server logs from Cloudflare or Vercel, and the pattern in that data is consistent with these observed sessions. Generic curl traffic is concentrated on the machine-readable documentation that coding agents seek, not on robots.txt as we would expect from an ordinary crawler.

About 51% of agent traffic may be hidden

User-agent analytics remain useful, but the label matters. A dashboard of named AI user agents measures self-identified agent traffic, not all agent traffic. In our observed Claude Code and Codex mix, 51.4% of llms.txt-reading runs used a request path that did not identify the agent.

That gives documentation teams a rough planning estimate: in a typical case, you should multiply named coding-agent llms.txt reads by about 2.1 to estimate the total.

How we measured it

We ran a sample of hundreds of real Codex sessions and instrumented the tool calls and network activity, using Gauge Agents. The ~600k server requests came from the AI Traffic feature in Gauge.