Yes, Codex reads llms.txt. But it usually doesn't identify itself.
In a sample of several hundred coding sessions, Codex opened llms.txt 33.6% of the time. Yet across ~600,000 real requests carrying the ChatGPT-User user agent in a separate server-log dataset, zero requests targeted the file.
The apparent contradiction has a simple explanation: Codex fetched llms.txt with shell curl, so the server recorded the request as curl rather than Codex.
The generic curl traffic in the server logs also had a distinctive shape. Compared with its base rate across all requests, curl was only 1.2 times as common on robots.txt— but 152 times as common on llms.txt and 106 times as common on Markdown files.
This is strong evidence that a large portion of agent traffic will look just like curl, typically from a Macbook.
Codex reads it through the shell
Codex does not have a general purpose WebFetch tool like Claude Code does. Its equivalent would be page_open, but that tool is only used to open URLs that appeared in search results during the conversation.
Those visits use OpenAI's ChatGPT-User/1.0 user agent, which identifies the request as a user-triggered OpenAI visit but not as Codex specifically.
When fetching a page at a known URL (like llms.txt), Codex uses a shell command rather than an identifying fetch tool.
A typical request looked like this:
curl https://docs.example.com/llms.txt
The server can see that request, but the default user agent only says curl. It does not say whether a developer, script, or coding agent ran the command.
In our sessions, we observe only 0.3% of these shell commands supplying a user-agent or other header flag.
Gauge's AI Traffic feature reads server logs from Cloudflare or Vercel, and the pattern in that data is consistent with these observed sessions. Generic curl traffic is concentrated on the machine-readable documentation that coding agents seek, not on robots.txt as we would expect from an ordinary crawler.
About 51% of agent traffic may be hidden
User-agent analytics remain useful, but the label matters. A dashboard of named AI user agents measures self-identified agent traffic, not all agent traffic. In our observed Claude Code and Codex mix, 51.4% of llms.txt-reading runs used a request path that did not identify the agent.
That gives documentation teams a rough planning estimate: in a typical case, you should multiply named coding-agent llms.txt reads by about 2.1 to estimate the total.
How we measured it
We ran a sample of hundreds of real Codex sessions and instrumented the tool calls and network activity, using Gauge Agents. The ~600k server requests came from the AI Traffic feature in Gauge.
Related Blogs
How to Write a Good llms.txt File (2026)
Give agents a useful directory of your pages, with context on what to read and how. Use the opening paragraphs to explain your brand, products, pricing, and who each offering helps.
Ethan FinkelChatGPT business profiles: How to set up your profile (Ultimate Guide)
A hands-on guide to claiming your business in ChatGPT, verifying your identity, adding products, configuring your Business Agent, and reviewing its answers, with screenshots from Gauge’s setup.
Ethan FinkelWhere AI Models Get Their Information: A Complete Guide to Getting Your Content Indexed
How ChatGPT, Claude, Google AI Overviews and AI Mode, and Perplexity discover content, and the steps you can take to make your pages available to them.
Ethan Finkel
Evan Doyle