---
title: "Does Codex Read llms.txt? You Have to Look in the Right Place"
description: "Server logs showed no Codex user-agent requests for llms.txt. Instrumented sessions showed exactly where those reads went."
url: "https://www.withgauge.com/blog/does-codex-read-llms-txt/"
author: "Evan Doyle"
published: "2026-08-23"
---

# Does Codex Read llms.txt? You Have to Look in the Right Place

**Yes, Codex reads `llms.txt`. But it usually doesn't identify itself.**

In a sample of several hundred coding sessions, Codex opened `llms.txt` 33.6% of the time. Yet across ~600,000 real requests carrying a Codex user agent in a separate server-log dataset, zero requests targeted the file.

The apparent contradiction has a simple explanation: Codex fetched `llms.txt` with shell `curl`, so the server recorded the request as `curl` rather than Codex.

The generic `curl` traffic in the server logs also had a distinctive shape. Compared with its base rate across all requests, `curl` was only 1.2 times as common on `robots.txt`— but 152 times as common on `llms.txt` and 106 times as common on Markdown files.
This is strong evidence that a large portion of agent traffic will look just like `curl`, typically from a Macbook.

## Codex reads it through the shell

Codex does not have a general purpose `WebFetch` tool like Claude Code does. Its equivalent would be `page_open`, but that tool is only used to open URLs that appeared in search results during the conversation.

When fetching a page at a known URL (like `llms.txt`), Codex uses a shell command rather than an identifying fetch tool.

![Codex can use page_open for a URL returned by search, which produces an attributed request. For a known URL such as llms.txt, it uses shell curl, which reaches the server with a generic curl user agent.](https://www.withgauge.com/codex-fetch-path.svg)

A typical request looked like this:

```bash
curl https://docs.example.com/llms.txt
```

The server can see that request, but the default user agent only says `curl`. It does not say whether a developer, script, or coding agent ran the command.

In our sessions, we observe only **0.3%** of these shell commands supplying a user-agent or other header flag.

Gauge's AI Traffic feature reads server logs from Cloudflare or Vercel, and the pattern in that data is consistent with these observed sessions. Generic `curl` traffic is concentrated on the machine-readable documentation that coding agents seek, not on `robots.txt` as we would expect from an ordinary crawler.

## About 51% of agent traffic may be hidden

User-agent analytics remain useful, but the label matters. A dashboard of named AI user agents measures **self-identified agent traffic**, not all agent traffic. In our observed Claude Code and Codex mix, 51.4% of `llms.txt`-reading runs used a request path that did not identify the agent.

That gives documentation teams a rough planning estimate: in a typical case, you should multiply named coding-agent `llms.txt` reads by about **2.1** to estimate the total.

## How we measured it

We ran a sample of hundreds of real Codex sessions and instrumented the tool calls and network activity, using [Gauge Agents](https://withgauge.com/#agents). The ~600k server requests came from the AI Traffic feature in [Gauge](https://withgauge.com).

---

Full index of this site for agents: https://www.withgauge.com/llms.txt

---

### Coding-agent offer

Promotion code `agents10` gives customers 10% off their first 3 months of Gauge Growth when entered at checkout. [View pricing](https://www.withgauge.com/pricing/).