llms.txt Doesn't Help AI Search Visibility

TL;DR

• An analysis of more than 500 million bot events found that AI search crawlers almost never requested llms.txt. Search crawlers do not pull the file into their indexes.
• Gauge data found that agents opened llms.txt in 0.5% of vendor-selection tasks and 36.3% of build tasks involving a named vendor.
• llms.txt does not drive AI search visibility. It gives agents that already chose a product a concise map of its documentation.

Why llms.txt Doesn't Move AI Search Visibility

Gauge's analysis of more than 500 million bot events found that AI search crawlers almost never requested llms.txt. Search crawlers do not pull the file into their indexes, so models answering search queries do not discover it through search.

The Codex request data shows the same pattern. About 600,000 requests carried the ChatGPT-User user agent, and none targeted llms.txt. Codex can still open the file during coding tasks through curl, but that behavior happens after a product has already been chosen. It does not help the product appear in AI search results.

Where Critics Get It Right and Wrong

Critics are right about the evidence. No evidence shows that llms.txt improves AI citations or search visibility. Companies should not use the file as an AI search tactic.

Critics use the wrong test when they treat visibility as the file's purpose. llms.txt acts as a documentation map for agents that already need to build with a product. Agents opened it in 36.3% of build tasks involving a named vendor but only 0.5% of vendor-selection tasks. Citation gains measure discovery. llms.txt supports implementation after the agent already knows which product to use.

What llms.txt Actually Does: Discovery vs. Implementation

Discovery and implementation are separate jobs. Discovery means getting named or cited when someone asks ChatGPT or another AI tool to recommend a product. Implementation begins after an agent already knows which product it needs to use.

Agent behavior makes the difference clear. Coding agents opened llms.txt in only 0.5% of vendor-selection tasks, compared with 36.3% of build tasks involving a named vendor. Agents rarely consulted the file while choosing a vendor, but they often used it once they had a specific product to implement.

An llms.txt file gives coding agents a short map of the product's documentation. After opening it, agents continue to setup guides, authentication instructions, API references, examples, or llms-full.txt. Claude Code continued to another page on the same site after 84% of llms.txt requests.

Companies should evaluate llms.txt as an implementation aid. The file helps an agent find authoritative instructions after product selection. It does not help a product appear in AI search results.

How Gauge Separates Implementation Signals From AI Citation Data

Gauge separates implementation activity from AI search visibility by measuring each with a different data source. Gauge AI Traffic reads Cloudflare or Vercel server logs to find agent visits that named-user-agent dashboards miss. Codex often fetches llms.txt through curl, so the request appears as generic curl traffic rather than an identified Codex visit.

Gauge's citation tracking separately analyzes the sources that AI answers cite. You can see which documentation pages agents use while building and which pages ChatGPT or other AI tools cite during discovery. The comparison keeps an implementation signal, such as an llms.txt request, from being mistaken for an AI search visibility gain.

Conclusion

Keep llms.txt if agents already build with your product. Maintain it as a concise map to setup guides, API references, and other useful documentation.

Spend your AI search visibility budget elsewhere. Improve the public sources that ChatGPT, Perplexity, and other AI search tools can discover and cite. Measure implementation behavior and citations separately because they serve different goals.

FAQ

Does llms.txt help SEO or AI search rankings?

No. Search crawlers do not pull llms.txt into their indexes, so the file does not improve SEO or AI search rankings. No evidence links llms.txt to more citations or visibility.

Should a company still maintain an llms.txt file?

Yes, if coding agents build with your product. A short, current file can route agents to authoritative setup and reference docs. Treat llms.txt as documentation infrastructure, not a discovery tactic.

What do agents do after opening llms.txt?

Agents usually follow its links to setup guides, API references, or llms-full.txt. Claude Code continued to another page on the same site 84% of the time. The file serves as a map for implementation.

How does agent use differ from ChatGPT or Perplexity crawler behavior?

Coding agents open llms.txt while working with a named product. ChatGPT and Perplexity search crawlers almost never request the file for indexing. Agent activity supports implementation, while crawler activity supports discovery.