llms.txt: The Standard for AI-Readable Sites

llms.txt: A Standard for AI Crawlers

llms.txt is a proposed, human‑ and machine‑readable standard that helps Large Language Models (LLMs) and AI agents discover, understand, and cite your website’s most important content. It lives at the root of your domain (e.g., https://example.com/llms.txt) and is typically written in Markdown, acting as a curated index rather than a blocking directive.

What llms.txt Is (and Isn’t)

llms.txt is not a replacement for robots.txt. Instead of access control, it focuses on routing and context: it tells AI crawlers which pages matter, why they matter, and how they relate. The file is a Markdown‑formatted list of URLs with short one‑line descriptions, often grouped by theme (e.g., “pricing”, “API docs”, “support”). It is optional for AI agents to obey, but major LLM‑driven systems increasingly treat it as a primary source of truth for citation and agentic workflows.

How AI Agents Use llms.txt

AI agents that perform web‑based tasks (e.g., research, booking, API discovery) use llms.txt to quickly locate high‑signal pages without parsing complex HTML. The file can also reference a companion llms-full.txt that concatenates Markdown‑stripped content from key pages, enabling agents to ingest a compact, structured corpus in one request. This reduces latency, hallucination risk, and bandwidth usage, especially for agents that need to maintain context over multiple steps.

HTTP 402 and Pay‑Per‑Crawl Scenarios

Some publishers experiment with HTTP 402 (Payment Required) responses for AI crawlers, effectively gating access to certain content behind micropayments or tokens. In this context, llms.txt can document which endpoints or pages require payment, what payment methods are accepted, and how agents should authenticate. This lets AI agents negotiate access programmatically, rather than blindly scraping and later facing blocks or legal issues.

Key Takeaways

• llms.txt is a Markdown‑based index at your domain root that tells LLMs and AI agents which pages are most important and how they relate.
• It complements robots.txt by focusing on routing and context, not blocking; agents are encouraged but not required to follow it.
• AI agents use llms.txt (and optionally llms-full.txt) to reduce parsing overhead, improve citation accuracy, and support multi‑step agentic workflows.
• When combined with HTTP 402 or pay‑per‑crawl models, llms.txt can document payment requirements and access policies so agents can negotiate access programmatically.

Synthesized by the AISA LLM layer with live web sources (AISA Perplexity + Tavily APIs). 2026-06-23.