robots.txt, llms.txt & Sitemaps for AI Crawlers

This guide is for publishers who want AI crawlers to find their content and pay for the deep data. You will see how AgentFlare publishes robots.txt, llms.txt and a sitemap as free discovery files, how llms.txt and pricing.json point agents to the paid tier, and how directory prices are set. You will also learn how to check each file with curl and how an agent reads a 402 or 403 response when it requests a priced URL. The goal is an open discovery surface with priced deep payloads behind it.

How the flow works

Before you start

• A publisher site that can serve static text files at its root
• curl installed in a terminal
• A list of directories you want to price

Step by step

1. Allow AI crawlers in robots.txt

   robots.txt is the first file a crawler reads. AgentFlare opens the site to everyone with User-agent: * and Allow: /, then repeats the rule for named bots: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Googlebot and Bingbot. The file ends with a Sitemap line. Pricing is not set here. It is enforced at the edge on the paid file types.

   # AgentFlare - AI crawlers welcome
   User-agent: *
   Allow: /
   
   User-agent: GPTBot
   Allow: /
   
   User-agent: ClaudeBot
   Allow: /
   
   User-agent: PerplexityBot
   Allow: /
   
   # ...same pattern for the other bots
   Sitemap: https://agentflare.org/sitemap.xml

   S2 · agentflare.org

2. Publish llms.txt as a site map

   llms.txt tells an agent how the site is organized and where the paid tier starts. AgentFlare's file opens with a title and a one-line summary. It explains that every article exists in three forms: HTML, a free Markdown teaser (.md), and a metered .data.json or .full.md. It then links the verify endpoint, pricing.json, registration and onboarding. Each section has its own llms.txt listing its pages.

   # AgentFlare
   
   > The agent-native web. Live data from the AISA paid-API network, with HTTP 402 pay-per-crawl access to structured deep data.
   
   Pricing map: https://agentflare.org/pricing.json
   Register: https://esa.aisa.one/agent/login.html
   Onboarding: https://esa.aisa.one/agent/guide.html
   
   Sections: Sections, Start here, Latest, API, Optional

   S3 · agentflare.org

3. Declare tiers and prices in pricing.json

   pricing.json is the price map an agent can fetch for free before buying anything. AgentFlare lists tier L1_free (index pages, .html articles, .md teasers, llms.txt, sitemap.xml, robots.txt) and tier L2_paid (*.data.json and *.full.md). Directory prices are news 0.005, research and scholar 0.02, and crypto, markets, social and predictions 0.01. Anything unmatched stays free under default allow. Import the rules into cdn-agent Content Controls.

   curl -s https://agentflare.org/pricing.json

   S4 · agentflare.org

4. Check every discovery file is free

   Fetch each discovery file the way an agent would, with no token. The free tier covers HTML pages, .md summaries and discovery files, and these cost $0. If any of these returns 402 or 403, a rule is catching it by mistake. Agents are advised to read these files first to learn the cost landscape, then read summaries to decide what is worth buying.

   for f in robots.txt llms.txt sitemap.xml pricing.json; do
     curl -s -o /dev/null -w '%{http_code} '$f'\n' https://agentflare.org/$f
   done

   S2 · agentflare.orgS3 · agentflare.orgS4 · agentflare.orgS5 · esa.aisa.one

5. Fetch a paid file with X-Crawler-Token

   An agent that wants deep data sends one extra header, X-Crawler-Token, which replaces the earlier name X-AISA-Crawler-Token still shown on older AISA pages. Send it with an honest User-Agent. Never put the token in Authorization: Bearer. A 200 means the token was charged the directory price. A 402 or 403 body explains what to do next.

   curl -i 'https://PUBLISHER-SITE/research/report.data.json' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS5 · esa.aisa.one

AgentFlare directory rules in pricing.json

Common pitfalls

• Do not send the crawler token in Authorization: Bearer. That header is reserved for the gateway layer; send X-Crawler-Token instead.
• A 403 access_blocked means the publisher withholds the resource, as with /members/. Payment does not unlock it, so agents should move to pay-mode directories instead.
• robots.txt only opens crawling and lists the sitemap. It carries no prices, so keep price rules in pricing.json and in the edge configuration.
• Fetching a priced file twice charges twice. Have agents read the free .md teaser first to decide whether the deep file is worth buying.

Key takeaways

• Free discovery files (robots.txt, llms.txt, sitemap.xml, pricing.json) stay open; only .data.json and .full.md are metered.
• llms.txt describes the three forms of each article and links pricing.json, so agents learn the cost before paying.
• Agents identify with X-Crawler-Token and an honest User-Agent; denials return JSON that says what to do next.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.