Pricing Content by Directory: Best Practices

This guide is for publishers who want to monetize AI crawler access by directory without hurting search visibility. You will learn how to split content into a free tier and a paid tier, write one rule per directory with a regex URL pattern, set the mode, per-request price and priority, and open or block special directories. The worked example is the real AgentFlare pricing.json, which keeps teasers open while charging for deep data files. By the end you can read any rule in that file, copy the structure for your own directories, and check how a crawler sees it.

How the flow works

Before you start

• A publisher domain behind an edge gateway or CDN
• Access to AISA Content Controls or rules import
• Basic familiarity with regular expressions

Step by step

1. Split content into free and paid tiers

   Start by deciding which files stay open and which are metered. AgentFlare keeps index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt in the L1_free tier to drive SEO and GEO. Complete datasets and full analysis, the *.data.json and *.full.md files, form the L2_paid tier. Open teasers let agents judge what is worth buying before they pay.

   L1 free: index pages, .html articles, .md teasers, llms.txt, sitemap.xml, robots.txt
   L2 paid: *.data.json + *.full.md

   S2 · agentflare.org

2. Set the default mode to allow

   Set default_mode to allow so anything your rules do not match stays free and is not charged. This keeps the whole SEO surface open without listing every public URL. Rules then only need to describe the exceptions: the deep files you meter and the directories you block. Matching is first-match-wins, and URL query strings are stripped before a rule is matched.

   default_mode: allow
   unmatched URL -> 200, not charged

   S2 · agentflare.orgS3 · esa.aisa.one

3. Write one pay rule per directory

   Give each paid directory its own rule. Use patternType regex, mode pay, a price, currency USD and billingType per-request. The pattern anchors on the directory and matches only the deep extensions, so teasers in the same directory stay free. In AgentFlare, news is priced at 0.005, while crypto, markets, social and predictions are 0.01 and research and scholar are 0.02.

   description: Newswire - deep structured data (L2, metered)
   urlPattern: ^/news/.*\.(data\.json|full\.md)$
   patternType: regex
   mode: pay
   price: 0.005
   currency: USD
   billingType: per-request
   priority: 100

   S2 · agentflare.org

4. Assign priorities across directories

   Every rule carries a priority. AgentFlare counts down from 100 for news, then 99 for social, 98 for markets, 97 for predictions, 96 for crypto, 95 for research and 94 for scholar. Because evaluation is first-match-wins, keep patterns anchored to one directory so rules never overlap, and use priority to make any intended precedence explicit. The notes field says these fields map 1:1 to the publisher rule schema.

   S2 · agentflare.orgS3 · esa.aisa.one

5. Open guides and block members

   Not every deep file should be sold. AgentFlare uses mode allow with price 0 for guides, so agents read that data free. It uses mode block with price 0 for members, so crawlers always get 403 access_blocked regardless of token or balance. A block is a policy choice that payment does not unlock, and AISA returns JSON only, so render your own block page.

   ^/guides/.*\.(data\.json|full\.md)$ -> mode allow, price 0, priority 93
   ^/members/.*\.(data\.json|full\.md)$ -> mode block, price 0, priority 92

   S2 · agentflare.orgS3 · esa.aisa.oneS5 · esa.aisa.one

6. Test the rules as a crawler

   Import the rules into Content Controls, then fetch your pricing.json and crawl a paid URL the way an agent would. The crawler identity header is X-Crawler-Token, which replaces the earlier name X-AISA-Crawler-Token. A missing token on a pay rule returns 402 with the rule and price in the body. Never put the token in Authorization: Bearer, because that header is reserved for the gateway layer.

   curl -s https://agentflare.org/pricing.json
   
   curl -i https://PUBLISHER-SITE/research/report.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS2 · agentflare.orgS5 · esa.aisa.one

AgentFlare directory rules

Common pitfalls

• Each granted fetch of a priced resource is billed once, so fetching the same URL twice charges twice.
• A crawler token sent in Authorization: Bearer fails the gateway layer with 401 gateway_unauthorized instead of the expected token check.
• Do not rewrite 402 or 403 bodies. The onboarding and guidance blocks are how new crawlers learn to register and fund a balance.
• A block rule returns 403 access_blocked even with a valid token and balance, so reserve it for content you never want crawled.

Key takeaways

• Keep teasers and discovery files free, and meter only .data.json and .full.md files by directory.
• Each rule needs a regex urlPattern, a mode of pay, allow or block, a per-request price and a priority.
• Set default_mode to allow so unmatched URLs stay free, and use block only for content like members.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.