{
  "@context": "https://agentflare.org/schema",
  "type": "Article",
  "tier": "L2-full",
  "title": "Pricing Content by Directory: Best Practices",
  "description": "How publishers price content by directory for AI agents (pay, free, blocked tiers) Publishers can now segment their site by directory and apply different AI‑access rules…",
  "canonical": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html",
  "category": "guides",
  "updated": "2026-06-23",
  "generated_at": "2026-06-23T15:42:04.744Z",
  "facts": [
    {
      "label": "Access",
      "value": "Free / open"
    },
    {
      "label": "Updated",
      "value": "2026-06-23"
    }
  ],
  "data": {
    "topic": "how publishers price content by directory for AI agents (pay, free, blocked tiers)",
    "access": "free",
    "summary": "How publishers price content by directory for AI agents (pay, free, blocked tiers) Publishers can now segment their site by directory and apply different AI‑access rules…"
  },
  "analysis_md": "# How publishers price content by directory for AI agents (pay, free, blocked tiers)\n\nPublishers can now segment their site by directory and apply different AI‑access rules (pay, free, or blocked) using HTTP‑based pay‑per‑crawl or per‑query protocols such as AISA HTTP 402. This guide walks through a practical setup you can implement today.\n\n## 1. Map your site into logical directories\n\nStart by grouping your content into directories that reflect value and risk to AI use (for example `/blog`, `/api/articles`, `/docs`, `/premium`). For each directory, decide the default policy:\n\n- **Free tier**: Public, low‑value, or marketing content you want AI to index freely (e.g., `/blog`).\n- **Pay tier**: High‑value or proprietary content you want AI agents to pay for (e.g., `/premium`, `/api/articles`).\n- **Blocked tier**: Sensitive, internal, or non‑public content you want to block entirely (e.g., `/admin`, `/internal`).\n\nDocument this mapping in a simple table so your team and AI partners can see which paths are free, pay, or blocked.\n\n## 2. Configure HTTP 402 and per‑query rules per directory\n\nUse a pay‑per‑crawl or per‑query service (such as Cloudflare’s pay‑per‑crawl or xpay‑style systems) that supports AISA HTTP 402 and custom headers like `X‑402‑Price`. For each directory:\n\n- **Free directories**: Return HTTP 200 for AI crawlers; no 402 or payment headers.\n- **Pay directories**: Return `402 Payment Required` with pricing metadata (e.g., `X‑402‑Price: 0.05`, `X‑402‑Currency: USDC`) when an AI agent hits that path. The agent must then send a signed payment header to proceed.\n- **Blocked directories**: Return `402` or `403` with no payment option, or simply block via bot‑management rules.\n\nMany services let you configure this per path in a dashboard or via config files, so you can enforce different tiers without changing your core application code.\n\n## 3. Communicate tiers and monitor usage\n\nPublish a simple `/ai-access` or `/robots.txt`‑style policy page that lists which directories are free, pay, or blocked, and what the typical per‑query or per‑crawl price is. For example:\n\n- `/blog` → free indexing  \n- `/api/articles` → pay per query via 402 + USDC  \n- `/internal` → blocked  \n\nUse the observability dashboards from your pay‑per‑crawl provider to track which directories get the most AI traffic, how many 402s convert to paid requests, and whether any agents repeatedly hit blocked paths. Adjust pricing or tiering (for example, lowering price on low‑conversion pay directories or tightening blocks on sensitive ones) based on this data.\n\n## Key takeaways\n\n- Group your site into **free**, **pay**, and **blocked** directories and enforce them via HTTP status codes and 402‑style headers.\n- Use **AISA HTTP 402** and per‑query protocols to charge AI agents only on high‑value paths while keeping public content open.\n- **Monitor and iterate**: treat AI‑access tiers like a product, adjusting prices and policies based on real‑time traffic and conversion data.",
  "sources": [
    {
      "url": "https://www.neuralstackly.com/blog/ai-content-pay-per-crawl"
    },
    {
      "url": "https://www.xpay.sh/blog/article/how-publishers-monetize-ai-traffic/"
    },
    {
      "url": "https://www.contentgrip.com/ai-publishers-crawl-fees/"
    },
    {
      "url": "https://www.mindstudio.ai/blog/creator-economy-ai-monetizing-agent-apps"
    },
    {
      "url": "https://scholarlykitchen.sspnet.org/2026/01/29/guest-post-ai-isnt-going-to-pay-for-content-part-two-the-path-forward/"
    },
    {
      "url": "https://www.leewayhertz.com/ai-agent-for-content-generation/"
    },
    {
      "url": "https://www.vellum.ai/blog/how-to-build-an-ai-agent-for-seo-research-and-content-generation"
    },
    {
      "url": "https://www.youtube.com/watch?v=-x9jEiQHWck"
    }
  ],
  "related": [
    {
      "name": "Getting Started with HTTP 402 Pay-Per-Crawl",
      "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
    },
    {
      "name": "Crawler Tokens & Auto-Pay: A How-To",
      "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html"
    },
    {
      "name": "Reading .data.json: A Field Guide for Agents",
      "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
    },
    {
      "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers",
      "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html"
    }
  ],
  "pricing": {
    "price_usd": 0,
    "method": "402",
    "endpoint": "https://esa.aisa.one/api/v1/access/verify",
    "autopay_hint": "set crawlerAutoPrice=true with X-AISA-Crawler-Token",
    "onboarding": "https://esa.aisa.one/cdn/guide.html"
  },
  "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)"
}