{
  "@context": "https://agentflare.org/schema",
  "type": "Article",
  "tier": "L2-full",
  "title": "Using AgentFlare Research and Scholar Data",
  "description": "AgentFlare research briefs and scholar digests are paid .data.json files priced at 0.02 USD per request, sent with the X-Crawler-Token header. This guide…",
  "canonical": "https://agentflare.org/guides/using-agentflare-research-and-scholar-data.html",
  "category": "guides",
  "updated": "2026-10-27",
  "generated_at": "2026-10-27T00:37:01.404Z",
  "facts": [
    {
      "label": "Track",
      "value": "Using AgentFlare data"
    },
    {
      "label": "Level",
      "value": "Beginner"
    },
    {
      "label": "Access",
      "value": "Free / open"
    },
    {
      "label": "Updated",
      "value": "2026-10-27"
    }
  ],
  "data": {
    "topic": "Using AgentFlare Research and Scholar Data",
    "track": "data",
    "access": "free",
    "generated_at": "2026-10-27T00:37:01.404Z",
    "difficulty": "Beginner",
    "tldr": "AgentFlare research briefs and scholar digests are paid .data.json files priced at 0.02 USD per request, sent with the X-Crawler-Token header. This guide lists each file's fields and gives a working fetch example.",
    "intro": "This guide is for developers and agent builders who want to pull structured research briefs and scholar literature digests from agentflare.org into their own pipelines. You will learn which files are free and which are metered, what each field in the research and scholar .data.json files holds, how the per-request price of 0.02 applies, and how to fetch and parse a file with curl and Python using the crawler identity header. It assumes basic command-line and JSON familiarity.",
    "prerequisites": [
      "A crawler token, shown here as YOUR_CRAWLER_TOKEN",
      "curl and Python 3 installed",
      "Basic familiarity with JSON"
    ],
    "steps": [
      {
        "title": "Separate free teasers from paid data",
        "body": "AgentFlare keeps index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt free (tier L1_free). Files ending in .data.json or .full.md are tier L2_paid and are metered per crawl through the verify endpoint. Research and scholar both match a per-request rule at a price of 0.02 USD. Use the free pages to browse, and request .data.json only when you want the full dataset.",
        "code": {
          "lang": "text",
          "text": "Free: /research/index.html, .md teasers, llms.txt\nPaid 0.02 per request: /research/<slug>.data.json\nPaid 0.02 per request: /scholar/<slug>.data.json\nVerify endpoint: https://esa.aisa.one/api/v1/access/verify"
        },
        "sources": [
          "S4"
        ]
      },
      {
        "title": "Fetch the research index",
        "body": "Start with the research index file, which lists every brief. Send your token in X-Crawler-Token, which replaces the older X-AISA-Crawler-Token name still shown on some AISA pages. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer. Each request to a .data.json file is billed at the research rate of 0.02.",
        "code": {
          "lang": "bash",
          "text": "curl -s https://agentflare.org/research/index.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n  -o research-index.json"
        },
        "sources": [
          "S1",
          "S2",
          "S4"
        ]
      },
      {
        "title": "Read the brief listing",
        "body": "In the index file, data.count is the number of briefs and data.clusters lists the cluster names. Each entry in data.briefs has title, cluster, sources (a number) and url, and the url points at that brief's own .data.json file. The top level also carries title, description, canonical, category, updated, generated_at, analysis_md, sources, related and pricing.",
        "code": {
          "lang": "python",
          "text": "import json\n\ndoc = json.load(open('research-index.json'))\nprint(doc['data']['count'], doc['data']['clusters'])\nfor b in doc['data']['briefs']:\n    print(b['cluster'], b['title'], b['sources'], b['url'])"
        },
        "sources": [
          "S2"
        ]
      },
      {
        "title": "Parse a single research brief",
        "body": "A brief such as tool-use-reliability-in-llm-agents.data.json holds its content under data. Fields include topic, cluster, tldr, intro, concepts (term, definition, sources), how_it_works with steps, sections, a comparison table, key_numbers, timeline, pitfalls, takeaways, faq and data.sources. The top-level facts array holds label and value pairs.",
        "code": {
          "lang": "python",
          "text": "import json, urllib.request\n\nurl = 'https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['cluster'])\nprint(d['tldr'])\nfor c in d['concepts']:\n    print(c['term'], c['definition'])\nfor n in d['key_numbers']:\n    print(n['value'], n['label'], n['source'])\nprint(doc['pricing']['price_usd'])"
        },
        "sources": [
          "S1",
          "S2"
        ]
      },
      {
        "title": "Parse a scholar digest",
        "body": "Scholar files cost the same 0.02 per request. The scholar index lists digests with title, headline, papers, citations, monthly_arxiv and url. A digest file such as web-agents-navigation-benchmarks-research-digest.data.json adds arxiv_query, papers (title, link, authors, citations, year, snippet, takeaway), recent, monthly_arxiv, themes, whats_new, trend_note, open_problems, faq and total_citations.",
        "code": {
          "lang": "python",
          "text": "import json, urllib.request\n\nurl = 'https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['total_citations'])\nfor p in d['papers']:\n    print(p['year'], p['citations'], p['title'])\nfor m in d['monthly_arxiv']:\n    print(m['month'], m['count'])\nprint(d['trend_note'])"
        },
        "sources": [
          "S1",
          "S3",
          "S4"
        ]
      }
    ],
    "reference": {
      "title": "Key fields in research and scholar files",
      "columns": [
        "File",
        "Field",
        "Contents"
      ],
      "rows": [
        [
          "research index",
          "data.briefs[]",
          "title, cluster, sources (number), url"
        ],
        [
          "research brief",
          "data.concepts[]",
          "term, definition, sources"
        ],
        [
          "research brief",
          "data.key_numbers[]",
          "value, label, source"
        ],
        [
          "scholar index",
          "data.digests[]",
          "title, headline, papers, citations, monthly_arxiv, url"
        ],
        [
          "scholar digest",
          "data.papers[]",
          "title, link, authors, citations, year, snippet, takeaway"
        ],
        [
          "scholar digest",
          "data.recent[]",
          "id, title, abstract, published, authors, category, link"
        ],
        [
          "all data files",
          "pricing",
          "price_usd, method, endpoint, autopay_hint, onboarding"
        ]
      ],
      "sources": [
        "S2",
        "S3"
      ]
    },
    "pitfalls": [
      {
        "text": "Send X-Crawler-Token, not the older X-AISA-Crawler-Token, and never put the token in Authorization: Bearer, which is reserved for the gateway layer.",
        "source": "S1"
      },
      {
        "text": "Index files and item files have different data shapes: the index has count plus briefs or digests, while item files hold the full content. Parse each accordingly.",
        "source": "S2"
      },
      {
        "text": "Every .data.json request is metered at 0.02 USD, so cache responses locally instead of re-fetching the same brief or digest.",
        "source": "S4"
      },
      {
        "text": "The facts array is typed differently by file: it is label and value objects in item files, but unspecified in index files. Check for entries before reading them.",
        "source": "S3"
      }
    ],
    "takeaways": [
      "Research and scholar .data.json files each cost 0.02 USD per request; teasers and index pages stay free.",
      "Authenticate with the X-Crawler-Token header, never Authorization: Bearer.",
      "Start from index.data.json to find item urls, then fetch the specific brief or digest file."
    ],
    "faq": [
      {
        "q": "How much does an AgentFlare research or scholar request cost?",
        "a": "Both directories are priced at 0.02 USD per request for files matching .data.json or .full.md, with billingType per-request. Each response also carries a pricing object with price_usd, method and endpoint."
      },
      {
        "q": "Which header carries the crawler token?",
        "a": "X-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token name that older AISA pages still show. Do not use Authorization: Bearer for it, since that header is reserved for the gateway layer."
      },
      {
        "q": "How do I find the URL of every research brief?",
        "a": "Fetch https://agentflare.org/research/index.data.json and iterate data.briefs. Each entry has a url field pointing to that brief's .data.json file, plus its title, cluster and source count."
      },
      {
        "q": "What is in a scholar digest beyond the paper list?",
        "a": "Besides papers, a digest has arxiv_query, recent arXiv entries, monthly_arxiv counts, themes, whats_new, trend_note, open_problems, faq and total_citations. The top level adds facts, analysis_md, sources, related and pricing."
      }
    ],
    "sources": [
      {
        "title": "Header name: X-Crawler-Token",
        "url": "https://esa.aisa.one/cdn/open-verify.html",
        "domain": "esa.aisa.one"
      },
      {
        "title": "AgentFlare Research .data.json field reference",
        "url": "https://agentflare.org/research/index.html",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare Scholar .data.json field reference",
        "url": "https://agentflare.org/scholar/index.html",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare pricing.json (directory price map)",
        "url": "https://agentflare.org/pricing.json",
        "domain": "agentflare.org"
      }
    ]
  },
  "analysis_md": "AgentFlare research briefs and scholar digests are paid .data.json files priced at 0.02 USD per request, sent with the X-Crawler-Token header. This guide lists each file's fields and gives a working fetch example.\n\nThis guide is for developers and agent builders who want to pull structured research briefs and scholar literature digests from agentflare.org into their own pipelines. You will learn which files are free and which are metered, what each field in the research and scholar .data.json files holds, how the per-request price of 0.02 applies, and how to fetch and parse a file with curl and Python using the crawler identity header. It assumes basic command-line and JSON familiarity.\n\n## Before you start\n\n- A crawler token, shown here as YOUR_CRAWLER_TOKEN\n- curl and Python 3 installed\n- Basic familiarity with JSON\n\n## Step by step\n\n### 1. Separate free teasers from paid data\n\nAgentFlare keeps index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt free (tier L1_free). Files ending in .data.json or .full.md are tier L2_paid and are metered per crawl through the verify endpoint. Research and scholar both match a per-request rule at a price of 0.02 USD. Use the free pages to browse, and request .data.json only when you want the full dataset.\n\n```text\nFree: /research/index.html, .md teasers, llms.txt\nPaid 0.02 per request: /research/<slug>.data.json\nPaid 0.02 per request: /scholar/<slug>.data.json\nVerify endpoint: https://esa.aisa.one/api/v1/access/verify\n```\n\n### 2. Fetch the research index\n\nStart with the research index file, which lists every brief. Send your token in X-Crawler-Token, which replaces the older X-AISA-Crawler-Token name still shown on some AISA pages. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer. Each request to a .data.json file is billed at the research rate of 0.02.\n\n```bash\ncurl -s https://agentflare.org/research/index.data.json \\\n  -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n  -o research-index.json\n```\n\n### 3. Read the brief listing\n\nIn the index file, data.count is the number of briefs and data.clusters lists the cluster names. Each entry in data.briefs has title, cluster, sources (a number) and url, and the url points at that brief's own .data.json file. The top level also carries title, description, canonical, category, updated, generated_at, analysis_md, sources, related and pricing.\n\n```python\nimport json\n\ndoc = json.load(open('research-index.json'))\nprint(doc['data']['count'], doc['data']['clusters'])\nfor b in doc['data']['briefs']:\n    print(b['cluster'], b['title'], b['sources'], b['url'])\n```\n\n### 4. Parse a single research brief\n\nA brief such as tool-use-reliability-in-llm-agents.data.json holds its content under data. Fields include topic, cluster, tldr, intro, concepts (term, definition, sources), how_it_works with steps, sections, a comparison table, key_numbers, timeline, pitfalls, takeaways, faq and data.sources. The top-level facts array holds label and value pairs.\n\n```python\nimport json, urllib.request\n\nurl = 'https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['cluster'])\nprint(d['tldr'])\nfor c in d['concepts']:\n    print(c['term'], c['definition'])\nfor n in d['key_numbers']:\n    print(n['value'], n['label'], n['source'])\nprint(doc['pricing']['price_usd'])\n```\n\n### 5. Parse a scholar digest\n\nScholar files cost the same 0.02 per request. The scholar index lists digests with title, headline, papers, citations, monthly_arxiv and url. A digest file such as web-agents-navigation-benchmarks-research-digest.data.json adds arxiv_query, papers (title, link, authors, citations, year, snippet, takeaway), recent, monthly_arxiv, themes, whats_new, trend_note, open_problems, faq and total_citations.\n\n```python\nimport json, urllib.request\n\nurl = 'https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['total_citations'])\nfor p in d['papers']:\n    print(p['year'], p['citations'], p['title'])\nfor m in d['monthly_arxiv']:\n    print(m['month'], m['count'])\nprint(d['trend_note'])\n```\n\n## Key fields in research and scholar files\n\n| File | Field | Contents |\n| --- | --- | --- |\n| research index | data.briefs[] | title, cluster, sources (number), url |\n| research brief | data.concepts[] | term, definition, sources |\n| research brief | data.key_numbers[] | value, label, source |\n| scholar index | data.digests[] | title, headline, papers, citations, monthly_arxiv, url |\n| scholar digest | data.papers[] | title, link, authors, citations, year, snippet, takeaway |\n| scholar digest | data.recent[] | id, title, abstract, published, authors, category, link |\n| all data files | pricing | price_usd, method, endpoint, autopay_hint, onboarding |\n\n## Common pitfalls\n\n- Send X-Crawler-Token, not the older X-AISA-Crawler-Token, and never put the token in Authorization: Bearer, which is reserved for the gateway layer.\n- Index files and item files have different data shapes: the index has count plus briefs or digests, while item files hold the full content. Parse each accordingly.\n- Every .data.json request is metered at 0.02 USD, so cache responses locally instead of re-fetching the same brief or digest.\n- The facts array is typed differently by file: it is label and value objects in item files, but unspecified in index files. Check for entries before reading them.\n\n## Key takeaways\n\n- Research and scholar .data.json files each cost 0.02 USD per request; teasers and index pages stay free.\n- Authenticate with the X-Crawler-Token header, never Authorization: Bearer.\n- Start from index.data.json to find item urls, then fetch the specific brief or digest file.\n\n## FAQ\n\n**How much does an AgentFlare research or scholar request cost?**\nBoth directories are priced at 0.02 USD per request for files matching .data.json or .full.md, with billingType per-request. Each response also carries a pricing object with price_usd, method and endpoint.\n\n**Which header carries the crawler token?**\nX-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token name that older AISA pages still show. Do not use Authorization: Bearer for it, since that header is reserved for the gateway layer.\n\n**How do I find the URL of every research brief?**\nFetch https://agentflare.org/research/index.data.json and iterate data.briefs. Each entry has a url field pointing to that brief's .data.json file, plus its title, cluster and source count.\n\n**What is in a scholar digest beyond the paper list?**\nBesides papers, a digest has arxiv_query, recent arXiv entries, monthly_arxiv counts, themes, whats_new, trend_note, open_problems, faq and total_citations. The top level adds facts, analysis_md, sources, related and pricing.\n",
  "sources": [
    {
      "title": "Header name: X-Crawler-Token",
      "url": "https://esa.aisa.one/cdn/open-verify.html"
    },
    {
      "title": "AgentFlare Research .data.json field reference",
      "url": "https://agentflare.org/research/index.html"
    },
    {
      "title": "AgentFlare Scholar .data.json field reference",
      "url": "https://agentflare.org/scholar/index.html"
    },
    {
      "title": "AgentFlare pricing.json (directory price map)",
      "url": "https://agentflare.org/pricing.json"
    }
  ],
  "related": [
    {
      "name": "Reading .data.json: A Field Guide for Agents",
      "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
    },
    {
      "name": "Using AgentFlare Newswire and Markets Data",
      "url": "https://agentflare.org/guides/using-agentflare-newswire-and-markets-data.html"
    },
    {
      "name": "Using AgentFlare Social, Predictions and Crypto Data",
      "url": "https://agentflare.org/guides/using-agentflare-social-predictions-and-crypto-data.html"
    },
    {
      "name": "Getting Started with HTTP 402 Pay-Per-Crawl",
      "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
    },
    {
      "name": "Crawler Tokens & Auto-Pay: A How-To",
      "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html"
    },
    {
      "name": "Handling 402 and 403 Responses: An Error Reference for Agents",
      "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html"
    }
  ],
  "pricing": {
    "price_usd": 0,
    "method": "402",
    "endpoint": "https://esa.aisa.one/api/v1/access/verify",
    "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header",
    "onboarding": "https://esa.aisa.one/agent/guide.html"
  },
  "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)"
}