Using AgentFlare Research and Scholar Data

This guide is for developers and agent builders who want to pull structured research briefs and scholar literature digests from agentflare.org into their own pipelines. You will learn which files are free and which are metered, what each field in the research and scholar .data.json files holds, how the per-request price of 0.02 applies, and how to fetch and parse a file with curl and Python using the crawler identity header. It assumes basic command-line and JSON familiarity.

How the flow works

Before you start

• A crawler token, shown here as YOUR_CRAWLER_TOKEN
• curl and Python 3 installed
• Basic familiarity with JSON

Step by step

1. Separate free teasers from paid data

   AgentFlare keeps index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt free (tier L1_free). Files ending in .data.json or .full.md are tier L2_paid and are metered per crawl through the verify endpoint. Research and scholar both match a per-request rule at a price of 0.02 USD. Use the free pages to browse, and request .data.json only when you want the full dataset.

   Free: /research/index.html, .md teasers, llms.txt
   Paid 0.02 per request: /research/<slug>.data.json
   Paid 0.02 per request: /scholar/<slug>.data.json
   Verify endpoint: https://esa.aisa.one/api/v1/access/verify

   S4 · agentflare.org

2. Fetch the research index

   Start with the research index file, which lists every brief. Send your token in X-Crawler-Token, which replaces the older X-AISA-Crawler-Token name still shown on some AISA pages. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer. Each request to a .data.json file is billed at the research rate of 0.02.

   curl -s https://agentflare.org/research/index.data.json \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -o research-index.json

   S1 · esa.aisa.oneS2 · agentflare.orgS4 · agentflare.org

3. Read the brief listing

   In the index file, data.count is the number of briefs and data.clusters lists the cluster names. Each entry in data.briefs has title, cluster, sources (a number) and url, and the url points at that brief's own .data.json file. The top level also carries title, description, canonical, category, updated, generated_at, analysis_md, sources, related and pricing.

   import json
   
   doc = json.load(open('research-index.json'))
   print(doc['data']['count'], doc['data']['clusters'])
   for b in doc['data']['briefs']:
       print(b['cluster'], b['title'], b['sources'], b['url'])

   S2 · agentflare.org

4. Parse a single research brief

   A brief such as tool-use-reliability-in-llm-agents.data.json holds its content under data. Fields include topic, cluster, tldr, intro, concepts (term, definition, sources), how_it_works with steps, sections, a comparison table, key_numbers, timeline, pitfalls, takeaways, faq and data.sources. The top-level facts array holds label and value pairs.

   import json, urllib.request
   
   url = 'https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json'
   req = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})
   doc = json.load(urllib.request.urlopen(req))
   d = doc['data']
   print(d['topic'], d['cluster'])
   print(d['tldr'])
   for c in d['concepts']:
       print(c['term'], c['definition'])
   for n in d['key_numbers']:
       print(n['value'], n['label'], n['source'])
   print(doc['pricing']['price_usd'])

   S1 · esa.aisa.oneS2 · agentflare.org

5. Parse a scholar digest

   Scholar files cost the same 0.02 per request. The scholar index lists digests with title, headline, papers, citations, monthly_arxiv and url. A digest file such as web-agents-navigation-benchmarks-research-digest.data.json adds arxiv_query, papers (title, link, authors, citations, year, snippet, takeaway), recent, monthly_arxiv, themes, whats_new, trend_note, open_problems, faq and total_citations.

   import json, urllib.request
   
   url = 'https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json'
   req = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})
   doc = json.load(urllib.request.urlopen(req))
   d = doc['data']
   print(d['topic'], d['total_citations'])
   for p in d['papers']:
       print(p['year'], p['citations'], p['title'])
   for m in d['monthly_arxiv']:
       print(m['month'], m['count'])
   print(d['trend_note'])

   S1 · esa.aisa.oneS3 · agentflare.orgS4 · agentflare.org

Key fields in research and scholar files

Common pitfalls

• Send X-Crawler-Token, not the older X-AISA-Crawler-Token, and never put the token in Authorization: Bearer, which is reserved for the gateway layer.
• Index files and item files have different data shapes: the index has count plus briefs or digests, while item files hold the full content. Parse each accordingly.
• Every .data.json request is metered at 0.02 USD, so cache responses locally instead of re-fetching the same brief or digest.
• The facts array is typed differently by file: it is label and value objects in item files, but unspecified in index files. Check for entries before reading them.

Key takeaways

• Research and scholar .data.json files each cost 0.02 USD per request; teasers and index pages stay free.
• Authenticate with the X-Crawler-Token header, never Authorization: Bearer.
• Start from index.data.json to find item urls, then fetch the specific brief or digest file.

FAQ

How this page is made. Written from 4 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.