/ref-hallucination-arena

name: ref-hallucination-arena
description: >
  Benchmark LLM reference recommendation capabilities by verifying every cited
  paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate,
  per-field accuracy (title/author/year/DOI), discipline breakdown, and year
  constraint compliance. Supports tool-augmented (ReAct + web search) mode.
  Use when the user asks to evaluate, benchmark, or compare models on academic
  reference hallucination, literature recommendation quality, or citation accuracy.

Reference Hallucination Arena Skill

Evaluate how accurately LLMs recommend real academic references using the OpenJudge `RefArenaPipeline`:

1. **Load queries** — from JSON/JSONL dataset 2. **Collect responses** — BibTeX-formatted references from target models 3. **Extract references** — parse BibTeX entries from model output 4. **Verify references** — cross-check against Crossref / PubMed / arXiv / DBLP 5. **Score & rank** — compute verification rate, per-field accuracy, discipline breakdown 6. **Generate report** — Markdown report + visualization charts

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for ref_hallucination_arena (chart generation)
pip install matplotlib

Gather from user before running

| Info | Required? | Notes | |------|-----------|-------| | Config YAML path | Yes | Defines endpoints, dataset, verification settings | | Dataset path | Yes | JSON/JSONL file with queries (can be set in config) | | API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. | | CrossRef email | No | Improves API rate limits for verification | | PubMed API key | No | Improves PubMed rate limits | | Output directory | No | Default: `./evaluation_results/ref_hallucination_arena` | | Report language | No | `"en"` (default) or `"zh"` | | Tavily API key | No | Required only if using tool-augmented mode |

Quick start

CLI

# Run evaluation with config file
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Resume from checkpoint (default behavior)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Start fresh, ignore checkpoint
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save

# Override output directory
python -m cookbooks.ref_hallucination_arena --config config.yaml \
  --output_dir ./my_results --save

Python API

import asyncio
from cookbooks.ref_hallucination_arena.pipeline import RefArenaPipeline

async def main():
    pipeline = RefArenaPipeline.from_config("config.yaml")
    result = await pipeline.evaluate()

    for rank, (model, score) in enumerate(result.rankings, 1):
        print(f"{rank}. {model}: {score:.1%}")

asyncio.run(main())

CLI options

| Flag | Default | Description | |------|---------|-------------| | `--config` | — | Path to YAML configuration file (required) | | `--output_dir` | config value | Override output directory | | `--save` | `False` | Save results to file | | `--fresh` | `False` | Start fresh, ignore checkpoint |

Minimal config file

task:
  description: "Evaluate LLM reference recommendation capabilities"

dataset:
  path: "./data/queries.json"

target_endpoints:
  model_a:
    base_url: "https://api.openai.com/v1"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

  model_b:
    base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
    api_key: "${DASHSCOPE_API_KEY}"
    model: "qwen3-max"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

Full config reference

task

| Field | Required | Description | |-------|----------|-------------| | `description` | Yes | Evaluation task description | | `scenario` | No | Usage scenario |

dataset

| Field | Default | Description | |-------|---------|-------------| | `path` | — | Path to JSON/JSONL dataset file (required) | | `shuffle` | `false` | Shuffle queries before evaluation | | `max_queries` | `null` | Max queries to use (`null` = all) |

target_endpoints.\<name\>

| Field | Default | Description | |-------|---------|-------------| | `base_url` | — | API base URL (required) | | `api_key` | — | API key, supports `${ENV_VAR}` (required) | | `model` | — | Model name (required) | | `system_prompt` | built-in | System prompt; use `{num_refs}` placeholder | | `max_concurrency` | `5` | Max concurrent requests for this endpoint | | `extra_params` | — | Extra API request params (e.g. `temperature`) | | `tool_config.enabled` | `false` | Enable ReAct agent with Tavily web search | | `tool_config.tavily_api_key` | env var | Tavily API key | | `tool_config.max_iterations` | `10` | Max ReAct iterations (1–30) | | `tool_config.search_depth` | `"advanced"` | `"basic"` or `"advanced"` |

verification

| Field | Default | Description | |-------|---------|-------------| | `crossref_mailto` | — | Email for Crossref polite pool | | `pubmed_api_key` | — | PubMed API key | | `max_workers` | `10` | Concurrent verification threads (1–50) | | `timeout` | `30` | Per-request timeout in seconds | | `verified_threshold` | `0.7` | Min composite score to count as VERIFIED |

evaluation

| Field | Default | Description | |-------|---------|-------------| | `timeout` | `120` | Model API request timeout in seconds | | `retry_times` | `3` | Number of retry attempts |

output

| Field | Default | Description | |-------|---------|-------------| | `output_dir` | `./evaluation_results/ref_hallucination_arena` | Output directory | | `save_queries` | `true` | Save loaded queries | | `save_responses` | `true` | Save model responses | | `save_details` | `true` | Save verification details |

###