/inno-idea-eval

name: inno-idea-eval
description: >
  Multi-persona idea evaluation with quality gate. Evaluates ideas across
  5 InnoEval dimensions (Clarity, Novelty, Validity, Feasibility, Significance)
  using 3 reviewer personas and a meta-review. Sits between inno-idea-generation
  and inno-code-survey in the Idea branch. Use after inno-idea-generation.

Inno Idea Eval

Directory structure

skills/inno-idea-eval/
├── SKILL.md                                    ← this file
├── prompts/
│   ├── build_eval_query.md                     ← Per-persona evaluation query (all 5 dims)
│   ├── build_evidence_assembly.md              ← How to compose evidence from pipeline artifacts
│   ├── build_meta_review_query.md              ← Area-chair aggregation of 3 persona reviews
│   ├── build_novelty_queries.md                ← Query extraction for novelty verification (Step 0.5a)
│   ├── build_novelty_analysis.md               ← Similarity analysis for novelty verification (Step 0.5c)
│   └── build_refinement_feedback_query.md      ← Structured feedback for refinement loop
└── references/
    ├── eval_agent_instructions.md              ← Full eval agent system prompt + scoring rubrics
    ├── novelty_verification_config.md          ← Novelty search config, threat levels, fast-fail protocol
    └── reviewer_personas.md                    ← 3 persona definitions + evidence filter logic

> **How to use the resource files**: Each prompt template in `prompts/` documents > the exact parameters, the full text template, and usage notes (when it is a new > conversation vs. appended message, how to format evidence blocks, etc.). > The `references/` directory contains the Eval Agent's complete system instructions > including its scoring rubrics, persona definitions, and evidence filter logic. > Consult these files for the authoritative details; the steps below provide a summary.

Inputs

Paths for `Ideation/ideas` and `Ideation/references` come from **`instance.json`** (`instance.Ideation.ideas`, `instance.Ideation.references`). They are **absolute** in Dr. Claw-created projects; use as-is. If relative, resolve with `path.join(project_path, value)`.

| Parameter | Required | Description | |----------------------|----------|-------------| | `selected_idea` | Yes | The idea to evaluate, read from `Ideation/ideas/selected_idea.txt` | | `references` | No\* | Pre-formatted list of the source papers (`source_papers` in `Ideation/references/logs/load_instance.json`, from inno-prepare-resources) | | `prepare_res` | No\* | Selected repositories and reasoning (`prepare_result` in `Ideation/references/logs/prepare_agent.json`, from inno-prepare-resources) | | `download_res` | No\* | arXiv download results (`Ideation/references/logs/download_arxiv_source_by_title.json`) | | `data_module` | No\* | Task description, exposed below as `data_module.TASK`: `task_instructions` in `load_instance.json`, or the task contract in `Ideation/ideas/idea_context.md`, or `sections.ideation` of the research brief | | `context_variables` | Yes | Shared context dictionary (must contain `final_selected_idea_data`) |

\*Standalone mode: only `selected_idea` required; evaluation proceeds ungrounded with a noted limitation.

Outputs

| Output | Description | |------------------------------------------------|-------------| | `eval_report` | Full markdown evaluation report (meta-review) | | `eval_scores` | Structured JSON: per-dimension, per-persona, aggregated | | `eval_decision` | One of: `strong_accept` / `accept` / `borderline_accept` / `borderline_reject` / `reject` | | `eval_feedback` | Strengths/weaknesses/suggestions (for refinement or downstream) | | `context_variables["idea_evaluation_result"]` | Complete structured result dict |

Cache file outputs

Each step produces **two kinds** of files:

1. **`.txt` files** (primary) -- the full markdown content of each review, written directly to `Ideation/ideas/` 2. **`.json` files** (derived) -- structured metadata under `Ideation/ideas/logs/`, whose text fields **must be copied verbatim** from the corresponding `.txt` files (never summarized)

Full directory layout

Ideation/ideas/
├── novelty_grounding_report.txt                ← Step 0.5: Active Novelty Verification report
├── eval_report.txt                             ← Step 4: full meta-review report (markdown)
├── eval_persona_1_review.txt                   ← Step 1: Senior ML Researcher review
├── eval_persona_2_review.txt                   ← Step 2: Domain Expert review
├── eval_persona_3_review.txt                   ← Step 3: Methods Specialist review
└── logs/
    ├── idea_eval_agent_novelty.json            ← Step 0.5: Novelty search + analysis structured data
    ├── idea_eval_agent_persona_1.json          ← Step 1: Persona 1 structured scores
    ├── idea_eval_agent_persona_2.json          ← Step 2: Persona 2 structured scores
    ├── idea_eval_agent_persona_3.json          ← Step 3: Persona 3 structured scores
    └── idea_eval_agent_meta_review.json        ← Step 4: Aggregated decision + full report

Write order (critical)

For every step, **always write the `.txt` file first**, then build the `.json` file by copying the `.txt` content into the appropriate field:

For the novelty verification step: 1. Write `novelty_grounding_report.txt` with the full novelty analysis report 2. Copy that full text into `report_text` 3. Write `logs/idea_eval_agent_novelty.json`

For each persona review: 1. Write `eval_persona_{N}_review.txt` with the agent's full review 2. Read it back (or keep in memory) and embed the full text into `review_text` 3. Write the corresponding `logs/idea_eval_agent_persona_{N}.json`

For the meta-review step: 1. Write `eval_report.txt` with the agent