/scholar-evaluation

name: scholar-evaluation
description: Systematically evaluate scholarly work using the ScholarEval framework, providing structured assessment across research quality dimensions including problem formulation, methodology, analysis, and writing with quantitative scoring and actionable feedback.
license: MIT license
metadata:
    skill-author: K-Dense Inc.

Scholar Evaluation

Overview

Apply the ScholarEval framework to systematically evaluate scholarly and research work. This skill provides structured evaluation methodology based on peer-reviewed research assessment criteria, enabling comprehensive analysis of academic papers, research proposals, literature reviews, and scholarly writing across multiple quality dimensions.

When to Use This Skill

Use this skill when:

• Evaluating research papers for quality and rigor
• Assessing literature review comprehensiveness and quality
• Reviewing research methodology design
• Scoring data analysis approaches
• Evaluating scholarly writing and presentation
• Providing structured feedback on academic work
• Benchmarking research quality against established criteria
• Assessing publication readiness for target venues
• Providing quantitative evaluation to complement qualitative peer review

Visual Enhancement with Scientific Schematics

**When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.**

If your document does not already contain schematics or diagrams:

• Use the **scientific-schematics** skill to generate AI-powered publication-quality diagrams
• Simply describe your desired diagram in natural language
• Nano Banana Pro will automatically generate, review, and refine the schematic

**For new documents:** Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text.

**How to generate schematics:**

python scripts/generate_schematic.py "your diagram description" -o figures/output.png

The AI will automatically:

• Create publication-quality images with proper formatting
• Review and refine through multiple iterations
• Ensure accessibility (colorblind-friendly, high contrast)
• Save outputs in the figures/ directory

**When to add schematics:**

• Evaluation framework diagrams
• Quality assessment criteria decision trees
• Scholarly workflow visualizations
• Assessment methodology flowcharts
• Scoring rubric visualizations
• Evaluation process diagrams
• Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

---

Evaluation Workflow

Step 1: Initial Assessment and Scope Definition

Begin by identifying the type of scholarly work being evaluated and the evaluation scope:

**Work Types:**

• Full research paper (empirical, theoretical, or review)
• Research proposal or protocol
• Literature review (systematic, narrative, or scoping)
• Thesis or dissertation chapter
• Conference abstract or short paper

**Evaluation Scope:**

• Comprehensive (all dimensions)
• Targeted (specific aspects like methodology or writing)
• Comparative (benchmarking against other work)

Ask the user to clarify if the scope is ambiguous.

Step 2: Dimension-Based Evaluation

Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.

Refer to `references/evaluation_framework.md` for detailed criteria and rubrics for each dimension.

**Core Evaluation Dimensions:**

1. **Problem Formulation & Research Questions**

• Clarity and specificity of research questions
• Theoretical or practical significance
• Feasibility and scope appropriateness
• Novelty and contribution potential

2. **Literature Review**

• Comprehensiveness of coverage
• Critical synthesis vs. mere summarization
• Identification of research gaps
• Currency and relevance of sources
• Proper contextualization

3. **Methodology & Research Design**

• Appropriateness for research questions
• Rigor and validity
• Reproducibility and transparency
• Ethical considerations
• Limitations acknowledgment

4. **Data Collection & Sources**

• Quality and appropriateness of data
• Sample size and representativeness
• Data collection procedures
• Source credibility and reliability

5. **Analysis & Interpretation**

• Appropriateness of analytical methods
• Rigor of analysis
• Logical coherence
• Alternative explanations considered
• Results-claims alignment

6. **Results & Findings**

• Clarity of presentation
• Statistical or qualitative rigor
• Visualization quality
• Interpretation accuracy
• Implications discussion

7. **Scholarly Writing & Presentation**

• Clarity and organization
• Academic tone and style
• Grammar and mechanics
• Logical flow
• Accessibility to target audience

8. **Citations & References**

• Citation completeness
• Source quality and appropriateness
• Citation accuracy
• Balance of perspectives
• Adherence to citation standards

Step 3: Scoring and Rating

For each evaluated dimension, provide:

**Qualitative Assessment:**

• Key strengths (2-3 specific points)
• Areas for improvement (2-3 specific points)
• Critical issues (if any)

**Quantitative Scoring (Optional):** Use a 5-point scale where applicable:

• 5: Excellent - Exemplary quality, publishable in top venues
• 4: Good - Strong quality with minor improvements needed
• 3: Adequate - Acceptable quality with notable areas for improvement
• 2: Needs Improvement - Significant revisions required
• 1: Poor - Fundamental issues requiring major revision

To calculate aggregate scores programmatically, use `scripts/calculate_scores.py`.

Step 4: Synthesize Overall Assessment

Provide an integrated evaluati