peer-review-engine

Orchestration Log: When this skill is activated, append a log entry to outputs/orchestration_log.md:

```

### Skill Activation: Peer Review Engine

Timestamp: [current date/time]

Actor: AI Agent (peer-review-engine)

Input: [paper source: draft.md / paper.tex, word count, number of sections]

Output: 2 simulated reviewer reports saved to simulated_reviews.md

Recommendation distribution: [R1: recommendation, R2: recommendation]

```

Peer Review Engine

Core Principle

The best time to discover weaknesses is before reviewers do. This engine generates two independent, simulated double-blind peer reviews that mimic the rigor, tone, and structure of top-tier IS/CS conference reviews (ICIS, ECIS, HICSS) and journal reviews (MISQ, ISR, EJIS, BISE). Each reviewer has a distinct persona and evaluation focus, producing complementary perspectives on the manuscript.

The reviews are actionable, not performative. Every weakness includes a concrete suggestion for improvement. Every strength is specific enough to preserve during revision. The output format is designed to feed directly into /respond-reviewers, creating a pre-submission quality loop: write -> self-review -> revise -> submit.

When to Activate

• User says "review my paper", "simulate a peer review", "give me reviewer feedback"
• User says "review paper", "pre-submission check", "what would reviewers say?"
• After Phase 6 (LaTeX export) when the user wants quality assurance before sharing
• Before sending a draft to co-authors
• When the user wants to identify weaknesses before formal submission
• User runs /review-paper

Prerequisites

• draft.md or latex/paper.tex exists (at least one)
• Paper has identifiable sections (Introduction, Background, Method, Results/Findings, Discussion)
• If paper.tex exists and is more recent than draft.md, prefer paper.tex

Step 1: LOCATE & READ the Paper

Find the Manuscript

Check for paper sources in this order of preference:

1. latex/paper.tex — if it exists and compiles, this is the most complete version
2. draft.md — the markdown draft from the production phase
3. User-specified path — if the user provides a specific file path

If both exist, compare timestamps. Use the more recent one, but note which version was reviewed.

Extract Paper Metadata

Read the full paper and extract:

1. Title (from \title{} or first # heading)
2. Abstract (from \begin{abstract} or the abstract section)
3. Section structure — list all sections and subsections with approximate word counts
4. Research questions — extract explicit RQs (look for "RQ1:", "Research Question", etc.)
5. Method — what methodology is described (SLR, case study, DSR, survey, experiment, etc.)
6. Theoretical lens — which theories/frameworks are applied
7. Key contribution claims — what does the paper claim to contribute
8. Number of references — count \citep/\citet or (Author, Year) patterns
9. Figures and tables — count and list captions
10. Total word count — approximate

Identify Target Venue

If the paper mentions a target venue (in metadata, framing.md, or paper_structure.md), use that venue's specific review criteria. Otherwise, default to ICIS/ECIS-level expectations for IS papers, or top-tier CS conference standards for CS papers.

Step 2: ASSESS Against Review Dimensions

Before writing individual reviews, perform a systematic assessment across all evaluation dimensions. This ensures both reviewers draw from a consistent quality analysis while emphasizing different aspects.

Evaluation Dimensions

| Dimension | Weight | What to Check | |-----------|--------|---------------| | 1. Contribution | High | Is the contribution clearly stated? Is it novel? Does it advance theory or practice? Is the gap well-motivated? | | 2. Theoretical Foundation | High | Is the theory well-chosen? Is it properly applied (not just cited)? Are constructs operationalized? | | 3. Research Design & Rigor | High | Is the method appropriate for the RQs? Are threats to validity discussed? Is the approach replicable? | | 4. Literature Coverage | Medium | Is the related work comprehensive? Are key papers cited? Is the positioning accurate? | | 5. Argumentation & Logic | Medium | Does the paper flow logically? Are claims supported by evidence? Are there logical gaps? | | 6. Writing Quality | Medium | Is the paper clearly written? Is it within page limits? Are figures/tables well-designed? | | 7. Practical Relevance | Low-Med | Are practical implications specific and actionable? Would practitioners find this useful? |

Quality Signals to Detect

Positive signals:

• Clear gap-to-contribution alignment
• Theory used as an analytical lens (not just background decoration)
• Method follows established protocols (e.g., Kitchenholz & Charters for SLR, Hevner for DSR)
• Discussion goes beyond restating results
• Limitations are honest and specific

Red flags:

• Gap is asserted without evidence ("surprisingly, no study has...")
• Theory is mentioned but not applied in analysis
• Method lacks detail (sampling unclear, coding process vague)
• Results section is purely descriptive without interpretation
• Discussion is generic ("future research should explore...")
• [CITE], [DATA], [TODO] placeholders remain
• Contribution claims are too broad for the evidence presented
• Mismatch between RQs and what Results actually address

For Each Dimension

Assign an internal rating:

• Strong — meets or exceeds top-venue expectations
• Adequate — acceptable but could be strengthened
• Weak — significant improvement needed
• Missing — not addressed at all

These ratings are for your internal use to calibrate the reviews. They do NOT appear in the reviewer output (real reviewers don't use rubrics this explicitly).

Step 3: GENERATE Reviewer 1 — "The Methodologist"

Reviewer 1 Persona

Profile: Senior IS/CS researcher, 15+ years experience. Associate Editor at a top journal. Known for methodological rigor. Has published extensively on research methods in IS. Reviews 15-20 papers per year for ICIS, ECIS, MISQ.

Evaluation focus:

• Research design quality and appropriateness
• Validity and reliability of findings
• Replicability of the approach
• Proper application of method templates/protocols
• Sample adequacy, data quality, analytical rigor
• Threats to validity (internal, external, construct, conclusion)
• Alignment between RQs and method

Tone: Constructive but demanding. Points out methodological gaps precisely. Acknowledges methodological strengths explicitly. Uses phrases like "The authors should clarify...", "It is unclear how...", "A stronger approach would be to..."

Bias tendencies (realistic):

• Favors quantitative evidence and structured protocols
• Skeptical of purely conceptual papers without empirical grounding
• Looks for explicit limitation sections
• Values transparency in research process documentation

Review Structure for Reviewer 1

Generate the review in this exact format:

## Reviewer 1

### Summary
[3-5 sentences summarizing the paper's objective, approach, and main findings.
Write as if the reviewer genuinely read the paper. Reference specific sections.]

### Strengths
[3-5 bullet points. Each strength must be SPECIFIC — reference a particular
section, argument, or methodological choice. Not generic praise.]

- **S1:** [Specific strength with section reference]
- **S2:** [Specific strength with section reference]
- **S3:** [Specific strength with section reference]

### Weaknesses
[3-5 bullet points. Each weakness must be ACTIONABLE — include what should be
done to address it. Focus on methodological issues.]

- **W1:** [Specific weakness + concrete suggestion for improvement]
- **W2:** [Specific weakness + concrete suggestion for improvement]
- **W3:** [Specific weaknes