Research Pipeline

Orchestrates the AgentSociety research workflow. Determines which skill to invoke based on the current workspace state and user intent.

Overview

The research pipeline is a revision-capable workflow: literature search → hypothesis → experiment config → run → analysis → paper, with explicit reroutes back to the smallest sufficient earlier stage. After a draft exists, paper-review can perform an optional robust author-side mock review using three parallel independent reviewers and a separate evidence-verifying MetaReview agent, then advise which earlier stage to revisit. The review remains read-only; the user or controlling coding agent closes the feedback loop by explicitly applying an accepted route through research-pipeline reroute. Supporting skills (scan-modules, create-agent, create-env-module, web-research, datasets) branch off the main trunk at specific points.

Scale Planning Gate

Before entering experiment-config, create-agent, or create-env-module, confirm the simulation scale budget:

• target agent count or range
• expected step budget
• acceptable runtime or compute budget
• preferred complexity tier, such as lean, balanced, or rich

If any of these are missing, ask one round of clarifying questions first. Present 2-3 approaches with trade-offs and a recommendation, then route into the appropriate skill once the budget is fixed.

Dataset Gate

If the task may depend on external data, treat dataset access as a first-class branch rather than an afterthought.

• Use agentsociety-use-dataset to search, inspect, and download datasets from the platform.
• If no suitable dataset exists locally or remotely, surface that gap before continuing with experiment design.
• Use agentsociety-create-dataset when the work needs packaging, validation, upload, or publishing of a dataset.
• If local files should be shared or reused, guide the user through dataset upload rather than folding the files into experiment config by hand.

Quick Reference

At the beginning of a new session or resumption, run:

bash
$PYTHON_PATH .agentsociety/bin/ags.py research-pipeline where-am-i --json

This reads .agentsociety/progress.json to determine the current stage. If the file does not exist, fall back to file-existence detection and then bootstrap tracking with init.

Git Checkpoint Discipline

Every pipeline transition must be persisted with an explicit git commit. Hook-based auto-commit approaches are unreliable — always commit manually.

Workspace Initialization

When bootstrapping a new workspace with research-pipeline init:

bash
# 1. Init the workspace directory if not already a git repo
git init
# 2. Create progress tracking
$PYTHON_PATH .agentsociety/bin/ags.py research-pipeline init --topic "TOPIC"
# 3. Initial commit
git add -A && git commit -m "init: bootstrap research pipeline"

Stage Transitions

After every call to research-pipeline update-stage, commit the current iteration's changes:

bash
$PYTHON_PATH .agentsociety/bin/ags.py research-pipeline update-stage STAGE STATUS
git add -A && git commit -m "pipeline: STAGE → STATUS"

For example:

bash
$PYTHON_PATH .agentsociety/bin/ags.py research-pipeline update-stage literature_search completed
git add -A && git commit -m "pipeline: literature_search → completed"

Apply an accepted review or analysis reroute with the same checkpoint discipline:

bash
$PYTHON_PATH .agentsociety/bin/ags.py research-pipeline reroute experiment_config \
  --reason "MetaReview found that the control cannot identify the claimed effect" \
  --source-artifact paper/reviews/acl-arr-review-r1.md
git add -A && git commit -m "pipeline: reroute to experiment_config"

Rules

• If git is not initialized in the workspace, run git init first.
• Commit messages follow the pattern: pipeline: STAGE → STATUS.
• Use git add -A to capture all changes produced by the current stage.
• Do not rely on git hooks (pre-commit, post-commit, etc.) for this — they are unreliable in this context. Commit explicitly.

Entry Conditions

Use where-am-i --json whenever the current stage is unclear.

• If .agentsociety/progress.json exists, trust current_stage as the primary routing signal.
• If the file is missing, infer the stage from workspace artifacts, then initialize progress tracking.
• If the current stage already has a failed status, route to the owning skill for repair rather than restarting the pipeline.
• If the current stage has needs_revision, route to the owning skill with the active reroute reason and source artifact as required context.
• If the work depends on simulation size or runtime budget and those values are missing, resolve the scale planning gate before routing onward.

| current_stage | Route to Skill | |-----------------|----------------| | literature_search | literature-search | | hypothesis | hypothesis | | experiment_config | experiment-config | | run_experiment | run-experiment | | analysis | analysis | | generate_paper | paper-toolkit |

Post-paper review is routed by user intent rather than current_stage: when a complete draft exists and the user asks for an internal review, venue score, or revision-stage advice, invoke agentsociety-paper-review. For PDF input, prepare one frozen page-aware intake and pass the same text, layout, and page renders to every agent; stop when its quality gate fails. A normal result requires three isolated reviewer subagents running concurrently followed by a separately spawned MetaReview subagent. The review bundle is advisory and must not update .agentsociety/progress.json; only the user or controlling coding agent may subsequently accept a stage-valued reroute and call research-pipeline reroute.

Progress File Quick Reference

| Command | Purpose | |---------|---------| | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline init --topic "TEXT" | Bootstrap progress.json (also git init + initial commit if needed) | | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline status | Show progress summary | | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline where-am-i --json | Get current stage as JSON | | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline update-stage STAGE STATUS | Update stage status (then git add -A && git commit) | | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline reroute STAGE --reason "TEXT" --source-artifact PATH | Start an auditable revision round at the current or an earlier stage | | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline set-verification STAGE STATUS | Update stage verification status | | $PYTHON_PATH .agentsociety/bin/ags.py research-pipeline next-action --json | Get the recommended next action |

Workflow

dot
digraph research_pipeline {
    rankdir=LR;
    node [shape=box, style=filled, fillcolor="#E8F4FD"];
    decide [label="where-am-i"];
    lit [label="literature-search"];
    hypo [label="hypothesis"];
    scan [label="scan-modules\noptional helper"];
    exp [label="experiment-config"];
    run [label="run-experiment"];
    analysis [label="analysis"];
    paper [label="paper-toolkit"];
    review [label="paper-review\n3 reviewers + MetaReview"];
    agent [label="create-agent"];
    env [label="create-env-module"];
    create_ds [label="create-dataset"];
    use_ds [label="use-dataset"];
    web [label="web-research"];

    decide -> lit;
    decide -> hypo;
    decide -> exp;
    decide -> run;
    decide -> analysis;
    decide -> paper;

    lit -> hypo;
    lit -> web [style=dashed, label="supplementary context"];
    hypo -> scan [style=dashed, label="names uncertain"];
    scan -> hypo;
    hypo -> exp;
    exp -> scan [style=dashed, label="need discovery or validation"];
    scan -> exp;
    exp -> agent [style=dashed, label="missing agent"];
    agent -> exp;
    exp -> env [style=dashed, label="missing env"];
    env -> exp;
    exp -> create_ds [style=dashed, label="publish dataset"];
    exp -> use_ds [style=dashed, label="need external data"];
    use_ds -> exp;
    exp -> run;
    run -> analysis;
    analysis -> paper;
    paper -> review [style=dashed, label="internal mock review"];
    review -> lit [style=dashed, label="accepted reroute"];
    review -> hypo [style=dashed, label="accepted reroute"];
    review -> exp [style=dashed, label="accepted reroute"];
    review -> run [style=dashed, label="accepted reroute"];
    review -> analysis [style=dashed, label="accepted reroute"];
    review -> paper [style=dashed, label="accepted reroute"];
    analysis -> hypo [style=dashed, label="revise hypothesis"];
    analysis -> use_ds [style=dashed, label="comparison data"];
    analysis -> web [style=dashed, label="supplementary context"];
}

Pipeline Map

| # | Skill | Produces | Consumes | |---|-------|----------|----------| | 1 | literature-search | papers/, papers/literature_index.json | TOPIC.md | | 2 | hypothesis | hypothesis_{id}/HYPOTHESIS.md, SIM_SETTINGS.json | literature index | | 3 | experiment-config | init_config.json, steps.yaml, config_params.py | SIM_SETTINGS.json | | 4 | run-experiment | run/replay/_schema.json, sharded replay JSONL, run/output.log, run/artifacts/ | init_config.json + steps.yaml | | 5 | analysis | presentation/hypothesis_{id}/report.md, charts, data | run/replay/ | | 6 | paper-toolkit | paper deliverables | analysis report + literature index | | 7 (optional) | paper-review | frozen input manifest + 3 independent reviews under paper/reviews/<venue>-review-r<N>/; final paper/reviews/<venue>-review-r<N>.md MetaReview | complete paper draft + selected evidence artifacts |

Branch Skills (called from trunk)

| Branch Skill | Called By | When | |-------------|-----------|------| | scan-modules | hypothesis, experiment-config | When module names are unknown or need validation | | create-agent | experiment-config | When needed agent class doesn't exist | | create-env-module | experiment-config | When needed env module doesn't exist | | web-research | literature-search, hypothesis, analysis | When supplementary non-academic context needed | | create-dataset | experiment-config | When packaging data for upload or publishing | | use-dataset | literature-search, hypothesis, experiment-config, analysis | When searching, downloading, or inspecting datasets |

Skill Index

| Skill | Trigger Keywords | |-------|-----------------| | literature-search | "literature", "papers", "related work", "background research" | | hypothesis | "hypothesis", "research question", "control", "treatment", "experiment groups" | | experiment-config | "configure experiment", "init_config", "steps.yaml", "set up experiment" | | run-experiment | "run experiment", "start simulation", "check status", "stop experiment" | | analysis | "analyze", "results", "visualization", "chart", "report" | | paper-toolkit | "write paper", "academic paper", "generate paper", "Nature paper" | | paper-review | "review paper", "paper score", "mock review", "review feedback", "where should revision go" | | scan-modules | "available modules", "list agents", "find environment" | | create-agent | "create agent", "custom agent", "new agent type" | | create-env-module | "create environment", "custom module", "env module" | | web-research | "web search", "current events", "recent developments" | | create-dataset | "create dataset", "upload dataset", "publish data" | | use-dataset | "download dataset", "find data", "browse datasets", "search datasets", "dataset search" |

Iterative Cycles

• Analysis → Hypothesis: results may refine or revise the hypothesis
• Experiment-config → Create-agent/Create-env-module: missing modules trigger creation
• Experiment-config → Scan-modules: names can be confirmed when discovery is needed
• Run → Config: failed validation or execution loops back to config fixes
• Paper → Review → Earlier stage: paper-review emits advisory reroutes; the user or controlling coding agent decides whether to apply one with research-pipeline reroute

reroute is the only supported backward transition. It:

1. rejects forward jumps;
2. increments workspace.revision_round;
3. changes the accepted target and already-completed downstream stages to needs_revision without deleting artifacts;
4. resets stale verification and stage metadata for those stages;
5. records the reason, source artifact, previous stage-state snapshots, and invalidated stages in the append-only transitions log;
6. keeps active_reroute until all invalidated stages are completed or explicitly skipped.

Do not apply human_decision or none as stages. Pause for accountable input on human_decision; leave pipeline state unchanged for none. When several review routes exist, apply the earliest accepted stage once—the normal forward flow will revisit its invalidated downstream stages.

Persistence Files

| File | Git | Purpose | |------|-----|---------| | .agentsociety/progress.json | Committable | Stage tracker with status, revision round, active reroute, attempts, and append-only transition history |

Hard Constraints

• Never run analysis before run-experiment produces run/replay/_schema.json
• Always run experiment-config check before run-experiment
• paper-toolkit requires analysis reports to exist
• paper-review requires a complete draft, a passing shared PDF intake when applicable, and a valid 3-reviewer ensemble before MetaReview; it writes only the reserved review bundle and never updates pipeline stage or executes reroute advice
• Apply backward movement only through research-pipeline reroute; do not edit current_stage or downstream statuses by hand
• Keep the controlling agent as the single writer of progress.json; subagents return artifacts and must not perform pipeline transitions
• Always call research-pipeline update-stage after completing a pipeline stage
• Always git commit after research-pipeline update-stage — see Git Checkpoint Discipline
• On workspace init, run git init (if needed) followed by an initial commit
• Do not rely on git hooks for auto-commit; commit explicitly after every stage transition