Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →human-agent-society/coral-new-task
End-to-end recipe for adding a new task under `examples/` — the three pieces that have to line up (`task.yaml`, `seed/`, and `grader/`), what to put in each, the `TaskGrader` API surface, the `coral validate` → smoke-test loop, and the common mistakes (repo_path pointing at the wrong dir, score direction backwards, hidden answer keys leaking into seed/, grader writing to codebase_path which the daemon force-removes, private-vs-public confusion, missing `run()` signature).
Passed 21 of 22 test cases with the skill, and 6 of 22 without it.
and 22 more in the full report
15 fixed6 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 15 test cases the skill fixed. Same model, same prompt, with and without the skill.
Inside a TaskGrader evaluate() method, what attribute provides the file path to hidden answer keys declared under grader.private?
| Checked | Without skill | With skill |
|---|---|---|
| Directs accessing secret files using self.private_dir. | Missed | Passed |
The instructions your model reads when the skill loads.
A CORAL task is three things that must line up:
examples/<task>/
├── task.yaml # config: name, description, grader entrypoint, agent count
├── seed/ # starter code agents see when they begin (the repo_path)
│ └── solution.py
└── grader/ # standalone Python package
├── pyproject.toml
└── src/<task>_grader/
├── __init__.py
└── grader.py # class Grader(TaskGrader): ...
The packaged form is the only supported form. The package gives the grader its own venv and ships everything the eval needs — grader code, helper modules, and hidden data (see "Hidden data" below).
Look at these before writing anything new — copy the closest one and edit:
| Reference | When to copy it |
|---|---|
| examples/erdos/ | Minimal packaged grader, single grader file, numpy-only deps |
| examples/dna_design/ | Packaged grader with bundled data files (importlib.resources) and [ml] optional-deps for heavy libs |
| examples/swebench-verified/ | Tiered eval (different instance counts per tier), private answer keys, harbor integration |
| examples/circle_packing/ | Smallest packaged task end-to-end — single solution file, single grader file |
| examples/mnist/ | Packaged grader with a hidden answer key (note: secret data belongs under grader.private in a taskdata/ sibling of grader/, never inside the grader package) |
Whatever lives in seed/ is what the agent sees on first checkout — it's the working directory the grader will later score. The contract between seed/ and the grader is the program file: a Python file with a function the grader imports and calls.
The convention across examples is:
solution.py (or initial_program.py) defining a top-level run() function.program_file: "solution.py" via grader.args.run()'s signature is whatever the grader expects — usually () -> result or (input_path) -> result.Put a real, runnable baseline here. Agents should be able to coral eval immediately and get a non-zero score, so they have a starting point to improve. A no-op skeleton that crashes is not a good baseline.
test cases passed, vs 6 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.