Use when: running SkillOpt, training a skill, evaluating a skill, using the local mitko model on port 8000, working with the dotnetdebug benchmark, adding a benchmark, adding a backend, or optimizing agent instructions with SkillOpt in this repository.
SkillScore82Maintained
Automatic
◆AI Engineer pack
Last verified 8/3/2026
What it changes
Passed 16 of 19 test cases with the skill, and 5 of 19 without it.
Without skill27%
With skill82%
Checks it passed that the model alone missed
Specifies openai_compat as the driver backend name
Uses the --cfg-options flag to supply parameter overrides
Invokes scripts/eval_only.py rather than the training entrypoint
and 13 more in the full report
13 fixed5 passed either way3 still failing1 broke
Pass-rate lift
+55 pp
Tokens
+11%
Agent turns
0%
Response time
-59%
Model gemini-3.6-flashMeasured 8/3/2026v118 of 22 cases passed with skill · 19 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 12
Prompt
What is the relative file path for the benchmark task dataset used by the dotnet debug sample?
What was checked, without and with the skill
Checked
Without skill
With skill
Identifies data/dotnetdebug/tasks.json as the sample task dataset
Missed
Passed
case-17 · gemini-3.6-flash · 8/3/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md118 linessingle file
PI SkillOpt Operator
Use this skill when the user wants the agent to operate the SkillOpt repository: run experiments, evaluate skills, configure local models, inspect outputs, or extend the repo with new benchmarks/backends.
Core repo facts
SkillOpt optimizes a skill document / system prompt, not model weights.
The repo supports a generic local OpenAI-compatible backend named openai_compat.
The local example model config is configs/dotnetdebug/local_mitko.yaml.
That config uses model mitko at http://localhost:8000/v1 for both optimizer and target.
The runnable sample benchmark is dotnetdebug.
The sample dataset is data/dotnetdebug/tasks.json.
The seed skill is skillopt/envs/dotnetdebug/skills/initial.md.
Training entry point: scripts/train.py.
Eval-only entry point: scripts/eval_only.py.
Default workflow
Identify the user goal:
run a SkillOpt training job
evaluate an existing skill
inspect outputs from a previous run
add or modify a benchmark
add or modify a backend
Confirm the benchmark, backend, target model, and desired output directory.
Prefer a small smoke test first before a larger run.
Use existing configs when possible instead of inventing new ones.
After a run, report:
exact command used
output directory
best skill path
headline metrics
next recommended action
Local model workflow
When the user asks to use a local model or mentions mitko, localhost:8000, or an OpenAI-compatible endpoint:
Prefer model.backend: openai_compat.
Set both optimizer_backend and target_backend to openai_compat unless the user explicitly wants a mixed setup.