Passed 14 of 18 test cases with the skill, and 9 of 18 without it.
Without skill50%
With skill77%
Checks it passed that the model alone missed
Formats show ip ospf using pyats_run_show_command for device R1
Formats show ip route summary using pyats_run_show_command for device R1
Executes show ip ospf neighbor using pyats_run_show_command for device R3
and 6 more in the full report
7 fixed10 passed either way4 still failing1 broke
Pass-rate lift
+27 pp
Tokens
+152%
Agent turns
0%
Response time
-29%
Model gemini-3.6-flashMeasured 8/16/2026v117 of 22 cases passed with skill · 18 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 7
Prompt
We need to inspect the forwarding table for the management VRF named 'MGMT' on device Core-1. Which show command execution format should be sent to the device runner?
What was checked, without and with the skill
Checked
Without skill
With skill
Formats the JSON payload as {"device_name":"Core-1","command":"show ip route vrf MGMT"} using pyats_run_show_command
Missed
Passed
case-08 · gemini-3.6-flash · 8/16/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md239 linessingle file
Routing Protocol Analysis
Routing Table Analysis
Always start here. The routing table is the source of truth for forwarding decisions.
Full Routing Table
PYATS_TESTBED_PATH=$PYATS_TESTBED_PATH python3 $MCP_CALL "${PYATS_PYTHON:-python3} -u $PYATS_MCP_SCRIPT" pyats_run_show_command '{"device_name":"R1","command":"show ip route"}'
Per-Protocol Routes
# OSPF routes only
PYATS_TESTBED_PATH=$PYATS_TESTBED_PATH python3 $MCP_CALL "${PYATS_PYTHON:-python3} -u $PYATS_MCP_SCRIPT" pyats_run_show_command '{"device_name":"R1","command":"show ip route ospf"}'
# BGP routes only
PYATS_TESTBED_PATH=$PYATS_TESTBED_PATH python3 $MCP_CALL "${PYATS_PYTHON:-python3} -u $PYATS_MCP_SCRIPT" pyats_run_show_command '{"device_name":"R1","command":"show ip route bgp"}'
# Connected and static
PYATS_TESTBED_PATH=$PYATS_TESTBED_PATH python3 $MCP_CALL "${PYATS_PYTHON:-python3} -u $PYATS_MCP_SCRIPT" pyats_run_show_command '{"device_name":"R1","command":"show ip route connected"}'
PYATS_TESTBED_PATH=$PYATS_TESTBED_PATH python3 $MCP_CALL "${PYATS_PYTHON:-python3} -u $PYATS_MCP_SCRIPT" pyats_run_show_command '{"device_name":"R1","command":"show ip route static"}'