Passed 18 of 22 test cases with the skill, and 5 of 22 without it.
Without skill23%
With skill82%
Checks it passed that the model alone missed
Recommends using static query indices.
Declines to author Terraform resource modules.
Declines to provide JVM heap sizing parameters.
and 18 more in the full report
14 fixed4 passed either way3 still failing1 broke
Pass-rate lift
+59 pp
Tokens
+92%
Agent turns
0%
Response time
-15%
Model gemini-3.6-flashMeasured 8/20/2026v118 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 14 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 14
Prompt
We just refactored 20 detection rules in our OpenSearch SIEM. Summarize the outcome for executive management by listing the total number of rules updated and hours spent by the engineering team.
What was checked, without and with the skill
Checked
Without skill
With skill
Frames outcomes using security operational metrics such as TTD or TTR instead of activity counts.
Missed
Passed
case-12 · gemini-3.6-flash · 8/20/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md170 lines4 files
OpenSearch Detection Engineering
Methodology for authoring and validating SIEM detections on OpenSearch Security Analytics: SIGMA rules, query DSL translation, MITRE ATT&CK mapping, anomaly detection, correlation, and SOC incident escalation. Vendor-neutral framework with OpenSearch-specific API patterns.
Load detection-engineering.md for translation patterns
Tune false positive rate or threshold
Load detection-engineering.md for calibration steps
Build escalation package or run SOC handoff
Load incident-escalation.md for 9-field gate
Diagnose detector creation failure or alert flood
Load detection-safety-patterns.md for OpenSearch failure modes
Map detection to MITRE ATT&CK
Load detection-engineering.md for tactic/technique catalog
Hardcoded Behaviors (Always Apply)
MITRE ATT&CK on every detection. Include technique ID (e.g., T1110.003), tactic name (e.g., Credential Access), and kill chain phase with every rule, alert, or detector. Tactic alone is insufficient — technique IDs enable coverage gap analysis.
Field-existence check before rule creation. Run GET {index}/_mapping and confirm every field referenced in the rule exists in the target index mapping before submitting the detector. Absent fields cause silent failure or misleading errors.
Concrete API commands. Provide PUT _mapping, POST _aliases, POST /_plugins/_security_analytics/... — not abstract advice.
Severity tier governs SLA. Reference response-time tiers as binding requirements, not advisory targets. Configure tier defaults per organization; defaults are documented in incident-escalation.md.
Tier-1 vs Tier-2 distinction. Distinguish runbook-driven triage and enrichment (Tier-1) from deep-dive investigation (Tier-2) when authoring use case docs and escalation paths.
Detection-owned index recommendation. When a detector bootstraps field aliases, recommend a dedicated index separate from the ingestion datastream — bootstrap is destructive on shared indices.
KPI framing. Frame detection changes in terms of TTD / TTR / MTTR / FP rate / escalation quality score impact. Track measurable outcomes, not activity.
Default Behaviors (ON unless disabled)
6-section use case lifecycle template. Produce use case documentation in the standardized format (General Info, Context, Outcomes, Detection Logic, Continuous Improvement, Analyst Support). See incident-escalation.md.
SIGMA → DSL translation with field validation. When given a SIGMA rule, translate to OpenSearch query DSL and flag any fields not present in the target mapping before proposing detector creation.
False positive suppression patterns. Apply CIDR exclusions, service-account prefixes, and time-window suppression in monitor queries — surfaced explicitly, never silently suppressed.
Optional Behaviors (OFF unless enabled)
Purple team / tabletop support. Map detection coverage against MITRE ATT&CK gaps for red/blue exercises.
Log source onboarding. Field availability analysis, cardinality checks, and coverage mapping for new sources.
Hard Gate Patterns
Before creating or modifying a detector, check for these. If found, STOP and resolve before continuing.