Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

ellmos-tests · ellmos-ai | DecimalAI

testingby ellmos-ai

ellmos-tests

ellmos-ai/ellmos-tests

Source ellmos-ai/ellmos-tests ↗License MIT

Standalone B/O/E testkit for evaluating SKILL.md-based LLM operating systems.

SkillScore100Maintained

Automatic

QA & Test Engineering pack

Last verified 8/7/2026

What it changes

Passed 22 of 22 test cases with the skill, and 4 of 22 without it.

Without skill18%

With skill100%

Checks it passed that the model alone missed

Includes the --only b flag in the command
Includes the --only o flag in the command
Includes the --profile STANDARD flag in the command

and 19 more in the full report

18 fixed4 passed either way0 still failing0 broke

Pass-rate lift: +82 pp
Tokens: -38%
Agent turns: 0%
Response time: -66%

Model gemini-3.6-flashMeasured 8/7/2026v122 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/ellmos-ai-ellmos-tests/SKILL.md https://app.decimal.ai/s/ellmos-ai-ellmos-tests/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add ellmos-ai-ellmos-tests, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/ellmos-ai-ellmos-tests/ and pins it in .decimal/skills.lock

decimalai skills add ellmos-ai-ellmos-tests

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/ellmos-ai-ellmos-tests/SKILL.md https://app.decimal.ai/s/ellmos-ai-ellmos-tests/SKILL.md

Raw file: /s/ellmos-ai-ellmos-tests/SKILL.md ↗ · machine-readable summary at /s/ellmos-ai-ellmos-tests.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 18 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 18

Prompt

Can you provide the bash invocation to run the standard benchmark profile against my target system directory at `/repos/agent-os`? I want the precise single-line command formatted as text.

What was checked, without and with the skill
Checked	Without skill	With skill
Includes system_diff_tests/testing/run_external.py /repos/agent-os in the terminal command	Missed	Passed
Includes the --profile STANDARD flag in the command	Missed	Passed

case-02 · gemini-3.6-flash · 8/7/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md38 lines3 files

ellmos-tests

Use this module when an LLM should evaluate a local LLM-OS, agent hub, or SKILL.md-based system with the B/O/E method.

What It Provides

B-tests: automated observation tests for inventory, structure, naming, docs, code metrics, dependencies, and age.
O-tests: functional output tests for task, memory, tool registry, backup, config, and export/import surfaces.
E-tests: manual experience tasks for orientation, memory, task workflows, tool use, recovery, and session startup.
Battery runner: curated checklist-style batteries under tests/batteries/.
Optional browser helpers: Playwright wrappers under system_diff_tests/testing/playwright/.

Standard Commands

PYTHONIOENCODING=utf-8 python system_diff_tests/run_all.py "<system_path>" --only b
PYTHONIOENCODING=utf-8 python system_diff_tests/run_all.py "<system_path>" --only o
PYTHONIOENCODING=utf-8 python system_diff_tests/testing/run_external.py "<system_path>" --profile STANDARD
PYTHONIOENCODING=utf-8 python tests/run_batteries.py --list

Known systems are resolved through system_diff_tests/config.py.

Boundaries

Results are generated under ignored output folders by default.
Do not include private corpora, local BACH context, .db files, result archives, __pycache__, or .pytest_cache in the module package.
Prefer environment variables (ELLMOS_BASE_PATH, ELLMOS_ONEDRIVE, BACH_SYSTEM_PATH, BACH_DB_PATH) over hardcoded user paths.
Playwright is optional; the core module remains Python-stdlib only.

Ownership and Extraction

This module originated by extracting the portable part of BACH system/tools/testing. For the reusable B/O/E testing surface, ellmos-tests is now the canonical feature source. BACH consumes it through a compatibility adapter and retains only its integration contract plus a rollback-safe legacy copy while the gated migration is validated. Changes to reusable runners, profiles, and B/O/E test definitions belong here first; BACH-specific lifecycle or handler behavior remains in BACH.

Track record

Benchmark

22 / 22

test cases passed, vs 4 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/2/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

ellmos-ai / ellmos-tests

SKILL.md

View on GitHub ↗

License MITImported 9/26/2026Version v1Bundled 2 files

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

ellmos-tests

What It Provides

Standard Commands

Boundaries

Ownership and Extraction

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills