M365 Agent Evaluator

M365 Agent Evaluator

Use this skill to help users evaluate Microsoft 365 Copilot declarative agents with @microsoft/m365-copilot-eval. The skill designs schema-compatible eval datasets, runs the public preview CLI, analyzes results, and recommends targeted fixes.

Default to Microsoft 365 Agents Toolkit (ATK) projects when detected, but do not hard-stop solely because the current directory is not ATK. The CLI can also evaluate deployed agents with an explicit M365_AGENT_ID or --m365-agent-id.

Always use this CLI invocation

npx -y --package @microsoft/m365-copilot-eval@latest runevals

Do not recommend the old private aka.ms installer, global installs, bare runevals, bare npx runevals, --input, or --html.

Activation workflow

1. Identify the user goal: setup, dataset authoring, running evals, analyzing results, or updating an existing eval suite.
2. Load only the reference needed for the current goal:
   • references/workflow.md for the end-to-end operator workflow and CLI commands.
   • references/azure-setup.md for prerequisites, env files, and secret handling.
   • references/eval-templates.md when creating or editing eval datasets.
   • references/pra-framework.md when deciding what scenarios to generate.
   • references/result-analysis.md after JSON/CSV/HTML results exist.
   • references/guardrails.md before writing files, handling secrets, clearing cache, signing out, or troubleshooting.
3. Detect project shape:
   • ATK: .env.local, .env.local.user, env\.env.local.user, m365agents.yml, or appPackage\declarativeAgent.json.
   • Non-ATK: an eval dataset plus M365_AGENT_ID, --m365-agent-id, or a named environment file such as env\.env.dev.
4. Verify prerequisites without exposing values:
   • Node.js 24.12.0 or newer.
   • Microsoft 365 Copilot license and a deployed M365 Copilot agent.
   • Tenant admin consent for the WorkIQ Client App.
   • TENANT_ID, Azure OpenAI in Foundry Models endpoint/key, and recommended/default gpt-4o-mini deployment.
5. Choose the workflow:
   • No dataset: create evals\evals.json.
   • Existing dataset: run, analyze prior results, or propose changes.
   • Quick check: use inline prompts.
   • Exploration: use interactive mode.

Current dataset contract