Adding a Scoring Check

Adding a Scoring Check

Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.

Critical

• Check must be deterministic: Same filesystem state → same result every time. No randomness, no external APIs.
• Point values come from constants.ts: Every earnedPoints and maxPoints must reference POINTS_* from src/scoring/constants.ts. Do NOT hardcode numbers.
• Always return Check[] array: Export a function check<Category>(dir: string): Check[] where category is one of: existence, quality, grounding, accuracy, freshness, bonus.
• Every check must have: id (kebab-case, unique), name, category, maxPoints, earnedPoints, passed, detail, and optional suggestion/fix.
• Fix object fields: action (string describing what to do), data (context for the fix), instruction (user-facing guidance).
• Register in src/scoring/index.ts: Add the import and spread the result into the allChecks array in computeLocalScore().
• Target filtering: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate *_ONLY_CHECKS set in constants.ts.

Instructions

Step 1: Define point constants in src/scoring/constants.ts

Verify before proceeding: Is your check measurable with a numeric point value?

Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):

// In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical

// If threshold-based, add a companion array:
export const YOUR_THRESHOLD_ARRAY = [
  { minValue: 10, points: 4 },
  { minValue: 5, points: 2 },
] as const;

Check existing patterns: Token budgets use TOKEN_BUDGET_THRESHOLDS, code blocks use CODE_BLOCK_THRESHOLDS, concreteness uses CONCRETENESS_THRESHOLDS.

Verify: Review CATEGORY_MAX object to ensure your check fits within its category's point budget.

Step 2: Create or edit check function in src/scoring/checks/

Choose the file based on category. Each file exports a check<Name>(dir: string): Check[] function: