Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →jeremylongshore/langchain-eval-harness
Build reproducible evaluation pipelines for LangChain 1.0 chains and LangGraph 1.0 agents — golden datasets, LangSmith evaluate(), ragas RAG metrics, deepeval LLM-as-judge, agent trajectory analysis, and CI gating on quality regressions.
Passed 20 of 23 test cases with the skill, and 11 of 23 without it.
and 11 more in the full report
9 fixed11 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
We are writing a deepeval G-Eval metric to check if an actual model answer matches an expected reference answer in meaning. Which `LLMTestCaseParams` should be passed to the evaluation parameters parameter list?
| Checked | Without skill | With skill |
|---|---|---|
| Passes INPUT, ACTUAL_OUTPUT, and EXPECTED_OUTPUT to GEval evaluation_params | Missed | Passed |
The instructions your model reads when the skill loads.
A team swapped gpt-4o for claude-sonnet-4-6 to save money and a week later CS
noticed answer quality dropped on 15% of refund tickets — the regression was
invisible in code review and invisible in CI because no golden set existed.
Fix: a versioned golden set, a stacked eval pipeline (LangSmith + ragas + deepeval + custom trajectory), and a PR-blocking regression gate with paired Wilcoxon significance. The tooling exists; the patterns for wiring it into a statistically honest loop are scattered across five doc sites.
Build a 100-example JSONL golden set, wire LangSmith evaluate() with a
custom correctness evaluator, add a ragas quartet (faithfulness, answer
relevance, context precision/recall) for RAG, add deepeval LLM-as-judge
with N=3 judge quorum, score LangGraph trajectories on coverage/precision/
order, and gate PRs on a 2% aggregate drop or 5% per-example drop. Pin:
langchain-core 1.0.x, langgraph 1.0.x, langsmith>=0.2, ragas>=0.2,
deepeval>=2.0. Pain-catalog anchors: P01, P11, P12, P22, P33.
langchain-core >= 1.0, < 2.0, langgraph >= 1.0, < 2.0 for the system under evalpip install langsmith>=0.2 ragas>=0.2 deepeval>=2.0 scipyLANGSMITH_API_KEY (free tier is sufficient for dataset versioning)OPENAI_API_KEY and/or ANTHROPIC_API_KEYFormat: JSONL, one example per line, with a dataset_version tag. Minimum 20
examples to start; grow to 100 for PR gating, 200+ for absolute-metric claims.
# evals/golden_set/v2026.04.jsonl
{"id": "gs-0001", "input": "Refund policy for SKU ABC-42?", "expected": "30 days with receipt", "contexts": ["policy_v3.md"], "tags": ["refund"], "difficulty": "easy", "dataset_version": "2026.04"}
{"id": "gs-0002", "input": "Return policy for opened software?", "expected": "No, opened software is final sale", "contexts": ["policy_v3.md#returns"], "tags": ["refund"], "difficulty": "medium", "dataset_version": "2026.04"}
test cases passed, vs 11 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.