Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

LLM Engineering – Operational Skill Hub · microck | DecimalAI

microck/ai-llm-engineering

SkillScore100Maintained

Operational skill hub for LLM system architecture, evaluation, deployment, and optimization (modern production standards). Links to specialized skills for prompts, RAG, agents, and safety. Integrates recent advances: PEFT/LoRA fine-tuning, hybrid RAG handoff (see dedicated skill), vLLM 24x throughput, multi-layered security (90%+ bypass for single-layer), automated drift detection (18-second response), and CI/CD-aligned evaluation.

AI/MLby microck

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/microck-ai-llm-engineering/SKILL.md

decimalai skills pull microck-ai-llm-engineering --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/microck-ai-llm-engineering/SKILL.md https://app.decimal.ai/s/microck-ai-llm-engineering/SKILL.md

Raw file: /s/microck-ai-llm-engineering/SKILL.md ↗ · machine-readable summary at /s/microck-ai-llm-engineering.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 38d ago

Yes — +9% lift · at higher cost+179% tokens0% turns100% of whole cases pass with skill (22/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-04	✗→✓	▲ Improved	168%	0%
case-10	✗→✓	▲ Improved	115%	0%
case-01	✓→✓	＝ Same ✓	116%	0%
case-02	✓→✓	＝ Same ✓	123%	0%
case-03	✓→✓	＝ Same ✓	155%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

microck / ordinary-claude-skills

skills_all/ai-llm-engineering/SKILL.md

View on GitHub ↗

License NOASSERTIONImported 8/3/2026Version v1

Skill contents

SKILL.md

LLM Engineering – Operational Skill Hub

A single resource for executing, validating, and scaling LLM systems with modern production standards, while delegating domain depth to specialized skills.

This skill provides quick reference, decision frameworks, and navigation to detailed operational patterns for:

Data, training, fine-tuning (PEFT/LoRA standard)
Evaluation (automated testing, metrics, rollout gates)
Deployment (vLLM 24x throughput, FP8/FP4 quantization)
LLMOps (automated drift detection, retraining)
Safety (multi-layered defenses, AI-powered guardrails)

For detailed patterns: See Resources and Templates sections below.

Quick Reference

| Task | Tool/Framework | Command/Pattern | When to Use | |------|----------------|-----------------|-------------| | RAG Pipeline | LlamaIndex, LangChain | Page-level chunking + hybrid retrieval | Dynamic knowledge, 0.648 accuracy | | Agentic Workflow | LangGraph, AutoGen, CrewAI | ReAct, multi-agent orchestration | Complex tasks, tool use required | | Prompt Design | Anthropic, OpenAI guides | CoT, few-shot, structured | Task-specific behavior control | | Evaluation | LangSmith, W&B, RAGAS | Multi-metric (hallucination, bias, cost) | Quality validation, A/B testing | | Production Deploy | vLLM, TensorRT-LLM | FP8/FP4 quantization, 24x throughput | High-throughput serving, cost optimization | | Monitoring | Arize Phoenix, LangFuse | Drift detection, 18-second response | Production LLM systems |

Decision Tree: LLM System Architecture

text
Building LLM application: [Architecture Selection]
    ├─ Need current knowledge?
    │   ├─ Simple Q&A? → Basic RAG (page-level chunking + hybrid retrieval)
    │   └─ Complex retrieval? → Advanced RAG (reranking + contextual retrieval)
    │
    ├─ Need tool use / actions?
    │   ├─ Single task? → Simple agent (ReAct pattern)
    │   └─ Multi-step workflow? → Multi-agent (LangGraph, CrewAI)
    │
    ├─ Static behavior sufficient?
    │   ├─ Quick MVP? → Prompt engineering (CI/CD integrated)
    │   └─ Production quality? → Fine-tuning (PEFT/LoRA)
    │
    └─ Best results?
        └─ Hybrid (RAG + Fine-tuning + Agents) → Comprehensive solution

See Decision Matrices for detailed selection criteria.

When to Use This Skill

Claude should invoke this skill when the user asks about:

LLM preflight/project checklists, production best practices, or data pipelines
Building or deploying RAG, agentic, or prompt-based LLM apps
Prompt design, chain-of-thought (CoT), ReAct, or template patterns
Troubleshooting LLM hallucination, bias, retrieval issues, or production failures
Evaluating LLMs: benchmarks, multi-metric eval, or rollout/monitoring
LLMOps: deployment, rollback, scaling, resource optimization
Technology stack selection (models, vector DBs, frameworks)
Production deployment strategies and operational patterns

Scope Boundaries (Use These Skills for Depth)

Prompt design & CI/CD → ai-prompt-engineering
RAG pipelines & chunking → ai-llm-rag-engineering
Search tuning (BM25, HNSW, hybrid) → ai-llm-search-retrieval
Agent architectures & tools → ai-agents-development
Serving optimization/quantization → ai-llm-ops-inference
Production deployment/monitoring → ai-ml-ops-production
Security/guardrails → ai-ml-ops-security

Resources (Best Practices & Operational Patterns)

Comprehensive operational guides with checklists, patterns, and decision frameworks:

Core Operational Patterns

Project Planning Patterns - Stack selection, FTI pipeline, performance budgeting
AI engineering stack selection matrix
Feature/Training/Inference (FTI) pipeline blueprint
Performance budgeting and goodput gates
Progressive complexity (prompt → RAG → fine-tune → hybrid)

Production Checklists - Pre-deployment validation and operational checklists
LLM lifecycle checklist (modern production standards)
Data & training, RAG pipeline, deployment & serving
Safety/guardrails, evaluation, agentic systems
Reliability & data infrastructure (DDIA-grade)
Weekly production tasks

Common Design Patterns - Copy-paste ready implementation examples
Chain-of-Thought (CoT) prompting
ReAct (Reason + Act) pattern
RAG pipeline (minimal to advanced)
Agentic planning loop
Self-reflection and multi-agent collaboration

Decision Matrices - Quick reference tables for selection
RAG type decision matrix (naive → advanced → modular)
Production evaluation table with targets and actions
Model selection matrix (GPT-4, Claude, Gemini, self-hosted)
Vector database, embedding model, framework selection
Deployment strategy matrix

Anti-Patterns - Common mistakes and prevention strategies
Data leakage, prompt dilution, RAG context overload
Agentic runaway, over-engineering, ignoring evaluation
Hard-coded prompts, missing observability
Detection methods and prevention code examples

Domain-Specific Patterns

LLMOps Best Practices - Operational lifecycle and deployment patterns
Evaluation Patterns - Testing, metrics, and quality validation
Prompt Engineering Patterns - Quick reference (canonical skill: ai-prompt-engineering)
Agentic Patterns - Quick reference (canonical skill: ai-agents-development)
RAG Best Practices - Quick reference (canonical skill: ai-llm-rag-engineering)

Note: Each resource file includes preflight/validation checklists, copy-paste reference tables, inline templates, anti-patterns, and decision matrices.

Templates (Copy-Paste Ready)

Production templates by use case and technology:

RAG Pipelines

Basic RAG - Simple retrieval-augmented generation
Advanced RAG - Hybrid retrieval, reranking, contextual embeddings

Prompt Engineering

Chain-of-Thought - Step-by-step reasoning pattern
ReAct - Reason + Act for tool use

Agentic Workflows

Reflection Agent - Self-critique and improvement
Multi-Agent - Manager-worker orchestration

Data Pipelines

Data Quality - Validation, deduplication, PII detection

Deployment

LLM Deployment - Production deployment with monitoring

Evaluation

Multi-Metric Evaluation - Comprehensive testing suite

Related Skills

This skill integrates with complementary Claude Code skills:

Core Dependencies

ai-llm-rag-engineering - Advanced RAG patterns, chunking strategies, hybrid retrieval, reranking
ai-llm-search-retrieval - Search optimization, BM25 tuning, vector search, ranking pipelines
ai-prompt-engineering - Systematic prompt design, evaluation, testing, and optimization
ai-agents-development - Agent architectures, tool use, multi-agent systems, autonomous workflows

Production & Operations

ai-llm-development - Model training, fine-tuning, dataset creation, instruction tuning
ai-llm-ops-inference - Production serving, quantization, batching, GPU optimization
ai-ml-ops-production - Deployment patterns, monitoring, drift detection, API design
ai-ml-ops-security - Security guardrails, prompt injection defense, privacy protection

External Resources

See data/sources.json for 50+ curated authoritative sources:

Official LLM platform docs - OpenAI, Anthropic, Gemini, Mistral, Azure OpenAI, AWS Bedrock
Open-source models and frameworks - HuggingFace Transformers, LLaMA, vLLM, PEFT/LoRA, DeepSpeed
RAG frameworks and vector DBs - LlamaIndex, LangChain, LangGraph, Haystack, Pinecone, Qdrant, Chroma
2025 Agentic frameworks - Anthropic Agent SDK, AutoGen, CrewAI, LangGraph Multi-Agent, Semantic Kernel
2025 RAG innovations - Microsoft GraphRAG (knowledge graphs), Pathway (real-time), hybrid retrieval
Prompt engineering - Anthropic Prompt Library, Prompt Engineering Guide, CoT/ReAct patterns
Evaluation and monitoring - OpenAI Evals, HELM, Anthropic Evals, LangSmith, W&B, Arize Phoenix
Production deployment - LiteLLM, Ollama, RunPod, Together AI, vLLM serving

Usage

For New Projects

Start with Production Checklists - Validate all pre-deployment requirements
Use Decision Matrices - Select technology stack
Reference Project Planning Patterns - Design FTI pipeline
Implement with Common Design Patterns - Copy-paste code examples
Avoid Anti-Patterns - Learn from common mistakes

For Troubleshooting

Check Anti-Patterns - Identify failure modes and mitigations
Use Decision Matrices - Evaluate if architecture fits use case
Reference Common Design Patterns - Verify implementation correctness

For Ongoing Operations

Follow Production Checklists - Weekly operational tasks
Integrate Evaluation Patterns - Continuous quality monitoring
Apply LLMOps Best Practices - Deployment and rollback procedures

Navigation Summary

Quick Decisions: Decision Matrices Pre-Deployment: Production Checklists Planning: Project Planning Patterns Implementation: Common Design Patterns Troubleshooting: Anti-Patterns

Domain Depth: LLMOps | Evaluation | Prompts | Agents | RAG

Templates: templates/ - Copy-paste ready production code

Sources: data/sources.json - Authoritative documentation links

Illustrative Want to see what microck/ai-llm-engineering would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/20/2026, 23:30 UTC · 22 cases

Lift

+9%

100% vs 91%

Agent turns

1.0 vs 1.0

Tokens

+179%

5,167.05 vs 1,851.91

fail → pass · 2pass → pass · 20

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 When tuning BM25 lexical search parameters k1 and b in an inverted index engine, many engineers set k1=0 and b=1 to maximize document length penalty. Explain the specific technical roles of k1 and b in BM25 scoring and why that setting is usually wrong.	pass→pass	17,143	19,947	+16%	1	1	0%	3,202	6,910	+116%	0	0	—
case-02 Some engineers attempt to fix CUDA out-of-memory errors during LLM batch inference by pre-allocating a single contiguous memory block for the maximum context window of every sequence. Explain how PagedAttention solves KV cache fragmentation instead.	pass→pass	14,957	13,868	-7%	1	1	0%	2,327	5,180	+123%	0	0	—
case-03 A team wants to run prompt evaluations on pull requests by triggering a manual Python script locally. Show how to automate prompt regression testing directly inside a GitHub Actions workflow step using Promptfoo.	pass→pass	11,936	11,252	-6%	1	1	0%	2,187	5,572	+155%	0	0	—
case-04 We have thousands of frequently updated internal policy documents and need to answer user Q&A accurately. A team member suggested fine-tuning Llama-3 on all policy text so the model memorizes the documents. What architecture should we use instead for dynamic policy knowledge?	fail→pass	13,749	17,495	+27%	1	1	0%	2,301	6,157	+168%	0	0	—
case-05 Our simple semantic vector search fails on multi-document technical queries because top-k cosine similarity returns irrelevant chunks. Someone proposed increasing top-k from 5 to 100 to catch everything. What architectural approach should we implement instead for complex retrieval?	pass→pass	15,742	15,023	-5%	1	1	0%	2,531	5,727	+126%	0	0	—
case-06 We are designing a system to handle complex multi-step customer support tasks requiring 12 different tools, multi-stage approval, and sub-task coordination. A developer suggested putting all tool definitions and system instructions into a single massive system prompt. What architectural pattern should we adopt instead?	pass→pass	13,558	12,644	-7%	1	1	0%	2,213	5,101	+131%	0	0	—
case-07 Our LLM MVP uses prompt engineering, but production quality requires deeply custom task adaptation and static behavioral control without changing context dynamically. The team plans to pre-train a 70B model from scratch. What efficient training method should we use instead?	pass→pass	13,100	14,880	+14%	1	1	0%	2,175	5,829	+168%	0	0	—
case-08 We need an open-source serving solution capable of FP8/FP4 quantization and delivering high-concurrency throughput (up to 24x over standard PyTorch). Someone suggested running default PyTorch model.generate() inside a FastAPI wrapper. What production serving engine should we deploy?	pass→pass	17,044	18,078	+6%	1	1	0%	2,735	6,117	+124%	0	0	—
case-09 For our production LLM system, an engineer suggested dumping raw model JSON output logs to stdout and filtering them daily via grep. What specialized LLM monitoring platform should we deploy for automated drift detection and operational trace observability?	pass→pass	15,288	18,780	+23%	1	1	0%	2,332	6,153	+164%	0	0	—
case-10 When building an enterprise LLM solution, an engineer proposed having the inference service query raw production SQL databases directly, run embedding calculations on the fly, and use the same unstructured scripts for training dataset generation. What standard pipeline blueprint should we implement to separate these concerns?	fail→pass	16,701	15,649	-6%	1	1	0%	2,539	5,466	+115%	0	0	—
case-11 When combining dense vector retrieval with BM25 sparse keyword search in a hybrid RAG pipeline, a developer suggests simply adding the raw cosine similarity score to the raw BM25 score. Why is direct score addition problematic and what rank-based algorithm should be used to combine the ranked lists?	pass→pass	13,115	14,878	+13%	1	1	0%	2,260	5,774	+155%	0	0	—
case-12 To evaluate whether an LLM generation in a RAG system contains hallucinations not supported by the retrieved context chunks, an engineer proposes calculating the BLEU score between the retrieved context and the generated response. What standard RAG evaluation metric explicitly measures how grounded the answer is in the retrieved context?	pass→pass	4,763	7,145	+50%	1	1	0%	807	4,139	+413%	0	0	—
case-13 When assessing if a vector retriever successfully fetches all necessary reference information required to answer a complex question, a teammate recommends measuring latency per query. What specific evaluation metric quantifies whether all relevant context statements were successfully retrieved?	pass→pass	6,760	7,859	+16%	1	1	0%	1,146	4,178	+265%	0	0	—
case-14 To ensure a RAG system never misses information, a developer configured the vector retriever to inject 50 full documents into the prompt context window. What operational anti-pattern does this trigger, and what is its negative impact on LLM output?	pass→pass	9,343	8,464	-9%	1	1	0%	1,471	4,251	+189%	0	0	—
case-15 To achieve a 100% evaluation score on our test benchmark before launch, a data scientist combined the evaluation test set directly into the training set for LoRA fine-tuning. What severe operational anti-pattern was committed?	pass→pass	6,745	7,144	+6%	1	1	0%	1,095	4,052	+270%	0	0	—
case-16 When designing an autonomous tool-calling prompt, a developer proposes having the LLM output raw Python code blocks directly without explicitly modeling reasoning steps before tool invocation. What standard prompt reasoning pattern structures tool interaction into explicit Thought, Action, Action Input, and Observation cycles?	pass→pass	3,950	7,378	+87%	1	1	0%	637	4,161	+553%	0	0	—
case-17 For parsing long unstructured legal documents into a vector database, an engineer proposes splitting text into fixed 200-character chunks with zero overlap. When semantic boundary loss and context fragmentation occur, what chunking strategy retrieves parent sections while indexing smaller child chunks for precise vector match?	pass→pass	9,218	11,794	+28%	1	1	0%	1,436	5,095	+255%	0	0	—
case-18 When configuring Low-Rank Adaptation (LoRA) for fine-tuning an LLM, a developer sets rank (r) to 16 and alpha to 0, assuming alpha is optional and ignored. What is the impact of setting alpha to 0, and what standard rule of thumb relates alpha to rank r?	pass→pass	9,173	7,198	-22%	1	1	0%	1,641	4,236	+158%	0	0	—
case-19 In multi-tenant LLM serving where thousands of users send queries sharing an identical 2,000-token system prompt, requests experience high prefill latency and redundant KV cache computations. What serving engine optimization feature should be enabled to reuse precomputed KV caches for shared context prefixes across requests?	pass→pass	4,414	8,503	+93%	1	1	0%	805	4,487	+457%	0	0	—
case-20 To defend an LLM application against direct and indirect prompt injection attacks, a developer suggests using a single regex pattern matching the word 'ignore'. Why is simple regex insufficient, and what architectural layer should sanitize inputs and enforce safety boundaries before passing prompts to the core model?	pass→pass	13,429	14,906	+11%	1	1	0%	2,134	5,257	+146%	0	0	—
case-21 An autonomous agent is configured to automatically execute database deletion queries and issue financial refunds without intermediate verification. To prevent catastrophic accidental tool actions during agent execution, what control pattern should be implemented before executing high-risk tools?	pass→pass	42,883	12,564	-71%	1	1	0%	1,411	4,974	+253%	0	0	—
case-22 To reduce vector database storage costs for 1536-dimensional embeddings, an engineer suggests truncating the float32 vector array by manually deleting the last 1000 dimensions from standard embeddings. What embedding model technique natively supports flexible vector dimension reduction without catastrophic loss of representation quality?	pass→pass	7,955	10,347	+30%	1	1	0%	1,357	4,859	+258%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +9 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 90.9%
Pass rate with the skill: 100%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/20/2026, 23:30 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-04

Prompt

We have thousands of frequently updated internal policy documents and need to answer user Q&A accurately. A team member suggested fine-tuning Llama-3 on all policy text so the model memorizes the documents. What architecture should we use instead for dynamic policy knowledge?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run100

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+184% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 37 days ago

Other measured skills in the registry, with their headline benchmark lift.