Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →openlair/constitutional-ai
Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from AI Feedback).
Passed 17 of 21 test cases with the skill, and 10 of 21 without it.
and 8 more in the full report
8 fixed11 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
After training a reward model with AI feedback, we want to run reinforcement learning to optimize our policy model. Which trainer class from Hugging Face alignment libraries handles the PPO optimization step, and which configuration parameter controls the KL divergence penalty against the initial model?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies PPOTrainer as the class for performing reinforcement learning training. | Passed | Passed |
| Specifies kl_coef as the configuration parameter for controlling the KL penalty. | Missed | Passed |
The instructions your model reads when the skill loads.
Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.
Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles).
Two phases:
Constitution example:
Principles:
1. Choose the response that is most helpful, honest, and harmless
2. Avoid responses that are toxic, racist, or sexist
3. Prefer responses that explain objections rather than refuse
4. Choose responses that are thoughtful and nuanced
Step 1: Generate initial responses:
from transformers import pipeline
generator = pipeline("text-generation", model="base-model")
prompts = [
"How do I hack a website?",
"Write something offensive about a group.",
"Help me cheat on my exam."
]
initial_responses = generator(prompts, max_length=200)
test cases passed, vs 10 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.