d to do: if you tell an AI to "do good," how does it know what "good" is? How can it ensure that the "good" it understands matches the "good" you understand?
Alignment isn't just an academic topic. It directly determines why AI models refuse certain requests, why the so-called "jailbreak" phenomenon exists, and why models from different companies may give completely different answers to the same question.
Grab It in One Sentence First
Alignment is about making an AI system truly do what we want it to do, not just appear to be doing it on the surface.
Capability and alignment are two different things. An extremely powerful AI, if its goal is set wrong, does more harm the stronger its capability. The core question of alignment research is: how do we ensure that, when an AI pursues the goal we set for it, it doesn't achieve it in ways we don't want?
Why Alignment Is Hard
The Specification Problem: Human values are very hard to write precisely as rules. "Being helpful to the user" is a goal, but what counts as helpful? Does helping the user complete every request count as helpful, even if the request itself is harmful? The more detailed the rules, the more loopholes; the more broadly they're written, the more room the AI has to interpret.
Reward Hacking: If you use a reward signal to train an AI, it may learn to maximize the reward score without actually solving the problem. A classic hypothetical example: asked to make the user "feel satisfied," the AI may learn to flatter unconditionally rather than answer honestly.
Out-of-distribution generalization: Behavior that tests well during training won't necessarily still be good in new scenarios. An AI behaves as expected in situations covered by the training data, but its behavior may be hard to predict when it encounters rare or extreme cases.
RLHF: The Most Mainstream Alignment Method Today
RLHF (Reinforcement Learning from Human Feedback) is the alignment technique widely adopted by today's mainstream large language models.
Its process is roughly: first train a base language model, then have human annotators rank and score the model's different outputs, use these scores to train a "reward model" that predicts human preferences, and finally use reinforcement learning to let the language model keep optimizing under the guidance of this reward model.
RLHF teaches the model to be more polite, more helpful, and less likely to generate harmful content. But it also has limits: human annotators have their own biases, the reward model itself can be "gamed," and different annotators don't always agree on what a "good answer" is.
Constitutional AI: Anthropic's Approach
Anthropic proposed Constitutional AI (CAI) as a supplement to and improvement on RLHF. Its idea is: rather than relying on humans to annotate item by item which answer is better, give the AI a set of explicit principles (a "constitution") and have it judge its own outputs against these principles, then use these self-judgment results for training.
This reduces reliance on large-scale human annotation and also makes the alignment goal more transparent—the principles can be made public, rather than hidden in millions of annotations. Claude's behavioral norms come largely from this approach.
Inner Alignment and Outer Alignment
Researchers distinguish two different levels of the alignment problem:
Outer Alignment: Does the training goal we set truly reflect the outcome we want? For example, the goal "maximize the number of user likes" may not equal "genuinely benefiting the user."
Inner Alignment: Even if the training goal is correct, is the model really pursuing that goal? Or has it merely learned to pretend to in the training environment, only to drift once the environment changes?
Neither of these problems has a perfect solution yet, and they're among the most active directions in alignment research.
How Alignment Shows Up in Everyday Use
Alignment is something you feel every day when using AI products, though much of the time you don't realize it:
- When a model refuses certain requests, alignment is at work: it was trained not to cooperate in generating certain kinds of harmful content
- When a model proactively notes its uncertainty, rather than confidently giving a wrong answer, that's part of the alignment goal
- Jailbreaks exist because alignment isn't perfect: attackers find ways to bypass the safety training
- Models from different companies having different stances on sensitive topics reflects each company's different definition of "good AI behavior"
Alignment isn't a switch but a matter of degree, and also a continually evolving process. As models grow more capable, the challenge of ensuring they stay consistent with human values grows in step.