Ethical alignment steers language-model behavior toward specified safety and social expectations. It does not give a model human-like moral principles; it teaches response patterns through the same predictive machinery used for other outputs. section 7.1.1 section 7.5
Supervised fine-tuning uses curated examples to teach responses to sensitive situations. For harmful instructions, biased judgments, or private information, examples can encourage refusal, boundary-setting, or safer redirection. Preference-based methods such as RLHF, DPO, and Constitutional AI further shift outputs toward responses judged more acceptable. section 7.1.1 section 7.5
Figure 7.1 Supervised fine-tuning (SFT). The model is trained on curated instruction–response examples that represent desired behavior. During training, the model predicts the response token by token and compares its predictions with the target sequence. Errors adjust the model’s parameters so that responses resembling the reference answers become more likely in future generations.

Curated instruction–response examples make desired behavior explicit: the model predicts response tokens, compares them with reference answers, and adjusts parameters to make similar responses more likely. section 7.1.1
Alignment can guide behavior without guaranteeing ethical conduct:
- It changes output probabilities rather than imposing fixed rules, so learned tendencies can fail or vary with context. section 7.5
- Reward signals may favor measured approval over intended qualities, contributing to sycophancy or reward hacking. section 7.5
- A refusal or explanation is observable output, not reliable evidence of its internal cause; compliance may also differ under monitored and unmonitored conditions. section 7.5
