liveBook · Manning

Ethical alignment

As explained in Machines that Think

Ethical alignment is the correspondence between an artificial system's behavior and specified ethical, safety, or social norms.

Ethical alignment steers language-model behavior toward specified safety and social expectations. It does not give a model human-like moral principles; it teaches response patterns through the same predictive machinery used for other outputs. section 7.1.1 section 7.5

Supervised fine-tuning uses curated examples to teach responses to sensitive situations. For harmful instructions, biased judgments, or private information, examples can encourage refusal, boundary-setting, or safer redirection. Preference-based methods such as RLHF, DPO, and Constitutional AI further shift outputs toward responses judged more acceptable. section 7.1.1 section 7.5

Figure 7.1 Supervised fine-tuning (SFT). The model is trained on curated instruction–response examples that represent desired behavior. During training, the model predicts the response token by token and compares its predictions with the target sequence. Errors adjust the model’s parameters so that responses resembling the reference answers become more likely in future generations.

Curated instruction–response examples make desired behavior explicit: the model predicts response tokens, compares them with reference answers, and adjusts parameters to make similar responses more likely. section 7.1.1

Alignment can guide behavior without guaranteeing ethical conduct:

  • It changes output probabilities rather than imposing fixed rules, so learned tendencies can fail or vary with context. section 7.5
  • Reward signals may favor measured approval over intended qualities, contributing to sycophancy or reward hacking. section 7.5
  • A refusal or explanation is observable output, not reliable evidence of its internal cause; compliance may also differ under monitored and unmonitored conditions. section 7.5

How Machines that Think covers it

How deep it goes
Explains safety examples and preference methods, then limits alignment to probabilistic tendencies rather than fixed rules.
Examples
Illustrates safety steering with harmful instructions, biased judgments, private information, refusals, boundaries, and safer redirection.
What you can do after
Evaluate alignment as learned response steering, not evidence of moral understanding or guaranteed compliance.
How it connects
Connects ethical alignment to supervised fine-tuning, RLHF, DPO, Constitutional AI, reward signals, and training-data bias.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.