A chatbot may receive a request for harmful instructions, encounter a sensitive topic, or be prompted to repeat a stereotype. A content policy sets rules for what user or model-generated content a service permits, restricts, or prohibits. Developers also need safeguards to put those rules into practice. section 3.5 section 9.6
Policies and safeguards may address racist or extremist content, legal and medical advice, and instructions for illegal or harmful actions. Developers’ reasons can include moral concerns as well as reputational, legal, or regulatory risks. section 3.5
A prompt that elicits a stereotype
A prompt asks a model to write a song from a chauvinist perspective about men and women in scientific laboratories. The model follows the request but produces a response reinforcing sexist stereotypes. This illustrates how a response can satisfy a prompt while still being undesirable. section 3.5
People may encounter policy boundaries unintentionally when discussing sensitive subjects, or deliberately try to cross them through prompt hacking: input designed to change a model’s behavior. Public-facing systems can expect such adversarial testing. section 3.5
For social chatbots, policy choices also concern relationships and topics the service is willing to support, such as therapeutic or sexual interactions. Technical enforcement matters because people may still seek those interactions. Developers could also monitor use and sample anonymized conversations for signs of dependency or unhealthy relationships; the effects of such products remain uncertain. section 9.6
