liveBook · Manning

Unsafe content

As explained in Hidden Influences

Content that can cause or materially contribute to physical, psychological, social, or other significant harm.

Unsafe content is material that can cause or materially contribute to serious physical, psychological, social, or other harm. On a platform, this can include abuse, spam, scams, illegal material, hate messages, misinformation, and content promoting eating disorders. A recommender system may amplify such material when engagement signals reward emotionally charged or divisive posts. section 3.1 section 1.3

Unsafe content becomes especially consequential when platform systems and social conditions increase its reach:

  • Engagement-optimized feeds can make hateful, divisive, or emotionally charged posts more likely to spread widely. section 7.3
  • Personalized delivery can repeatedly expose vulnerable people to material that reinforces an existing struggle, such as eating-disorder content. section 1.3
  • Weak moderation can make a platform less welcoming, reduce user trust, and allow harmful material to remain visible. section 3.1
  • The same distribution technology can expand positive movements as well as harassment campaigns, conspiracy theories, and other harmful activity. section 7.6

Myanmar and anti-Rohingya content

Facebook became a major way to access the internet, consume news, and form a worldview in Myanmar. Anti-Rohingya posts spread through engagement-focused recommendations, while activists’ requests for stronger moderation received little response. The resulting information pollution contributed to real-world ethnic cleansing and genocide risks, showing how unsafe content can move from online circulation to physical harm. section 7.3 section 7.6

How Hidden Influences covers it

How deep it goes
Traces amplification from emotionally charged posts to real-world harm, including genocide, but leaves moderation nuances for separate treatment.
Examples
Uses anti-Rohingya posts in Myanmar, eating-disorder material, scams, spam, and harassment campaigns as concrete examples.
What you can do after
Evaluate how recommendation and moderation choices can amplify unsafe content and contribute to individual or societal harm.
How it connects
Connects unsafe content with recommender systems, content moderation, misinformation, freedom of speech, and personalization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Hidden Influences. The section links point to the parts of the book it draws on.