liveBook · Manning

Chapter 3: Content moderation

What you will learn in chapter 3 of Hidden Influences

This introduction defines content moderation and highlights the challenges of applying platform policies, including contextual interpretation and legal limits.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction defines content moderation and highlights the challenges of applying platform policies, including contextual interpretation and legal limits.

Key terms: content moderation, user-generated content, content policies, content moderators, abusive, free speech rights, private companies

3.1 Why platforms moderate

This section explains why platforms moderate content to keep spaces usable, enforce their values and rules, and retain users.

Key terms: content moderation, bad actors, post removals, growth function, long-term value, loyal customers

3.2 The impossibility of moderation at scale

This section explains why moderation is difficult to do well: decisions involve contested values and detailed rules, while platform-scale volumes turn even small error rates into many mistakes.

Key terms: content moderation, human values, personal sensibilities, nudity, special cases, definitions, accuracy, failure rate, moderation enforcement

3.3 Content taxonomy

This section introduces a platform-centered taxonomy of user-generated content and its moderation needs.

3.3.1 Illegal content

This section explains illegal content, platforms’ incentives to remove it, and how its legal status varies across jurisdictions.

Key terms: illegal content, jurisdictions, enforcement actions, jurisdiction, local regulations, freedom of speech and expression

3.3.2 Content in violation of the terms of service

This section explains how platforms use their terms of service to set content rules beyond the law and remove violations.

Key terms: Terms of Service, ToS, content moderators, human and automated moderators

3.3.3 Restricted content

This section explains how platforms can limit the reach of questionable content without removing it or treating it as illegal or a terms-of-service violation.

Key terms: restricted content, terms of service, free speech is not the same as free reach, out-of-network suggestions, toxic content

3.3.4 The rest of the content

This section describes unrestricted content and its place in platform recommendations.

Key terms: content not restricted, recommender systems

3.4 How content moderation is done

This section contrasts proactive and reactive content moderation and explains ways to prioritize review at platform scale.

Key terms: proactive moderation, reactive moderation, algorithmic moderators, prioritization, brigading, potential spread, human moderators, outsourcing, Global Majority

3.4.1 The use of automation in content moderation

This section explains how automation prioritizes and moderates content, including the benefits and risks of automated decisions and pre-posting nudges.

Key terms: queue, recommender systems, predicted severity or impact, auto-flag, auto-moderators, human moderators, toxicity threshold, bias, transparency, amplification

3.4.2 Decisions and consequences

This section describes how moderation decisions can remove content, restrict users, or limit content visibility without users realizing it.

Key terms: removal, terms of service, ban, restricted content, filtering, shadowbanning, amplification, platform design, content moderation

3.4.3 Appeals

This section explains the limited availability of appeals and the possible human review of automated moderation decisions.

Key terms: appeal, moderator, decision, human review, algorithmic decisions, bias, nuanced decisions

3.5 Challenges in content moderation

This section explains how the scale of social media content magnifies moderation errors.

Key terms: scale, percentage error, mislabeled examples

3.5.1 Lack of precedents

The section uses examples of unexpected platform use and unprecedented political situations to illustrate difficult moderation decisions.

Key terms: content moderation, terms of service, blocking, official recognition, visibility, trade-offs, precedent, ban evasion, institutional account, incitement of violence

3.5.2 Impact on moderators

This section explains the psychological, employment, and decision-making pressures faced by content moderators.

Key terms: content moderators, PTSD, burnout, turnover, mental health support, subcontracted, third-party firms, Global South, colonialism, quota

3.5.3 Cultural context

This section explains how cultural context complicates content moderation and why guidelines and local understanding matter.

Key terms: cultural context, moderation guidelines, NSFW, reclaimed speech, in-group interaction, local context, automated translation, Myanmar

3.5.4 Nuanced decisions

This section explains why moderating false or misleading content involves difficult definitions and nuanced judgments about intent, truth, and politics.

Key terms: misinformation, disinformation, political climate, intent, cherry-picking statistics, moderation decisions

3.5.5 Legal landscape

This section explains how differing national laws complicate platform moderation, especially when legal compliance conflicts with free expression and employee safety.

Key terms: illegal content, moderation funnel, political speech, free expression, free speech, legal action

3.5.6 Evolving field

This section explains why changing content, language, and user behavior require moderation policies and moderators to adapt.

Key terms: policies, moderators, coconut, content moderation, reach

3.6 Lack of transparency

This section explains why platforms generally do not provide public explanations for moderation decisions.

Key terms: transparency, moderation decisions, report, consistent, reports, decisions reversed, best judgment, exceptional cases

3.7 Legal basis

This section explains the US legal basis for platform liability, content moderation, and the distinction between freedom of speech and freedom of reach.

Key terms: Section 230, publisher, third-party content, distributor, constitutionally protected, objectionable, content moderation, freedom of speech, freedom of reach, First Amendment

3.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: Anatomy of a recommender system Chapter 4: Algorithmic amplification: Measuring RecSys →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.