Chapter 3: Content moderation
This introduction defines content moderation and highlights the challenges of applying platform policies, including contextual interpretation and legal limits.
After this chapter you will be able to
- Content moderation and policies. Explain what content moderation does and why user-generated platforms make it more complicated.
- Context in moderation decisions. Explain how the meaning of the same phrase can change with context in a moderation decision.
- Private platforms and speech. State the section's distinction between private platform content removal and a breach of free speech rights in the United States.
- Usable spaces and moderation. Explain why platforms moderate to keep their spaces usable and appealing.
- Platform values and rules. Explain how moderation can enforce a platform's values, audience needs, and rules, including rules beyond what the law requires.
- Moderation and customer loyalty. Explain the business case for moderation despite its costs, using the Twitter example as a possible illustration rather than proof.
- Contested moderation boundaries. Explain why many moderation decisions require difficult boundaries despite reasonable disagreement.
- Policy definitions and exceptions. Describe how the nudity example shows the need for detailed definitions and case-specific handling.
- Errors at platform scale. Explain how a very small moderation error rate can produce many incorrect decisions at platform scale, using the section's example.
- Illegal content and enforcement. Explain what makes content illegal and why platforms have strong incentives to remove it quickly.
- Jurisdictional differences in legality. Explain how the legality of content can differ between countries and what that means for platforms operating internationally.
- Platform content rules. Explain how platforms’ terms of service set rules for content beyond what the law prohibits.
- Removal of ToS violations. Describe how platforms respond to content that violates their terms of service and the role moderators play in removal decisions.
- Restricted content and reach. Explain what restricted content is and how a platform can limit its distribution while keeping it accessible.
- Unrestricted content category. Describe what distinguishes the remaining content category and how it relates to recommender systems.
- Proactive and reactive moderation. Distinguish proactive moderation from reactive moderation and explain the scale and reliability trade-off of automation.
- Review prioritization and brigading. Explain how report counts and potential spread can help prioritize human review, and why report counts can be manipulated.
- Moderation queue prioritization. Explain how recommender systems can prioritize flagged content for human review and why prioritization matters.
- Automated moderation authority. Describe the range of actions an automated moderation system can take when it flags content.
- Bias and model opacity. Explain how an inaccurate toxicity score can affect amplification and why automated decisions can be difficult to understand.
- Pre-posting moderation nudges. Describe how pre-moderation nudges give users a chance to reconsider potentially harmful posts and what happened in the Twitter experiment.
- Post removal and user consequences. Describe post removal and the additional consequences a poster may face for violating a platform’s terms of service.
- Shadowbanning and restricted content. Explain how platforms can limit the spread of restricted content without deleting it, and why this can be called shadowbanning.
- Limited platform appeal rights. Describe what users are told after a moderation decision and how their opportunity to appeal differs from a judicial process in a democratic country.
- Human review of automation. Explain how a platform may review an automated moderation decision and why such review can be useful.
- Scale and moderation errors. Explain why even a small moderation error rate can produce many mislabeled examples on social media.
- Unprecedented Political Moderation Choices. Explain why a platform's choices in a disputed election can be consequential without an obvious right answer.
- Trump Ban and Accounts. Trace Twitter's response to Trump's accounts after January 6, including the distinction made for institutional accounts.
- Psychological harm to moderators. Explain how repeated exposure to disturbing content affects moderators at work and in their personal lives, and why available support may not be enough.
- Speed pressure and context. Explain how moderation quotas and job pressure can affect the consideration of context and the resulting decisions.
- Cultural differences and guidelines. Explain how moderators’ cultural norms can influence decisions and how platforms can reduce inconsistent interpretation.
- Reclaimed speech and identity. Explain why judging reclaimed speech depends on who uses it and the context of the interaction.
- Misinformation and disinformation. Distinguish misinformation from disinformation by the intent involved, and describe how platforms’ approaches to such content can vary.
- Nuanced moderation judgments. Explain why intent, misleading truths, and political consequences make moderation decisions difficult to define and apply.
- Country-specific legal moderation. Explain why multinational platforms may need to moderate the same content differently across countries, using the German example.
- Political speech compliance tension. Explain the tension between complying with political speech laws and protecting free expression, business interests, and employees, using the Twitter example.
- Adaptation to emerging content. Explain why moderation policies and moderators must adapt as social media discourse changes, using the response to ISIS execution videos as an example.
- User evasion of moderation. Explain how users adapt words or use symbols to evade moderation while communicating a meaning to other people.
- Speed and reporting costs. Explain how the desire for quick moderation and the resources needed for reports discourage platforms from explaining decisions.
- Consistency and platform discretion. Explain why inconsistent moderation and the desire for discretion can discourage platforms from providing reports.
- Liability for hosted content. Explain how Section 230 treats platforms hosting third-party content and how this differs from responsibility for content in traditional media.
- Moderation rights and limits. Describe platforms’ ability to remove objectionable content, including constitutionally protected speech, and the limits on their legal protections.
- Speech and reach distinction. Explain the section’s distinction between freedom of speech and freedom of reach in platform moderation.
What it builds on
- Heavy ranking (chapter 2) · Recommender systems rank content according to a predicted value.
- Recommendation as amplification (chapter 1) · Recommendation affects how widely content is amplified.
- Filtering (chapter 2) · Filtering can remove an item before ranking.
Section by section
Introduction
This introduction defines content moderation and highlights the challenges of applying platform policies, including contextual interpretation and legal limits.
- Content moderation and policies: Explain what content moderation does and why user-generated platforms make it more complicated.
Why it matters: This explains what moderators are trying to manage and why the task can be difficult. - Context in moderation decisions: Explain how the meaning of the same phrase can change with context in a moderation decision.
Why it matters: Context can change whether an apparently abusive message actually warrants moderation. - Private platforms and speech: State the section's distinction between private platform content removal and a breach of free speech rights in the United States.
Why it matters: This distinction helps avoid treating every platform removal as a free speech rights violation.
Key terms: content moderation, user-generated content, content policies, content moderators, abusive, free speech rights, private companies
3.1 Why platforms moderate
This section explains why platforms moderate content to keep spaces usable, enforce their values and rules, and retain users.
- Usable spaces and moderation: Explain why platforms moderate to keep their spaces usable and appealing.
Why it matters: Moderation can prevent a platform from becoming a place users no longer want to visit. - Platform values and rules: Explain how moderation can enforce a platform's values, audience needs, and rules, including rules beyond what the law requires.
Why it matters: A platform's moderation choices can shape the kind of community it wants to host. - Moderation and customer loyalty: Explain the business case for moderation despite its costs, using the Twitter example as a possible illustration rather than proof.
Why it matters: Moderation can be an investment in retaining users, not just an expense.
Key terms: content moderation, bad actors, post removals, growth function, long-term value, loyal customers
3.2 The impossibility of moderation at scale
This section explains why moderation is difficult to do well: decisions involve contested values and detailed rules, while platform-scale volumes turn even small error rates into many mistakes.
- Contested moderation boundaries: Explain why many moderation decisions require difficult boundaries despite reasonable disagreement.
Why it matters: This explains why moderation cannot always rely on an agreed definition of what should stay. - Policy definitions and exceptions: Describe how the nudity example shows the need for detailed definitions and case-specific handling.
Why it matters: A simple-sounding rule can require many detailed judgments in practice. - Errors at platform scale: Explain how a very small moderation error rate can produce many incorrect decisions at platform scale, using the section's example.
Why it matters: A low error percentage can still mean a large number of people or posts are affected.
Key terms: content moderation, human values, personal sensibilities, nudity, special cases, definitions, accuracy, failure rate, moderation enforcement
3.3 Content taxonomy
This section introduces a platform-centered taxonomy of user-generated content and its moderation needs.
3.3.1 Illegal content
This section explains illegal content, platforms’ incentives to remove it, and how its legal status varies across jurisdictions.
- Illegal content and enforcement: Explain what makes content illegal and why platforms have strong incentives to remove it quickly.
Why it matters: Legal enforcement gives platforms a strong reason to remove unlawful content promptly. - Jurisdictional differences in legality: Explain how the legality of content can differ between countries and what that means for platforms operating internationally.
Why it matters: The same content can pose different legal issues depending on where a platform operates.
Key terms: illegal content, jurisdictions, enforcement actions, jurisdiction, local regulations, freedom of speech and expression
3.3.2 Content in violation of the terms of service
This section explains how platforms use their terms of service to set content rules beyond the law and remove violations.
- Platform content rules: Explain how platforms’ terms of service set rules for content beyond what the law prohibits.
Why it matters: Platform rules can restrict content even when the law does not prohibit it. - Removal of ToS violations: Describe how platforms respond to content that violates their terms of service and the role moderators play in removal decisions.
Why it matters: Understanding who participates in removal decisions helps explain how platform rules are applied.
Key terms: Terms of Service, ToS, content moderators, human and automated moderators
3.3.3 Restricted content
This section explains how platforms can limit the reach of questionable content without removing it or treating it as illegal or a terms-of-service violation.
- Restricted content and reach: Explain what restricted content is and how a platform can limit its distribution while keeping it accessible.
Why it matters: This distinction shows that allowing content and actively spreading it are different platform choices.
Key terms: restricted content, terms of service, free speech is not the same as free reach, out-of-network suggestions, toxic content
3.3.4 The rest of the content
This section describes unrestricted content and its place in platform recommendations.
- Unrestricted content category: Describe what distinguishes the remaining content category and how it relates to recommender systems.
Why it matters: This category shows that not all platform content needs special moderation handling before recommendation.
Key terms: content not restricted, recommender systems
3.4 How content moderation is done
This section contrasts proactive and reactive content moderation and explains ways to prioritize review at platform scale.
- Proactive and reactive moderation: Distinguish proactive moderation from reactive moderation and explain the scale and reliability trade-off of automation.
Why it matters: The timing and scalability of review shape how platforms handle vast amounts of content. - Review prioritization and brigading: Explain how report counts and potential spread can help prioritize human review, and why report counts can be manipulated.
Why it matters: Prioritization helps scarce reviewers focus their attention, but some signals can be exploited. - Outsourcing human moderation: Explain why platforms outsource some human moderation work and where that work is often sent.
Why it matters: The cost of human review affects where moderation work is performed and under what protections.
Key terms: proactive moderation, reactive moderation, algorithmic moderators, prioritization, brigading, potential spread, human moderators, outsourcing, Global Majority
3.4.1 The use of automation in content moderation
This section explains how automation prioritizes and moderates content, including the benefits and risks of automated decisions and pre-posting nudges.
- Moderation queue prioritization: Explain how recommender systems can prioritize flagged content for human review and why prioritization matters.
Why it matters: Prioritization helps moderators focus limited attention where it may prevent the most harm. - Automated moderation authority: Describe the range of actions an automated moderation system can take when it flags content.
Why it matters: The degree of automation determines whether a human reviews a flag before a moderation action occurs. - Bias and model opacity: Explain how an inaccurate toxicity score can affect amplification and why automated decisions can be difficult to understand.
Why it matters: Automated moderation can quietly restrict some content while making the reason difficult to explain. - Pre-posting moderation nudges: Describe how pre-moderation nudges give users a chance to reconsider potentially harmful posts and what happened in the Twitter experiment.
Why it matters: A warning before publication can reduce harmful sharing without waiting for a post to be seen and removed. - Nudge false positive risks: Explain how a false-positive nudge on controversial content can create an impression of platform bias and how reported similar content might trigger it.
Why it matters: A mistaken warning can affect users’ trust, especially in a heated political debate.
Key terms: queue, recommender systems, predicted severity or impact, auto-flag, auto-moderators, human moderators, toxicity threshold, bias, transparency, amplification
3.4.2 Decisions and consequences
This section describes how moderation decisions can remove content, restrict users, or limit content visibility without users realizing it.
- Post removal and user consequences: Describe post removal and the additional consequences a poster may face for violating a platform’s terms of service.
Why it matters: Moderation can affect both an individual post and what its author is allowed to do. - Shadowbanning and restricted content: Explain how platforms can limit the spread of restricted content without deleting it, and why this can be called shadowbanning.
Why it matters: A post can lose reach through moderation even when it remains undeleted and its author is unaware of the restriction. - Platform design and moderation: Explain why a moderation approach may differ across types of platform.
Why it matters: A moderation decision has to fit the kind of platform where it is applied.
Key terms: removal, terms of service, ban, restricted content, filtering, shadowbanning, amplification, platform design, content moderation
3.4.3 Appeals
This section explains the limited availability of appeals and the possible human review of automated moderation decisions.
- Limited platform appeal rights: Describe what users are told after a moderation decision and how their opportunity to appeal differs from a judicial process in a democratic country.
Why it matters: A moderation decision can affect a user without giving them a right to challenge it. - Human review of automation: Explain how a platform may review an automated moderation decision and why such review can be useful.
Why it matters: Human review offers a way to reconsider automated decisions that may be biased or miss nuance.
Key terms: appeal, moderator, decision, human review, algorithmic decisions, bias, nuanced decisions
3.5 Challenges in content moderation
This section explains how the scale of social media content magnifies moderation errors.
- Scale and moderation errors: Explain why even a small moderation error rate can produce many mislabeled examples on social media.
Why it matters: This helps explain why moderation remains difficult even when errors are relatively rare.
Key terms: scale, percentage error, mislabeled examples
3.5.1 Lack of precedents
The section uses examples of unexpected platform use and unprecedented political situations to illustrate difficult moderation decisions.
- Unexpected Uses of Features: Explain how unexpected uses of platform features can create moderation problems, using the Tony the Tiger account as an example.
Why it matters: Unexpected behavior can create moderation problems that existing rules do not clearly settle. - Unprecedented Political Moderation Choices: Explain why a platform's choices in a disputed election can be consequential without an obvious right answer.
Why it matters: Moderation decisions can shape political visibility even when no clear precedent points to the right choice. - Trump Ban and Accounts: Trace Twitter's response to Trump's accounts after January 6, including the distinction made for institutional accounts.
Why it matters: The case shows how a platform had to apply rules while facing an unprecedented decision about a political leader and institutional accounts.
Key terms: content moderation, terms of service, blocking, official recognition, visibility, trade-offs, precedent, ban evasion, institutional account, incitement of violence
3.5.2 Impact on moderators
This section explains the psychological, employment, and decision-making pressures faced by content moderators.
- Psychological harm to moderators: Explain how repeated exposure to disturbing content affects moderators at work and in their personal lives, and why available support may not be enough.
Why it matters: Moderation can protect users while imposing serious psychological costs on the people doing it. - Outsourcing and unequal conditions: Describe how subcontracting affects moderators' pay and benefits, and why some people compare outsourcing the work to the Global South to colonialism.
Why it matters: Who performs moderation affects who bears its costs and who receives the benefits of tech work. - Speed pressure and context: Explain how moderation quotas and job pressure can affect the consideration of context and the resulting decisions.
Why it matters: Pressure to process more content can make it harder to reach decisions that account for context.
Key terms: content moderators, PTSD, burnout, turnover, mental health support, subcontracted, third-party firms, Global South, colonialism, quota
3.5.3 Cultural context
This section explains how cultural context complicates content moderation and why guidelines and local understanding matter.
- Cultural differences and guidelines: Explain how moderators’ cultural norms can influence decisions and how platforms can reduce inconsistent interpretation.
Why it matters: A rule that seems clear in one culture can lead to unintended censorship in another. - Reclaimed speech and identity: Explain why judging reclaimed speech depends on who uses it and the context of the interaction.
Why it matters: The same word can carry different meanings depending on who is using it. - Limits of local understanding: Describe why cultural context can be difficult for platforms to establish and what the Myanmar example illustrates.
Why it matters: Missing local context can have consequences far beyond an individual moderation decision.
Key terms: cultural context, moderation guidelines, NSFW, reclaimed speech, in-group interaction, local context, automated translation, Myanmar
3.5.4 Nuanced decisions
This section explains why moderating false or misleading content involves difficult definitions and nuanced judgments about intent, truth, and politics.
- Misinformation and disinformation: Distinguish misinformation from disinformation by the intent involved, and describe how platforms’ approaches to such content can vary.
Why it matters: The distinction helps explain why platforms may treat inaccurate content differently. - Nuanced moderation judgments: Explain why intent, misleading truths, and political consequences make moderation decisions difficult to define and apply.
Why it matters: These complications show why a simple rule against false content cannot settle every moderation decision.
Key terms: misinformation, disinformation, political climate, intent, cherry-picking statistics, moderation decisions
3.5.5 Legal landscape
This section explains how differing national laws complicate platform moderation, especially when legal compliance conflicts with free expression and employee safety.
- Country-specific legal moderation: Explain why multinational platforms may need to moderate the same content differently across countries, using the German example.
Why it matters: National differences can require a platform to apply different moderation rules to different audiences. - Political speech compliance tension: Explain the tension between complying with political speech laws and protecting free expression, business interests, and employees, using the Twitter example.
Why it matters: Political speech decisions can put a platform’s stated principles at odds with legal and human consequences.
Key terms: illegal content, moderation funnel, political speech, free expression, free speech, legal action
3.5.6 Evolving field
This section explains why changing content, language, and user behavior require moderation policies and moderators to adapt.
- Adaptation to emerging content: Explain why moderation policies and moderators must adapt as social media discourse changes, using the response to ISIS execution videos as an example.
Why it matters: Moderation cannot rely on a fixed set of policies when new kinds of harmful content emerge. - Changing and contextual meanings: Describe how the coconut example illustrates that a word can acquire a popular new use while also having an offensive meaning elsewhere.
Why it matters: A word's meaning in one conversation may differ sharply from its meaning elsewhere. - User evasion of moderation: Explain how users adapt words or use symbols to evade moderation while communicating a meaning to other people.
Why it matters: User workarounds add another moving target for moderation.
Key terms: policies, moderators, coconut, content moderation, reach
3.6 Lack of transparency
This section explains why platforms generally do not provide public explanations for moderation decisions.
- Speed and reporting costs: Explain how the desire for quick moderation and the resources needed for reports discourage platforms from explaining decisions.
Why it matters: Explaining every decision could make moderation slower and more resource-intensive. - Consistency and platform discretion: Explain why inconsistent moderation and the desire for discretion can discourage platforms from providing reports.
Why it matters: Transparency could expose inconsistent decisions and constrain platforms’ room for judgment.
Key terms: transparency, moderation decisions, report, consistent, reports, decisions reversed, best judgment, exceptional cases
3.7 Legal basis
This section explains the US legal basis for platform liability, content moderation, and the distinction between freedom of speech and freedom of reach.
- Liability for hosted content: Explain how Section 230 treats platforms hosting third-party content and how this differs from responsibility for content in traditional media.
Why it matters: This distinction helps explain why platforms can host users’ posts without taking publisher responsibility for each one. - Moderation rights and limits: Describe platforms’ ability to remove objectionable content, including constitutionally protected speech, and the limits on their legal protections.
Why it matters: Legal protection for speech does not mean a platform must host it, and platform protections do not cover every kind of content. - Speech and reach distinction: Explain the section’s distinction between freedom of speech and freedom of reach in platform moderation.
Why it matters: The distinction clarifies why limiting a post’s reach is not the same as denying someone the freedom to speak.
Key terms: Section 230, publisher, third-party content, distributor, constitutionally protected, objectionable, content moderation, freedom of speech, freedom of reach, First Amendment
3.8 Summary
Where this is used later
- The firehose problem: scale and speed (chapter 8) · The difficulty of moderating content at scale.
- Meme wars (chapter 7) · Moderation decisions and consequences
- Platforms: power and duty over users (chapter 10) · Decisions about content and their consequences provide context for platforms’ direct power.
- Power over users, and lack of contestability (chapter 10) · Moderation decisions and their consequences.
- Platform portability (chapter 10) · The challenges platforms face when making content moderation decisions.
- The risk of information pollution: genocide (chapter 7) · Content moderation depends on cultural context.
- Homogenization (chapter 7) · The broader problem of limited platform transparency.
Concepts explained in this chapter
- content moderationThe review, filtering, labeling, or removal of user-generated or platform-hosted content according to defined rules.
- moderatorA person who reviews user-generated content and decides whether it complies with a platform's rules.
- algorithmic moderatorA computational system that automatically evaluates user content or users and can flag, rank, remove, or restrict them under moderation rules.
- proactive moderationContent moderation performed automatically when content is created or before a triggering user report or other event.
- free speechThe civil liberty to communicate opinions, ideas, and information without unjustified restriction.
- precedentA prior case or established decision that provides guidance for resolving a later comparable issue.
- moderation policyA set of platform rules defining which content is permitted, restricted, or removed.
- nudgeA user-interface intervention that prompts a person to reconsider an intended action without directly preventing it.
- social normA shared expectation or rule governing behavior within a group or society.
- online platformAn internet-based service that provides digital functionality and connects users, content, or transactions.
- unsafe contentContent that can cause or materially contribute to physical, psychological, social, or other significant harm.
- error rateThe proportion of predictions that are incorrect among all predictions made by a system.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
