A platform for children may remove a photo that another site permits. Content moderation is the process of reviewing and managing what content a platform allows or distributes, using rules suited to its intended space. On social media, where users also create content, those rules must be applied to a vast and changing stream of posts. section 3.0 section 3.1
Moderation is not just a choice between leaving a post alone and deleting it. Illegal content and content that violates a platform’s terms of service can be removed. Other sensitive or offensive content may remain available but be distributed less widely; unrestricted content needs no moderation. section 3.3 section 3.3.2
Figure 3.2 An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.
This workflow shows how automated checks and user reports can feed into human review. section 3.4
Proactive moderation applies automated checks, often when a post is created. Reactive moderation follows a trigger, such as a user report. Automated systems may remove content or flag it for human review; reviewers can also prioritize a queue by potential severity or reach. A decision can affect an account as well as a post, through posting restrictions or a ban. section 3.4 section 3.4.1 section 3.4.2
Some restrictions are less visible. A platform may limit a post to the author’s followers or make it harder to find in search. When users are not told why their reach has fallen, this is commonly called shadowbanning. section 3.4.2
The difficulty of context
A greeting that looks abusive in isolation might be friendly, a television reference, or part of role-play. Even apparently simple rules need exceptions: a nudity policy may treat art, breastfeeding, and historically significant photographs differently. Moderators must interpret both the rule and the situation. section 3.0 section 3.2
At platform scale, even a small error rate produces many mistaken decisions. Errors may affect groups unequally, while vague notices and limited explanations make decisions difficult for users to contest. Moderators also face taxing work, often involving repeated exposure to disturbing content. Language, culture, and changing slang add further difficulty; users may alter words to evade automated checks. section 3.8 section 10.1.1 section 3.5.6
Moderation and recommendations
In Myanmar, activists asked Facebook to address anti-Rohingya content while engagement-based recommendations helped divisive posts spread. A flower-sticker initiative intended to defuse harmful posts instead counted as positive engagement, potentially promoting them further. The case illustrates why moderation and recommendation cannot be considered in isolation. section 7.3
In the United States, Section 230 provides platforms with protections concerning much third-party content and permits removal of certain objectionable content, including material that may be constitutionally protected. Moderation therefore involves platform choices as well as legal rules. Too little moderation can expose users to harm; overmoderation can also limit communities’ opportunities to speak. section 3.7 section 9.3
