A newly uploaded post on a platform that prohibits nudity may be scanned automatically before anyone reports it. An algorithmic moderator is a computational system—often a machine-learning model trained to spot a specific violation of the platform’s terms of service—that evaluates content or users and may flag or remove content, or trigger a user ban. section 3.4 section 3.4.1
Automated moderation can act at different points in a platform’s review process:
- Proactive systems scan posts as they are created. They can remove content directly or send uncertain cases for human review. section 3.4 section 3.4.1
- Reactive review begins after a trigger, such as a user report. Reported and flagged content may enter a queue for human moderators, who make final decisions. section 3.4
- Automated ranking can help moderators prioritize a growing queue by predicted severity or potential impact, rather than by predicted value to users. section 3.4.1
Automation can scale to far more content than human review alone, but it is not error-free. Its decisions raise questions about bias, transparency, and users’ agency; whether failures affect some people more than others is also a concern. section 3.4 section 3.4.1
Figure 3.2 An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.
A workflow diagram shows automated scanning, removal or escalation to human review, and reports entering the same queue. section 3.4
