liveBook · Manning

Algorithmic moderator

As explained in Hidden Influences

A computational system that automatically evaluates user content or users and can flag, rank, remove, or restrict them under moderation rules.

A newly uploaded post on a platform that prohibits nudity may be scanned automatically before anyone reports it. An algorithmic moderator is a computational system—often a machine-learning model trained to spot a specific violation of the platform’s terms of service—that evaluates content or users and may flag or remove content, or trigger a user ban. section 3.4 section 3.4.1

Automated moderation can act at different points in a platform’s review process:

  • Proactive systems scan posts as they are created. They can remove content directly or send uncertain cases for human review. section 3.4 section 3.4.1
  • Reactive review begins after a trigger, such as a user report. Reported and flagged content may enter a queue for human moderators, who make final decisions. section 3.4
  • Automated ranking can help moderators prioritize a growing queue by predicted severity or potential impact, rather than by predicted value to users. section 3.4.1

Automation can scale to far more content than human review alone, but it is not error-free. Its decisions raise questions about bias, transparency, and users’ agency; whether failures affect some people more than others is also a concern. section 3.4 section 3.4.1

Figure 3.2 An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.

A workflow diagram shows automated scanning, removal or escalation to human review, and reports entering the same queue. section 3.4

How Hidden Influences covers it

How deep it goes
Distinguishes proactive scanning from reactive review and describes possible actions, trade-offs, and queue prioritization.
Examples
Uses nudity detection, report-driven review queues, follower-based impact prioritization, and toxicity scoring as examples.
What you can do after
Understand how automated systems scan, flag, remove, or restrict content and route uncertain cases to human review.
How it connects
Connects content moderation with machine learning, human review, user reporting, and recommender systems.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Hidden Influences. The section links point to the parts of the book it draws on.