liveBook · Manning

Proactive moderation

As explained in Hidden Influences

Content moderation performed automatically when content is created or before a triggering user report or other event.

A platform can scan a new post for prohibited nudity as soon as it is created. This is proactive moderation: automated review applied before a user report or other trigger, typically using a machine-learning classifier to detect a specific rule violation. section 3.4

Automated checks can act in different ways, with trade-offs:

  • A system can remove a post or send it for human review when it cannot decide. Automation can scale to very large volumes, unlike manual review of every post. section 3.4
  • A pre-posting filter can instead warn users that similar content was previously actioned, giving them a chance to revise or delete a post before sharing it. section 3.4.1
  • These systems can make mistakes. Uneven errors may affect some people more, and a false warning on political or controversial content can appear to signal a platform agenda. section 3.4 section 3.4.1

A nudge before posting

Twitter experimented with pre-posting warnings in 2021. Shared harmful content decreased as some users revised or deleted posts after seeing a warning. section 3.4.1

Figure 3.2 An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.

Follow how a new post may be scanned, removed, or flagged for human review, while user reports also enter the review queue. section 3.4

How Hidden Influences covers it

How deep it goes
Defines automated checks, illustrates removal and warning workflows, and discusses trade-offs without detailing classifier internals.
Examples
Uses nudity detection on new posts and Twitter’s 2021 pre-posting warning experiment.
What you can do after
Evaluate automation’s scalability alongside its error, fairness, and user-trust risks.
How it connects
Connects to reactive moderation, machine learning, toxicity scoring, human review, and false positives.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Hidden Influences. The section links point to the parts of the book it draws on.