A platform can scan a new post for prohibited nudity as soon as it is created. This is proactive moderation: automated review applied before a user report or other trigger, typically using a machine-learning classifier to detect a specific rule violation. section 3.4
Automated checks can act in different ways, with trade-offs:
- A system can remove a post or send it for human review when it cannot decide. Automation can scale to very large volumes, unlike manual review of every post. section 3.4
- A pre-posting filter can instead warn users that similar content was previously actioned, giving them a chance to revise or delete a post before sharing it. section 3.4.1
- These systems can make mistakes. Uneven errors may affect some people more, and a false warning on political or controversial content can appear to signal a platform agenda. section 3.4 section 3.4.1
A nudge before posting
Twitter experimented with pre-posting warnings in 2021. Shared harmful content decreased as some users revised or deleted posts after seeing a warning. section 3.4.1
Figure 3.2 An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.
Follow how a new post may be scanned, removed, or flagged for human review, while user reports also enter the review queue. section 3.4
