When a post is reported or flagged for review, a human moderator considers whether it violates a platform’s rules. Moderators apply those rules to individual cases, where context can make an apparently simple decision difficult. section 3.4 section 3.2
A moderator’s work sits within a larger system:
- Automated systems may remove content or flag uncertain cases for human review; user reports can also send content to a review queue. section 3.4
- Reviewers may decide to remove content or sanction a user. High volumes and pressure to work quickly can leave little time to consider context. section 3.4 section 3.5.2
- Repeated exposure to disturbing material can cause serious psychological strain. Outsourcing and performance pressure can compound the difficulty of the work. section 3.5.2
Why context matters
A rule against nudity can require exceptions: art may be allowed, as may breastfeeding. A historical image can also prompt a policy change. Moderators must apply definitions while handling cases whose meaning depends on context. section 3.2
Figure 3.2 An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.
The workflow shows how automated checks and user reports can lead to a shared queue for human decisions. section 3.4
