Chapter 8: The unreasonable effectiveness of engagement
This introduction frames engagement-based recommender systems as socio-technical systems shaped by users, technology, and historical context.
After this chapter you will be able to
- Socio-technical context of engagement. Explain why understanding the effectiveness of engagement-based recommender systems requires more than examining their technology alone.
- User recommender feedback loop. Trace the feedback loop between a recommender system and its users.
- Fast and deliberate thinking. Distinguish System 1 and System 2 thinking and explain why people do not always fully activate System 2.
- Fast reactions and rage-baiting. Explain how fast thinking can shape social media reactions and describe rage-baiting as an example.
- Social comparison and highlights. Explain social comparison and how curated and recommended social media content can distort comparisons with other people's lives.
- Local views and majority illusion. Explain how network connections can make a minority opinion appear to be the majority view.
- Social identity and group bias. Explain how identifying with a group can affect perceptions of members and outsiders, even when group boundaries are arbitrary.
- Group pressure and conformity. Describe how group pressure can lead people to conform against their own judgment, using the line-matching experiment as an example.
- Gossip and moral outrage. Explain how gossip, moral outrage, and public shaming reinforce group norms, and describe how public shaming can intensify on social media.
- Novelty and infinite scroll. Explain how the search for novelty and infinite scroll can keep people engaged with a platform.
- Unpredictable rewards and persistence. Explain how unpredictable rewards can encourage persistent behavior, using the Skinner box example.
- Email and social media habits. Explain the slot-machine analogy for checking email and social media, including where the variability comes from.
- Versatility of engagement recommenders. Explain why engagement-based recommender systems can recommend different kinds of content and identify hidden relationships between content and people.
- Technology and belonging needs. Explain how recommender capabilities and design choices can interact with people’s need to belong.
- Preferences and algorithmic influence. Explain the feedback between users' preferences and recommended content, including why separating a real self from an algorithmic self is difficult.
- Interaction links and community creation. Trace how interactions can draw users into communities and explain why the process can work for newly emerging content.
- Pathways to harmful content. Trace how a benign interaction can lead a user to harmful community content and how a curious click can increase exposure.
- Community activation and normalization. Explain how connecting like-minded people can activate both supportive movements and harmful groups.
- In-Network and Out-of-Network content. Distinguish In-Network content from Out-of-Network content and explain how the latter expands a post’s audience.
- Context collapse across audiences. Explain context collapse using the contrast between distinct offline audiences and posts that travel beyond their intended group.
- Normalization of extreme content. Explain how extreme content mixed with ordinary feed content can come to seem more accepted and benign.
- Competition for limited attention. Explain how growth in content volume creates competition for visibility and can encourage attention-grabbing posts.
- Engagement exposure feedback loop. Trace how engagement and recommendation can reinforce each other, including beyond a user's network.
- Escalation of extreme content. Explain why competition to stand out can create an arms race of increasingly extreme content.
- Normalization of harmful content. Describe how promotion of increasingly extreme content can affect users' exposure and perceptions.
- Rabbit holing and autoplay. Explain what rabbit-holing means in social media and how autoplay can encourage it.
- Video recommendation feedback. Trace how a sequence starting with a cake recipe can drift toward very different and potentially extreme videos.
- Shared membership group pathways. Trace how recommendations based on shared group membership could take an organic-food enthusiast toward conspiracy groups.
- Content agnosticism and harm. Distinguish harmless niche discovery from the risk of a content-agnostic recommender treating harmful and legitimate recommendations alike.
- Data void exploitation. Explain how a data void can let a harmful group dominate search results, using the section’s example.
- Speed and repair gap. Explain why data voids can arise faster than people can address them and what the section says about fixing them.
- Popularity driven amplification. Trace the popularity feedback described in the section and explain why human supervision is needed.
- Preexisting Social Trends. Explain how the section situates recommender systems within existing social and political problems.
- Partisan Media and Polarization. Explain how the section connects changes in US media and political rhetoric to polarization.
- Socio Technical User Interaction. Explain why the section calls recommender systems socio-technical and how it characterizes users' responses.
What it builds on
- Feedback loops in recommender systems and social media (chapter 4) · Feedback loops in recommender systems and social media.
- How to train a machine learning model (chapter 2) · How a machine learning model is trained.
- Organizing online (chapter 7) · Examples of movements organizing online
- The rich-getting-richer effect (chapter 4) · The rich-getting-richer effect in recommender systems.
- Attention economy (chapter 6) · The attention economy model
- Restricted content (chapter 3) · The idea that harmful content can pose challenges for platforms.
- The impossibility of moderation at scale (chapter 3) · The difficulty of moderating content at scale.
- Decisions and consequences (chapter 3) · Platform decisions and their consequences for content.
Section by section
Introduction
This introduction frames engagement-based recommender systems as socio-technical systems shaped by users, technology, and historical context.
- Socio-technical context of engagement: Explain why understanding the effectiveness of engagement-based recommender systems requires more than examining their technology alone.
Why it matters: This perspective keeps us from treating technology alone as the explanation for its effects. - User recommender feedback loop: Trace the feedback loop between a recommender system and its users.
Why it matters: The loop shows how users and the recommendation technology continually influence each other.
Key terms: socio-technical, engagement-based recommender systems, feedback loop, validation, training data
8.1 The people
This section connects evolved human psychology and fast, automatic thinking to engagement with social media.
- Evolution and social media: Explain how the section connects the evolution of the human brain to social media engagement.
Why it matters: Social media engages tendencies that did not evolve specifically for online platforms. - Fast and deliberate thinking: Distinguish System 1 and System 2 thinking and explain why people do not always fully activate System 2.
Why it matters: The distinction helps explain why our reactions are not always deliberate. - Fast reactions and rage-baiting: Explain how fast thinking can shape social media reactions and describe rage-baiting as an example.
Why it matters: Recognizing a provocation can help explain why an immediate response may later be regretted.
Key terms: evolution, reward cycles, habit loops, System 1, System 2, rage-baiting, recommender systems
8.1.1 Group belonging
This section explains how group belonging shapes comparison, identity, conformity, and public shaming, especially on social media.
- Cooperation and group belonging: Explain the proposed connection between human survival, cooperation, and the continuing desire to belong to a group.
Why it matters: The proposed survival value of groups helps explain why belonging can influence us so strongly today. - Social comparison and highlights: Explain social comparison and how curated and recommended social media content can distort comparisons with other people's lives.
Why it matters: A feed of highlights can make an unfair comparison feel like a realistic view of other people's lives. - Local views and majority illusion: Explain how network connections can make a minority opinion appear to be the majority view.
Why it matters: What seems popular in your feed may not reflect what the wider population believes. - Social identity and group bias: Explain how identifying with a group can affect perceptions of members and outsiders, even when group boundaries are arbitrary.
Why it matters: Even seemingly trivial group boundaries can affect how fairly we view other people and their ideas. - Group pressure and conformity: Describe how group pressure can lead people to conform against their own judgment, using the line-matching experiment as an example.
Why it matters: The experiment illustrates how the pressure to fit in can affect even seemingly straightforward judgments. - Gossip and moral outrage: Explain how gossip, moral outrage, and public shaming reinforce group norms, and describe how public shaming can intensify on social media.
Why it matters: Public outrage can both enforce group expectations and attract engagement through harsher language.
Key terms: group belonging, cooperation, social comparison theory, sampling bias, engagement, majority illusion, local view, global view, social identity theory, in-group
8.1.2 The search for novelty
This section explains how novelty, low-friction scrolling, and unpredictable rewards can encourage repeated social media use.
- Novelty and infinite scroll: Explain how the search for novelty and infinite scroll can keep people engaged with a platform.
Why it matters: The interface makes it easy to keep seeking something new without making a deliberate decision to continue. - Unpredictable rewards and persistence: Explain how unpredictable rewards can encourage persistent behavior, using the Skinner box example.
Why it matters: Unpredictable rewards help explain why an activity can be repeated even when it often delivers little. - Reward anticipation and dopamine: Describe what the cited gambling experiment suggests about reward uncertainty, anticipation, and dopamine.
Why it matters: The prospect of a reward can itself be compelling, even before a reward arrives. - Email and social media habits: Explain the slot-machine analogy for checking email and social media, including where the variability comes from.
Why it matters: The analogy connects an everyday checking habit to the anticipation of an uncertain payoff.
Key terms: novelty, infinite scroll, System 1 thinking, System 2 thinking, Skinner box, reward, Schedules of Reinforcement, dopamine, intermittent reinforcement, reward probability
8.2 The technology
This section explains how engagement-based recommender systems’ versatility and interaction with human needs contribute to their influence.
- Versatility of engagement recommenders: Explain why engagement-based recommender systems can recommend different kinds of content and identify hidden relationships between content and people.
Why it matters: This helps explain why the same recommendation approach can operate across very different kinds of content. - Technology and belonging needs: Explain how recommender capabilities and design choices can interact with people’s need to belong.
Why it matters: It connects the technology’s capabilities to the human needs that can make it influential.
Key terms: engagement data, hidden correlations, recommender systems, communities, group belonging
8.2.1 Communities and like-minded users
This section explains how recommender systems connect and activate communities, shape tastes and language, and can spread both valuable and harmful content.
- Community discovery and support: Explain how recommender systems help people find communities and why those connections can be valuable.
Why it matters: Recommendations can make connection possible for people who are isolated in their local surroundings. - Preferences and algorithmic influence: Explain the feedback between users' preferences and recommended content, including why separating a real self from an algorithmic self is difficult.
Why it matters: Your feed can influence your tastes as well as reflect them. - Interaction links and community creation: Trace how interactions can draw users into communities and explain why the process can work for newly emerging content.
Why it matters: The same interaction-based connections can assemble many different communities without someone explicitly designing each one. - Community language and signaling: Describe how online communities use language to signal belonging and how dog-whistles convey meanings understood by an intended group.
Why it matters: Community language can both signal membership and convey meanings that outsiders may miss. - Pathways to harmful content: Trace how a benign interaction can lead a user to harmful community content and how a curious click can increase exposure.
Why it matters: A harmless initial interest can become a route to repeated exposure to damaging material. - Community activation and normalization: Explain how connecting like-minded people can activate both supportive movements and harmful groups.
Why it matters: The power to unite people can support social movements while also making harmful views seem more acceptable.
Key terms: communities, recommender systems, virtual support groups, feedback loop, real self, algorithmic self, socio-technical, community creation, interactions, links
8.2.2 Out-of-Network content
This section explains how Out-of-Network recommendations expand content’s reach and can contribute to context collapse, self-consciousness, and the normalization of extreme content.
- In-Network and Out-of-Network content: Distinguish In-Network content from Out-of-Network content and explain how the latter expands a post’s audience.
Why it matters: This distinction explains how a post can reach people beyond the audience its author had in mind. - Context collapse across audiences: Explain context collapse using the contrast between distinct offline audiences and posts that travel beyond their intended group.
Why it matters: A post that makes sense to one group can be misunderstood when recommendations carry it elsewhere. - Loss of private self: Explain how an unbounded audience can affect a person’s self-presentation and self-image.
Why it matters: The possibility of a worldwide audience can affect how people present and value themselves. - Normalization of extreme content: Explain how extreme content mixed with ordinary feed content can come to seem more accepted and benign.
Why it matters: The surrounding feed can make harmful material look more socially accepted than it is.
Key terms: In-Network, Out-of-Network, engaging, context collapse, loss of the private self, social anxiety, engagements, Out-of-Network recommendations, extreme content, normalizing
8.2.3 Feedback loops and extreme content
This section explains how competition for attention and engagement feedback can encourage increasingly extreme content and normalize harmful material.
- Competition for limited attention: Explain how growth in content volume creates competition for visibility and can encourage attention-grabbing posts.
Why it matters: Limited attention helps explain why creators may resort to sensational content. - Engagement exposure feedback loop: Trace how engagement and recommendation can reinforce each other, including beyond a user's network.
Why it matters: This loop explains how initial engagement can lead to much wider exposure. - Escalation of extreme content: Explain why competition to stand out can create an arms race of increasingly extreme content.
Why it matters: The changing baseline helps explain why sensational content can escalate over time. - Normalization of harmful content: Describe how promotion of increasingly extreme content can affect users' exposure and perceptions.
Why it matters: A gradual shift in recommendations can change what harmful material users encounter and consider normal.
Key terms: System 1 thinking, clickbait, FOMO, engagement, Out-of-Network recommendation, feedback loop, extreme content, baseline, arms race, harmful content
8.2.4 Rabbit holing
This section explains how recommendation feedback can gradually lead users from benign interests toward harmful content and communities.
- Rabbit holing and autoplay: Explain what rabbit-holing means in social media and how autoplay can encourage it.
Why it matters: Recognizing gradual drift helps explain how a user can end up far from their original interest. - Video recommendation feedback: Trace how a sequence starting with a cake recipe can drift toward very different and potentially extreme videos.
Why it matters: A sequence of individually plausible recommendations can change a user's viewing trajectory. - Shared membership group pathways: Trace how recommendations based on shared group membership could take an organic-food enthusiast toward conspiracy groups.
Why it matters: Connections between communities can redirect recommendations even when the original interest is harmless. - Content agnosticism and harm: Distinguish harmless niche discovery from the risk of a content-agnostic recommender treating harmful and legitimate recommendations alike.
Why it matters: The danger is not niche interests themselves but a recommendation process that does not judge their differing impacts.
Key terms: rabbit-holing, autoplay, baseline shifts, feedback loop, popular content, Facebook groups’ recommendations, shared membership, content-agnostic, niche, value judgments
8.2.5 The firehose problem: scale and speed
This section explains how rapidly emerging data voids, evasive language, and popularity-driven recommendations can outpace human responses to harmful content.
- Data void exploitation: Explain how a data void can let a harmful group dominate search results, using the section’s example.
Why it matters: A search result can look prominent because few others address the term, not because it is trustworthy. - Speed and repair gap: Explain why data voids can arise faster than people can address them and what the section says about fixing them.
Why it matters: The speed mismatch helps explain why harmful results can circulate before a coordinated response catches up. - Algospeak and guardrails: Describe how new wording can help content evade platform guardrails, using the section’s example.
Why it matters: Changing language can let content continue spreading while platforms and researchers try to catch up. - Popularity driven amplification: Trace the popularity feedback described in the section and explain why human supervision is needed.
Why it matters: A feedback loop that helps useful movements spread can also accelerate harmful material.
Key terms: data voids, search term, computer speed, human speed, algospeak, safety guardrails, downranked, popularity, content-agnostic, human supervision
8.3 Society
This section places recommender systems in a social and historical context, arguing that they accelerated existing trends rather than creating all the problems associated with them.
- Preexisting Social Trends: Explain how the section situates recommender systems within existing social and political problems.
Why it matters: This distinction helps put the effects of recommendations in their wider historical context. - Post Cold War Tribalism: Describe the section's account of how the end of the Cold War changed the Western political context.
Why it matters: This historical account explains one source of the divisions that preceded recommender systems. - Free Trade and Dislocation: Explain the promise of free trade and the local economic feedback loop described in the section.
Why it matters: The contrast shows how economic change could bring both promised benefits and insecurity. - Partisan Media and Polarization: Explain how the section connects changes in US media and political rhetoric to polarization.
Why it matters: These examples show how division and reinforcement were developing before recommendation-driven feeds. - Decline of Third Places: Describe third places and the consequences of their decline discussed in the section.
Why it matters: The loss of shared gathering places is another part of the social context in which platforms operate. - Socio Technical User Interaction: Explain why the section calls recommender systems socio-technical and how it characterizes users' responses.
Why it matters: It frames recommendations as an interaction between people and technology rather than a technology acting alone.
Key terms: recommender systems, polarized, Cold War, liberal democracy, tribalist mentality, free trade, specialize, feedback loop, Fairness Doctrine, partisan cable news
8.4 Summary
Where this is used later
- Seeing through the lie: meta awareness (chapter 10) · People’s instincts and motivations that platforms can leverage.
- “On the internet, nobody knows you are a dog” (chapter 10) · The appeal of belonging to a group of like-minded people.
- The power of incentives (chapter 10) · Feedback loops and extreme content
- Duty of care (chapter 10) · The influence of recommender systems on users and their exposure to content.
Concepts explained in this chapter
- group identificationA person's sense of self derived from membership in social groups.
- conformityThe adjustment of a person's behavior or expressed judgment to align with a group's expectations or actions.
- majority illusionA phenomenon in which a belief appears more widely accepted through a person's network than it is in the population overall.
- moral outrageA strong emotional response of condemnation to conduct perceived as violating moral norms.
- selection biasA distortion that occurs when the observations presented or collected do not accurately represent the population of interest.
- social comparisonThe evaluation of one's abilities, circumstances, or status by comparing them with those of other people.
- social comparison theoryA psychological theory that explains how people evaluate themselves by comparing their opinions, abilities, or circumstances with those of others.
- social identity theoryA psychological theory that explains group behavior through people's identification with social groups.
- socio-technical systemA system in which technology, users, and their interactions form an interdependent unit that cannot be adequately understood by studying its components in isolation.
- third placeA social setting outside the home and workplace where people regularly gather and interact.
- dopamineA neurotransmitter involved in brain processes including reward, motivation, and movement.
- endless scrollA user-interface pattern that loads additional content continuously as a user scrolls, without requiring navigation to another page.
- intermittent reinforcementA reinforcement schedule in which a behavior is rewarded only on some occasions rather than after every instance.
- novelty seekingA tendency to seek new or unfamiliar stimuli and experiences.
- rabbit holingA gradual shift in a user’s recommendations toward increasingly unexpected or harmful content.
- in-network contentSocial-media content originating from users whom a reader explicitly follows, including content they reshare.
- out-of-network contentContent produced by users or sources that a reader does not directly follow.
- recommender systemA computational system that selects, ranks, or presents items to users based on inferred preferences or other criteria.
- engagement-based recommender systemA recommendation approach that selects or ranks content according to predicted or measured user engagement.
- feedback loopA cyclic process in which a system's outputs influence subsequent inputs, causing later outputs to be shaped by earlier ones.
- social normA shared expectation or rule governing behavior within a group or society.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
