Chapter 7: Societal impact
This introduction frames societal impacts as technology-attributable changes, including spillovers, and cautions against simplistic claims about recommender systems’ role.
After this chapter you will be able to
- Societal effects and spillovers. Explain how technology can change behavior and produce unintended effects that reach non-users, using an example from the section.
- Recommender system impact attribution. Explain why isolating the societal effects of recommender systems from those of broader internet growth is difficult.
- Acceleration of existing problems. Explain how social media and recommender systems can intensify pre-existing societal problems rather than originate them, using political polarization as an example.
- Social contagion and sentiment. Explain social contagion and describe the study’s observed relationship between feed sentiment and users’ posts.
- Feed experiment design. Describe how the researchers used an A/B test to change users’ exposure to positive or negative posts without inserting new candidate posts.
- Study interpretation and limitations. Explain the section’s cautions about interpreting the results, including the small effect and limitations of word-based sentiment measurement.
- Voting Experiment and Measurement. Describe the three experiences compared in Facebook’s voting experiment and explain how researchers distinguished online signaling from actual voting.
- Turnout Effects at Scale. Explain the estimated effect of the friend-visible banner and why a small change in turnout can matter at platform scale and in close elections.
- Targeted Political Influence Risks. Explain how the same technique could be used to influence voting selectively and why the section treats platform control as a concern.
- Amplification of anti Rohingya hate. Explain how Facebook’s reach and engagement-optimized recommendations contributed to the spread of anti-Rohingya hate and helped the genocide in Myanmar.
- Flower speech engagement reversal. Trace how the Panzagar flower-sticker initiative unintentionally increased the promotion of harmful posts.
- Moderation warnings and language barriers. Explain how Facebook’s response to warnings and its limited Burmese-language moderation allowed harmful content to remain online.
- Decentralized organizing and tradeoffs. Explain how internet connections enable decentralized organizing and describe the tradeoff illustrated by the Occupy movements.
- Social media and movement information. Describe how social media supported movements such as the Arab Spring and Gezi Park protests, and distinguish that role from algorithmic feeds.
- Hashtag origins and organizing. Describe the origins of #BlackLivesMatter and how protesters used it during the Ferguson protests and after George Floyd’s murder.
- Viral video and reach. Explain how Darnella Frazier’s video affected the account of George Floyd’s murder and the reach of the movement.
- Wider awareness and pressure. Explain how the spread of activists’ posts contributed to public discussion and pressure on institutions.
- Consequences of the Movement. Explain what the #MeToo movement brought to light and the changes and discussions it prompted.
- Distributed Virality and Coverage. Explain how social media and recommender systems helped reveal the extent of abuse and how news coverage reinforced attention to the movement.
- Virality and Accuracy Risks. Explain why a viral public accusation or emotionally charged post should not automatically be treated as accurate.
- Gamergate harassment and amplification. Explain how coordinated Gamergate harassment was amplified online and affected its targets.
- Political memes and reach. Explain how memes helped spread alt-right political messages and foster participation.
- QAnon and offline action. Trace how QAnon gained attention online and connect the conspiracy narrative to an offline event.
- Culture and social comparison. Explain what culture includes and how social media changes the scale of social comparison.
- Shift in cultural gatekeeping. Describe how social media and recommender systems changed who selects the content people encounter.
- Personalized feed differences. Explain how personalization distinguishes recommender systems from earlier gatekeepers and why two users can see different feeds.
- News and information diets. Explain why news that caters only to immediate preferences can leave the public poorly informed, and contrast that with the balance a good legacy news service could provide.
- Shared reality and polarization. Explain how personalized, engagement-driven news can weaken a shared awareness of important issues and accelerate existing polarization.
- News Economics and Attention. Explain how the shift of attention and advertising to platforms pressures news organizations, and how attention competition can reduce nuance.
- Extremity and Press Distrust. Explain how hyper-personalized content can foster distrust of mainstream news and why the section regards that distrust as dangerous for democracy.
- Critical News Consumption. Describe the section’s advice for evaluating political information encountered on social media.
- Creator incentives and homogenization. Explain how recommendation incentives can lead creators to change their work and make visible content look more alike.
What it builds on
- Lack of a baseline (chapter 4) · The difficulty of establishing a baseline for assessing recommender systems.
- Post processing (chapter 2) · Post processing as a step in preparing recommendations
- Testing recommender systems: A/B tests (chapter 4) · A/B tests compare versions across groups
- Recommendation as amplification (chapter 1) · Recommendation as amplification
- Users’ preferences and proxy metrics (chapter 4) · Engagement is used as a proxy in recommender systems.
- Cultural context (chapter 3) · Content moderation depends on cultural context.
- Nuanced decisions (chapter 3) · Moderation decisions can be nuanced.
- Feedback loops in recommender systems and social media (chapter 4) · Feedback loops in recommender systems and social media
Section by section
Introduction
This introduction frames societal impacts as technology-attributable changes, including spillovers, and cautions against simplistic claims about recommender systems’ role.
- Societal effects and spillovers: Explain how technology can change behavior and produce unintended effects that reach non-users, using an example from the section.
Why it matters: This helps you recognize that a technology’s consequences can extend beyond its direct users. - Recommender system impact attribution: Explain why isolating the societal effects of recommender systems from those of broader internet growth is difficult.
Why it matters: This distinction helps prevent attributing every consequence of internet growth to recommender systems alone. - Acceleration of existing problems: Explain how social media and recommender systems can intensify pre-existing societal problems rather than originate them, using political polarization as an example.
Why it matters: This helps you distinguish amplifying a problem from causing it in the first place.
Key terms: societal impact, offline effects, second-order effects, recommender systems, internet usage, isolate, political polarization, magnitude and diffusion
7.1 Social contagion
This section examines a large-scale social contagion study, its findings, limitations, and ethical implications.
- Social contagion and sentiment: Explain social contagion and describe the study’s observed relationship between feed sentiment and users’ posts.
Why it matters: This study illustrates how the emotional tone people encounter online can be reflected in what they post. - Feed experiment design: Describe how the researchers used an A/B test to change users’ exposure to positive or negative posts without inserting new candidate posts.
Why it matters: The experiment shows how changing the ordering of available posts can alter what people encounter. - Study interpretation and limitations: Explain the section’s cautions about interpreting the results, including the small effect and limitations of word-based sentiment measurement.
Why it matters: These limitations keep an observed change in posting from being treated as conclusive evidence of a substantial mood change. - Informed consent in experiments: Explain why the study raised questions about informed consent for platform experiments.
Why it matters: A user’s agreement to broad service terms may not settle whether they were meaningfully informed about an experiment. - Social comparison interpretation: State the researchers’ reported interpretation of the study for social comparison with successful friends.
Why it matters: The researchers’ interpretation challenges the assumption that seeing friends’ accomplishments necessarily makes people sadder.
Key terms: social contagion, sentiment, A/B test, post-processing, candidate-generation, correlation, causation, informed consent, terms of service, A/B tests
7.2 Boosting voter turnout
This section examines Facebook’s voter-turnout experiment and its implications for social influence, elections, and platform power.
- Voting Experiment and Measurement: Describe the three experiences compared in Facebook’s voting experiment and explain how researchers distinguished online signaling from actual voting.
Why it matters: The comparison shows how researchers tested social influence without mistaking a button click for a ballot. - Turnout Effects at Scale: Explain the estimated effect of the friend-visible banner and why a small change in turnout can matter at platform scale and in close elections.
Why it matters: A modest effect on each person can translate into a consequential number of votes across a large platform. - Targeted Political Influence Risks: Explain how the same technique could be used to influence voting selectively and why the section treats platform control as a concern.
Why it matters: A tool that increases turnout can also concentrate political influence in private hands. - Positive Platform Narratives: Explain how a positive mission or turnout result can shape public, employee, and regulatory perceptions of a platform.
Why it matters: A genuine benefit does not by itself settle whether a platform’s other practices deserve scrutiny. - Recommender System Relevance: Explain why the voting study is relevant to recommender systems even though it did not change a recommendation algorithm.
Why it matters: The study connects social influence to what platforms choose to make prominent, even without testing a recommender change.
Key terms: control group, I Voted, public voting records, social signaling, social influence, voter turnout, electoral colleges, demographic data, for-profit company, incentives
7.3 The risk of information pollution: genocide
This section examines how Facebook’s engagement-driven recommendations and inadequate moderation helped spread anti-Rohingya hate in Myanmar, contributing to real-world violence.
- Amplification of anti Rohingya hate: Explain how Facebook’s reach and engagement-optimized recommendations contributed to the spread of anti-Rohingya hate and helped the genocide in Myanmar.
Why it matters: This case shows how recommendations can amplify harmful content into real-world consequences without being the sole cause of those consequences. - Flower speech engagement reversal: Trace how the Panzagar flower-sticker initiative unintentionally increased the promotion of harmful posts.
Why it matters: The example shows that an engagement signal can have the opposite effect from what users intend. - Moderation warnings and language barriers: Explain how Facebook’s response to warnings and its limited Burmese-language moderation allowed harmful content to remain online.
Why it matters: The case shows why responding to warnings and understanding local language and context matter for moderation.
Key terms: engagement-optimized recommender systems, anti-Rohingya posts, genocide, Panzagar, flower speech, positive engagements, content moderation, Burmese-speaking content moderator, automatic translation systems
7.4 Organizing online
This section explains how internet connections and social media support collective organizing, using Occupy and other movements to illustrate their benefits and limitations.
- Decentralized organizing and tradeoffs: Explain how internet connections enable decentralized organizing and describe the tradeoff illustrated by the Occupy movements.
Why it matters: Decentralized organizing can mobilize people without an institution, but sustaining its impact can be difficult. - Social media and movement information: Describe how social media supported movements such as the Arab Spring and Gezi Park protests, and distinguish that role from algorithmic feeds.
Why it matters: Movements can benefit from social media's communication space even without algorithmic feeds.
Key terms: decentralized movement, centralized governance structure, Occupy movements, social media platforms, algorithmic recommendation engines, Arab Spring, Gezi Park protests
7.4.1 #BlackLivesMatter
This section shows how online sharing and hashtags supported Black Lives Matter organizing, public awareness, and international reach, while also noting an opposing movement.
- Hashtag origins and organizing: Describe the origins of #BlackLivesMatter and how protesters used it during the Ferguson protests and after George Floyd’s murder.
Why it matters: The example shows how an online hashtag can help people organize action offline. - Viral video and reach: Explain how Darnella Frazier’s video affected the account of George Floyd’s murder and the reach of the movement.
Why it matters: The video illustrates how widely shared evidence can challenge an official account and draw attention to a movement. - Wider awareness and pressure: Explain how the spread of activists’ posts contributed to public discussion and pressure on institutions.
Why it matters: The spread of posts can connect individual experiences to broader public discussion and calls for change. - Blue Lives Matter opposition: Describe the emergence and stated position of Blue Lives Matter in relation to Black Lives Matter.
Why it matters: The countermovement shows that the movement also encountered organized opposition.
Key terms: #BlackLivesMatter, hashtag, Ferguson, Darnella Frazier, virality, George Floyd, systemic racism, public institutions, reforms, Blue Lives Matter
7.4.2 #MeToo
This section traces the emergence and impact of #MeToo and considers both the power and the risks of social media virality.
- Origins of MeToo: Describe how Tarana Burke’s use of “me too” grew into a viral international movement in 2017.
Why it matters: The movement’s history shows how an existing phrase for solidarity reached a much larger audience. - Consequences of the Movement: Explain what the #MeToo movement brought to light and the changes and discussions it prompted.
Why it matters: The movement’s impact went beyond online conversation to expose abuse and question the conditions that allowed it. - Distributed Virality and Coverage: Explain how social media and recommender systems helped reveal the extent of abuse and how news coverage reinforced attention to the movement.
Why it matters: This illustrates how many individual disclosures and media attention can reinforce one another. - Virality and Accuracy Risks: Explain why a viral public accusation or emotionally charged post should not automatically be treated as accurate.
Why it matters: A widely shared claim can have real consequences without its popularity proving it true.
Key terms: Tarana Burke, Alyssa Milano, #MeToo, cultural norms, patriarchy, recommender systems, virality, legacy media, engagement, accuracy
7.4.3 Meme wars
This section traces how online harassment, political memes, and conspiracy theories helped the alt-right gain reach and influence offline events.
- Gamergate harassment and amplification: Explain how coordinated Gamergate harassment was amplified online and affected its targets.
Why it matters: This example shows how coordinated abuse can be mistaken for engagement and produce real harm. - Alt-right narratives and community: Describe why alt-right narratives and online communities resonated with some socially isolated young men.
Why it matters: The account connects political messaging with the social needs and frustrations of its audience. - Political memes and reach: Explain how memes helped spread alt-right political messages and foster participation.
Why it matters: Memes can carry political messages to new audiences while making participation feel easy. - Irony and plausible deniability: Explain irony poisoning and Poe's law as they relate to extreme political memes.
Why it matters: Ambiguous humor can spread extreme ideas while making the speaker's intent difficult to judge. - QAnon and offline action: Trace how QAnon gained attention online and connect the conspiracy narrative to an offline event.
Why it matters: The QAnon example shows how online conspiracy narratives and their amplification can be linked to consequential offline action. - Delayed platform intervention: Describe the platforms' response to the Capitol assault and the section's explanation of its timing.
Why it matters: The timing of moderation raises questions about how platform business interests affect intervention.
Key terms: Gamergate, coordinated harassment, engaging, alt-right, incels, Third Places, memes, virality, Pepe, identity politics
7.5 The effect on culture
This section explains how social media and personalized recommendations reshape cultural comparison and the selection of content people encounter.
- Culture and social comparison: Explain what culture includes and how social media changes the scale of social comparison.
Why it matters: This connects recommender systems to the social comparisons through which people form beliefs. - Shift in cultural gatekeeping: Describe how social media and recommender systems changed who selects the content people encounter.
Why it matters: This shows why recommendation systems have a significant role in shaping access to cultural content. - Personalized feed differences: Explain how personalization distinguishes recommender systems from earlier gatekeepers and why two users can see different feeds.
Why it matters: Personalization means people can encounter different versions of the same platform.
Key terms: culture, social media, recommender systems, group comparisons, gatekeepers, algorithmic recommender systems, news, entertainment, political commentary, personalization
7.5.1 News
This section examines how personalized news affects information diets, journalism, shared reality, and democracy, while urging critical engagement with news sources.
- News and information diets: Explain why news that caters only to immediate preferences can leave the public poorly informed, and contrast that with the balance a good legacy news service could provide.
Why it matters: What attracts attention is not always what citizens need to understand. - Shared reality and polarization: Explain how personalized, engagement-driven news can weaken a shared awareness of important issues and accelerate existing polarization.
Why it matters: A society can have diverse perspectives without losing a common awareness of major issues. - Limits of legacy journalism: Describe why the section cautions against idealizing older media, using its example of reporting on Iraq.
Why it matters: Comparing platforms with journalism does not require pretending that older media were free of bias or commercial pressure. - News Economics and Attention: Explain how the shift of attention and advertising to platforms pressures news organizations, and how attention competition can reduce nuance.
Why it matters: Platform competition can change not only what people see but also how news organizations produce and present reporting. - Extremity and Press Distrust: Explain how hyper-personalized content can foster distrust of mainstream news and why the section regards that distrust as dangerous for democracy.
Why it matters: A misleading sense of what is normal can undermine trust in independent sources of information. - Critical News Consumption: Describe the section’s advice for evaluating political information encountered on social media.
Why it matters: Looking beyond a personalized feed can help you understand an issue rather than only encounter views you already agree with.
Key terms: personalization, information diet, gatekeeper-based system, shared reality, polarization, engagement-optimization algorithm, Manufacturing Consent, mass media, ad spending, attention span
7.5.2 Homogenization
This section explains how recommender systems shape tastes, creators’ incentives, cultural trends, and the visibility of personalized media.
- Creator incentives and homogenization: Explain how recommendation incentives can lead creators to change their work and make visible content look more alike.
Why it matters: Recommendation incentives can shape not just what you see, but also what creators make. - Recommendations and personal taste: Explain how repeated algorithmic selection can shape personal taste.
Why it matters: What feels like an entirely independent preference may also reflect what you have been shown. - Opaque and changeable incentives: Distinguish adapting creative work to customers from adapting it to platform algorithms.
Why it matters: Creators must make decisions about their work without a clear or stable explanation of what will be rewarded. - Scale and speed of trends: Explain what platforms change about the spread and duration of cultural trends.
Why it matters: A like can reward a fast-moving trend, even when engaging content is not especially valuable to you. - Personalized feed audit challenges: Explain why personalized feeds make outside scrutiny of media more difficult.
Why it matters: Personalization makes it harder for researchers and the public to see what a platform is showing people.
Key terms: incentive structure, watch time, viral, personal taste, algorithm's influence, commercially viable, algorithm, virality, trends, memes
7.6 Summary
Where this is used later
- Society (chapter 8) · The societal impacts associated with recommender systems
- Feedback loops and extreme content (chapter 8) · The discussion of feedback loops and extreme content examines influences on the spread and intensity of content.
- Power over users, and lack of contestability (chapter 10) · Considers platforms’ power over users
- The great lie: making the World a better place (chapter 9) · Examines the claim that technology companies make the world a better place
- (chapter intro) (chapter 10) · The societal risk of harmful information spreading.
- The power of incentives (chapter 10) · The risk of harmful information pollution
- Communities and like-minded users (chapter 8) · Examples of movements organizing online
- (chapter intro) (chapter 8) · The next chapter takes up the spread of misinformation mentioned in this discussion.
Concepts explained in this chapter
- recommender systemA computational system that selects, ranks, or presents items to users based on inferred preferences or other criteria.
- personalizationThe adaptation of item selection, ordering, or presentation to an individual user's inferred characteristics or preferences.
- Poe's lawThe principle that an extreme parody or sarcastic statement may be interpreted as sincere when the author's intent is not explicit.
- conspiracy theoryAn explanation of events that attributes them to a secretive, coordinated group or plot, typically without reliable supporting evidence.
- memeA cultural unit or expression that spreads among people through imitation, copying, and modification.
- engagement optimizationAn algorithm that selects or orders content primarily to maximize user interaction.
- polarizationA social or political condition in which opinions, groups, or identities become increasingly divided into opposing extremes.
- user engagementThe degree to which users interact with or remain attentive to content or a platform.
- extreme contentContent expressing unusually intense, radical, or socially extreme viewpoints or language.
- informed publicA population with sufficient reliable knowledge to understand and deliberate about matters of public concern.
- shared understandingA commonly held understanding of events, facts, or social conditions among members of a community.
- viralityThe tendency of content or information to spread rapidly and widely through social networks and media channels.
- social mediaAn online service that supports user-generated content, communication, and social interaction.
- content homogenizationThe increasing similarity of content in format, style, topics, or expression.
- information imbalanceA condition in which some topics, perspectives, or levels of importance receive substantially more informational attention than others.
- traditional mediaEstablished mass communication such as newspapers and magazines in which organizations oversee, edit, fact-check, and publish content.
- cultureThe shared beliefs, norms, practices, and artifacts through which a community interprets the world and relates to others.
- social influenceThe effect of other people or social connections on an individual's attitudes, decisions, or behavior.
- voter turnoutThe number or proportion of eligible voters who cast ballots in an election.
- voting behaviorThe actions and decisions people make concerning whether, when, and how to vote.
- unsafe contentContent that can cause or materially contribute to physical, psychological, social, or other significant harm.
- A/B testAn online experiment that randomly exposes different user groups to alternative system variants and compares their outcomes.
- candidate sourcingA recommender-system stage that selects a relatively small set of potentially relevant items from a larger item collection.
- correlationA statistical association in which two variables vary together without by itself establishing that one causes the other.
- informed consentA person's voluntary agreement to participate in an activity after receiving understandable information about its purpose, procedures, and potential risks.
- rank orderAn ordering of items according to their relative position or assessed importance.
- rerankingAn optional recommendation-system stage that adjusts an already ranked item list before results are delivered.
- sentimentThe emotional polarity or affective tone expressed by a piece of text or communication.
- social contagionThe spread of a behavior, idea, or emotion across members of a social network through their interactions or exposure to one another.
- algorithmic feedA feed whose contents or ordering are selected by algorithms rather than solely by chronological or follow-based rules.
- algorithmic influenceThe effect that algorithmic selection, ordering, or distribution has on people's preferences, behavior, or content production.
- recommendation algorithmAn algorithm that selects or ranks items for recommendation based on user, item, or other information.
- content moderationThe review, filtering, labeling, or removal of user-generated or platform-hosted content according to defined rules.
- externalityA consequence of an activity that affects parties beyond those directly participating in it.
- offline effectA consequence of online technology that changes behavior or conditions outside the online environment.
- second-order effectAn unintended consequence that arises indirectly from a technology’s initial effects on behavior or conditions.
- societal impactThe effects that a technology, policy, or activity has on society, including its effects on behavior, culture, institutions, and public life.
- algorithmic amplificationAn increase in the visibility, reach, or influence of content caused by automated algorithmic selection and distribution.
- third placeA social setting outside the home and workplace where people regularly gather and interact.
- cohortA group of users or participants analyzed together because they share an experimental assignment or other defining characteristic.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
