R. Stuart Geiger

R. Stuart Geiger

Ethnographer of computation and computational ethnographer

Assistant Professor at UC San Diego, Department of Communication & Halıcıoğlu Data Science Institute

Articles

You can also find my articles on my Google Scholar profile.

Unsafe at any AUC: Unlearned Lessons From Sociotechnical Disasters for Responsible AI

Published in Harvard Data Science Review, 2026

Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or “responsibly designed” components to understand risks at a systems level.

Recommended citation: Kroll, J., Smart, A., Geiger, R.S., and Jacobs, A. (2026). "Unsafe at any AUC: Unlearned Lessons From Sociotechnical Disasters for Responsible AI." Harvard Data Science Review 8(3). https://doi.org/10.1162/99608f92.91b8834f

Asking an AI for salary negotiation advice is a matter of concern: Controlled experimental perturbation of ChatGPT for protected and non-protected group discrimination on a contextual task with no clear ground truth answers

Published in PLoS ONE, 2025

We audited four ChatGPT versions with 98,800 prompts each for bias in salary negotiation advice. We found statistically significant gaps by gender, university, and major, with the largest variations between model versions and employee vs. employer perspectives. Results were inconsistent across versions, raising concerns about using ChatGPT for contextual tasks without clear ground truth.

Recommended citation: Geiger, R.S., O'Sullivan, F., Wang, E., & Lo, J. (2025). "Asking an AI for salary negotiation advice is a matter of concern: Controlled experimental perturbation of ChatGPT for protected and non-protected group discrimination on a contextual task with no clear ground truth answers." PLoS ONE 20(2): e0318500. https://doi.org/10.1371/journal.pone.0318500

Making Algorithms Public: Reimagining Auditing From Matters of Fact to Matters of Concern

Published in International Journal of Communication, 2024

Through four algorithmic audit cases, we demonstrate how scoping decisions shape audit outcomes and reflect institutional power. We propose moving beyond technical certification toward building infrastructures for democratic understanding and contestation, recognizing auditing as a political practice that must engage with institutional reform, not just algorithmic behavior.

Recommended citation: Geiger, R. Stuart, Udayan Tandon, Anoolia Gakhokidze, Lian Song, and Lilly Irani. (2024). "Making Algorithms Public: Reimagining Auditing From Matters of Fact to Matters of Concern." International Journal of Communication, 18, 634-655. https://ijoc.org/index.php/ijoc/article/download/20811/4455

Opinion: ChatGPT is this generation’s Wikipedia. We have an opportunity to learn from the past

Published in San Diego Union Tribune, 2023

This is an op-ed published in the San Diego Union Tribune as part of a set of commentaries on ChatGPT.

Recommended citation: Geiger, R. Stuart. (2023, June 19). "Opinion: ChatGPT is this generation's Wikipedia. We have an opportunity to learn from the past." San Diego Union Tribune. https://www.sandiegouniontribune.com/opinion/commentary/story/2023-06-19/opinion-chatgpt-wikipedia-artificial-intelligence-ai-schools-education

Community, Time, and (Con)text: A Dynamical Systems Analysis of Online Communication and Community Health among Open-Source Software Communities

Published in Cognitive Science, 2022

Open-source software projects, maintained largely by unpaid volunteers, face recruitment and retention challenges. Using dynamical systems analysis of community communications, we found that sentiment and gratitude expressions significantly shape newcomer retention, with impacts modulated by the context of first contact.

Recommended citation: Paxton, A., Varoquaux, N., Holdgraf, C. and Geiger, R.S. (2022), Community, Time, and (Con)text: A Dynamical Systems Analysis of Online Communication and Community Health among Open-Source Software Communities. Cognitive Science, 46: e13134. https://doi.org/10.1111/cogs.13134

‘Garbage In, Garbage Out’ Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?

Published in Quantitative Science Studies, 2021

Supervised machine learning, in which models are automatically derived from labeled training data, is only as good as the quality of that data. We report to what extent a random sample of ML application papers across disciplines give specific details about whether best practices were followed in labeling training data.

Recommended citation: R. Stuart Geiger, Dominique Cope, Jamie Ip, Marsha Lotosh, Aayush Shah, Jenny Weng, and Rebekah Tang. 2021. "'Garbage in, garbage out' revisited: What do machine learning application papers report about human-labeled training data?" Quantitative Science Studies 2(3). https://doi.org/10.1162/qss_a_00144
Download Paper

The Labor of Maintaining and Scaling Free and Open-Source Software Projects

Published in Proceedings of the ACM on Human-Computer Interaction (CSCW 2021), 2021

We report findings from an interview-based study of maintainers of free and/or open-source software (F/OSS) projects. F/OSS maintainers perform complex and often-invisible interpersonal and organizational work to keep their projects operating as active communities of users and contributors. We particularly focus on how this labor of maintaining and sustaining changes as projects and their software grow and scale across many dimensions.

Recommended citation: R. Stuart Geiger, Dorothy Howard, and Lilly Irani. 2021. "The Labor of Maintaining and Scaling Free and Open-Source Software Projects." Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1, Article 175 (April 2021), 28 pages. https://doi.org/10.1145/3449249 https://stuartgeiger.com/papers/maintaining-scaling-foss-cscw2021.pdf
Download Paper

Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?

Published in Proceedings of the 2020 ACM Conference on Fairness, Accountability, and Transparency (FAT* 2020), 2019

Many machine learning projects for new application areas involve teams of humans who label data for a particular purpose, from hiring crowdworkers to the paper’s authors labeling the data themselves. In this paper, we investigate to what extent a sample of machine learning application papers in social computing – specifically papers from ArXiv and traditional publications performing an ML classification task on Twitter data – give specific details about whether best practices in human annotation were followed.

Recommended citation: R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. "Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?" In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona, Spain. ACM, New York, NY, USA, 18 pages. https://stuartgeiger.com/papers/gigo-fat2020.pdf https://doi.org/10.1145/3351095.3372862
Download Paper

ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia

Published in Proceedings of the ACM on Human-Computer Interaction (CSCW 2020), 2019

This paper presents an overview and case studies of ORES, Wikipedia’s real-time machine learning as a service platform, which is designed in line with Wikipedia’s values of open participation, decentralization, and continual iteration. ORES decouples and reduces incidental complexity around several aspects of applying machine learning in a user-generated content platform, including curating training data sets, building models to serve predictions, auditing predictions, and developing interfaces or automated agents that act on those predictions.

Recommended citation: Aaron Halfaker and R. Stuart Geiger. 2019. "ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia."Proceedings of the ACM on Human-Computer Interaction, 4, CSCW2, Article 148 (October 2020), 37 pages. https://arxiv.org/pdf/1909.05189.pdf https://doi.org/10.1145/3415219
Download Paper

The Rise and Fall of the Note: Changing Paper Lengths in ACM CSCW, 2000-2018

Published in Proceedings of the ACM on Human-Computer Interaction (PACMHCI, CSCW 2019), 2019

A short paper (or note) quantitatively examining changing paper lengths in the Proceedings of the ACM Conference on Computer-Supported Cooperative Work, focusing on the rise and fall of the 4-page note format.

Recommended citation: R. Stuart Geiger. 2019. "The Rise and Fall of the Note: Changing Paper Lengths in ACM CSCW, 2000-2018." Proceedings of the ACM Human Computer Interaction (PACMHCI) 3, CSCW, Article 222 (November 2019), 10 pages. https://doi.org/10.1145/3359324 https://arxiv.org/pdf/1908.10808.pdf
Download Paper

Reports from the BIDS Best Practices in Data Science Series

Published in , 2018

An ongoing series of short papers that report from discussions where we share our experiences doing data science well (or at least better), for many definitions of the term.

Recommended citation: R. Stuart Geiger, Dan Sholler, Aaron Culich, Ciera Martinez, Fernando Hoces de la Guardia, François Lanusse, Kellie Ottoboni, Marla Stuart, Maryam Vareth, Nelle Varoquaux, Sara Stoudt, and Stéfan van der Walt. "Challenges of Doing Data-Intensive Research in Teams, Labs, and Groups: Report from the BIDS Best Practices in Data Science Series." _BIDS Best Practices in Data Science Series._ Berkeley Institute for Data Science: Berkeley, California. 2018. doi:10.31235/osf.io/a7b3m

Career Paths and Prospects in Academic Data Science: Report of the Moore-Sloan Data Science Environments Survey

Published in , 2018

This report of a survey of academic data scientists discusses what data science in the academy is, and various issues around the career paths for those in universities who practice and support data science. We provide evidence-based recommendations about how universities can better support an emerging set of roles and responsibilities around data and computation within and across academic fields.

Recommended citation: R. Stuart Geiger, Charlotte Mazel-Cabasse, Chihoko Cullens, Laura Noren, Brittany Fiore-Gartland, Diya Das, and Henry Brady (2018). _Career Paths and Prospects in Academic Data Science: Report of the Moore-Sloan Data Science Environments Survey._ Report. Berkeley, California: UC-Berkeley Institute for Data Science. https://osf.io/preprints/socarxiv/xe823/

The Types, Roles, and Practices of Documentation in Data Analytics Open Source Software Libraries: A Collaborative Ethnography of Documentation Work

Published in Computer-Supported Cooperative Work (JCSCW), 2018

Data analytics increasingly relies on open source software (OSS) libraries that extend scripted languages like python and R. Software documentation for these libraries is crucial for people across all experience levels, but documentation work raises many challenges, particularly in open source communities. In this collaboration between ethnographers and data scientists, we discuss the types, roles, practices, and motivations around documentation in data analytics OSS libraries.

Recommended citation: Geiger, R.S., Varoquaux, N., Mazel-Cabasse, C., and Holdgraf, C. (2018). "The Types, Roles, and Practices of Documentation in Data Analytics Open Source Software Libraries: A Collaborative Ethnography of Documentation Work." Computer-Supported Cooperative Work (JCSCW), 27(3). DOI:10.1007/s10606-018-9333-1. https://link.springer.com/article/10.1007/s10606-018-9333-1

Beyond opening up the black box: Investigating the role of algorithmic systems in Wikipedian organizational culture

Published in Big Data & Society, 2017

Scholars and practitioners across domains are increasingly concerned with algorithmic transparency and opacity, interrogating the values and assumptions embedded in automated, black-boxed systems, particularly in user-generated content platforms. I report from an ethnography of infrastructure in Wikipedia to discuss an often understudied aspect of this topic: the local, contextual, learned expertise involved in participating in a highly automated social-technical environment.

Recommended citation: R. Stuart Geiger. (2017). "Beyond opening up the black box: Investigating the role of algorithmic systems in Wikipedian organizational culture." Big Data & Society 4(2). https://doi.org/10.1177/2053951717730735
Download Paper

Operationalizing conflict and cooperation between automated software agents in Wikipedia: A replication and expansion of ‘Even Good Bots Fight’

Published in Proceedings of the ACM on Human-Compter Interaction, 2017

A mixed-method trace ethnographic analysis of issues around the governance of automated software agents in Wikipedia, focusing on how to interpret cases where bots reverted each other’s edits.

Recommended citation: R. Stuart Geiger and Aaron Halfaker. 2017. "Operationalizing conflict and cooperation between automated software agents in Wikipedia: A replication and expansion of Even Good Bots Fight." Proceedings of the ACM on Human-Computer Interaction (Nov 2017 issue, CSCW 2018 Online First) 1, 2, Article 49. DOI:https://doi.org/10.1145/3134684. https://commons.wikimedia.org/wiki/File:conflict-bots-wp-cscw.pdf.
Download Paper

Summary Analysis of the 2017 GitHub Open Source Survey

Published in SocArxiv Preprints, 2017

This report is a high-level summary analysis of the 2017 GitHub Open Source Survey dataset, presenting frequency counts, proportions, and frequency or proportion bar plots for every question asked in the survey.

Recommended citation: R. Stuart Geiger. (2017). Summary Analysis of the 2017 GitHub Open Source Survey. _SocArXiv Preprints._ doi: 10.17605/OSF.IO/ENRQ5

Bot-based collective blocklists in Twitter: the counterpublic moderation of harassment in a networked public space

Published in Information, Communication, and Society, 2016

This article introduces and discusses bot-based collective blocklists (or blockbots) in Twitter, which have been developed by volunteers to combat harassment in the social networking site in a more decentralized and counterpublic way than actions taken by Twitter, Inc. staff. I discuss how such forms of automation require that communities encode specific understandings of what harassment is and how to identify it, relating these cases to several longstanding issues around the governance and moderation of the public sphere.

Recommended citation: Geiger, R. Stuart. (2016). “Bot-based collective blocklists in Twitter: the counterpublic moderation of harassment in a networked public space.” Information, Communication, and Society 19(6). http://stuartgeiger.com/blockbots-ics.pdf
Download Paper

Defining, Designing, and Evaluating Civic Values in Human Computation and Collective Action Systems

Published in Proceedings of HCOMP, Citizen-X Workshop, 2014

We review various crowdsourcing and collective action systems, identifying particular sets of civic values and assumptions.

Recommended citation: Matias, N. and Geiger, R.S. “Defining, Designing, and Evaluating Civic Values in Human Computation and Collective Action Systems.” In Proceedings of HCOMP 2014, Citizen-X Workshop. http://stuartgeiger.com/defining-civic-values-hcomp-matias-geiger.pdf.
Download Paper

Snuggle: Designing for efficient socialization and ideological critique

Published in Proceedings of CHI, 2014

This paper discusses the Snuggle project, built to support newcomer socialization and reflexive critique of Wikipedia's existing socialization processes.

Recommended citation: Halfaker, Aaron., Geiger, R. Stuart., and Treveen, Loren. (2014). “Snuggle: Designing for Efficient Socialization and Ideological Critique.” In Proceedings of the 2014 ACM Conference on Human Factors in Computing (CHI 2014). http://www-users.cs.umn.edu/~halfak/publications/Snuggle/halfaker14snuggle-personal.pdf
Download Paper

Bots, bespoke code, and the materiality of software platforms

Published in Information, Communication, and Society, 2014

This article introduces and discusses the role of bespoke code in Wikipedia, which is code that runs alongside a platform or system, rather than being integrated into server-side codebases.

Recommended citation: Geiger, R. Stuart. (2014). “Bots, Bespoke Code, and the Materiality of Software Platforms.” Information, Communication, and Society 17. http://stuartgeiger.com/bespoke-code-ics.pdf
Download Paper

The Next Generation of Scientists: Examining the Experiences of Graduate Students in Network-Level Social-Ecological Science

Published in Ecology and Society, 2013

We examined how graduate students experienced and social-ecological research initiative within the large-scale, geographically distributed Long Term Ecological Research (LTER) Network.

Recommended citation: Romolini, Michele., Sydne Record, Rebecca. Garvoille, Y. Marusenko, and R. Stuart Geiger. (2013) “The Next Generation of Scientists: Examining the Experiences of Graduate Students in Network-Level Science." In Ecology and Society 18(3). http://stuartgeiger.com/lter-network-level-science-es.pdf
Download Paper

When the Levee Breaks: Without Bots, What Happens to Wikipedia’s Quality Control Processes?

Published in Proceedings of WikiSym, 2013

This paper examines what happened when one of Wikipedia's counter-vandalism bots unexpectedly went offline.

Recommended citation: Geiger, R. Stuart and Halfaker, Aaron. (2013). “When the Levee Breaks: Without Bots, What Happens to Wikipedia’s Quality Control Processes?” In Proceedings of the 9th International Symposium on Wikis and Open Collaboration (WikiSym 2013). http://stuartgeiger.com/wikisym13-cluebot.pdf
Download Paper

The Rise and Decline of an Open Collaboration Community: How Wikipedia’s reaction to sudden popularity is causing its decline

Published in American Behavioral Scientist, 2013

A mixed-method, multi-study analysis of editor retention, socialization, gatekeeping, and governance in Wikipedia.

Recommended citation: Halfaker, Aaron., R. Stuart Geiger, Jonathan Morgan, and John Riedl. (2013). “The Rise and Decline of an Open Collaboration System: How Wikipedia’s reaction to sudden popularity is killing it." American Behavioral Scientist 57(5). http://dx.doi.org/10.1177/0002764212469365
Download Paper

Artifacts that Organize: Delegation in the Distributed Organization

Published in Information and Organization, 2012

This paper studies the role of computational infrastructure and organizational structure in the Open Science Grid.

Recommended citation: Ribes, David, Steve Jackson, R. Stuart Geiger, Matt C. Burton, and Tom Finholt (2012). “Artifacts that organize: Delegation in the distributed organization.” Information and Organization 23:1–14. http://www.stuartgeiger.com/artifacts-that-organize.pdf
Download Paper

“Writing up rather than writing down”: Becoming Wikipedia Literate

Published in Proceedings of WikiSym, 2012

We introduce and advocate a multi-faceted theory of literacy to investigate the knowledges and organizational forms are required to improve participation in Wikipedia’s communities.

Recommended citation: Ford, Heather and R. Stuart Geiger. (2012). “”Writing up rather than writing down”: Becoming Wikipedia Literate.” In Proceedings of the 8th International Symposium on Wikis and Open Collaboration (WikiSym 2012). New York: ACM Digital Library. http://www.stuartgeiger.com/becoming-wikipedia-literate.pdf
Download Paper

Defense Mechanism or Socialization Tactic? Improving Wikipedia’s Notifications to Rejected Contributors

Published in Proceedings of ICWSM, 2012

A descriptive study of Wikipedia's highly-automated socialization processes and an A/B test to improve templated messages to newcomers.

Recommended citation: Geiger, R. Stuart, Aaron Halfaker, Maryana Pinchuk, and Steven Walling (2012). “Defense Mechanism or Socialization Tactic? Improving Wikipedia’s Notifications to Rejected Contributors.” In Proceedings of the 2012 International Conference on Weblogs and Social Media (ICWSM 2012). http://stuartgeiger.com/defense-mechanism-icwsm.pdf
Download Paper

Black-boxing the user: internet protocol over xylophone players (IPoXP)

Published in Proceedings of CHI (alt.CHI), 2012

We introduce IP over Xylophone Players (IPoXP), a novel Internet protocol between two computers using xylophone-based Arduino interfaces

Recommended citation: Geiger, R. Stuart, Yoon J. Jeong, and Emily Manders (2012). “Black-Boxing the User: Internet Protocol over Xylophone Players.” In Proceedings of the 2012 ACM Conference on Human-Computer Interaction (alt.CHI 2012). New York: ACM Digital Library. http://stuartgeiger.com/ipoxp.pdf

The Lives of Bots

Published in Wikipedia: A Critical Point of View, 2011

I describe the complex social and technical environment in which bots exist in Wikipedia, emphasizing not only how bots produce order and enforce rules, but also how humans produce bots and negotiate rules around their operation.

Recommended citation: Geiger, R. Stuart. (2011). “The Lives of Bots.” In G. Lovink and N. Tkacz (eds.) In Wikipedia: A Critical Point of View. Amsterdam: Institute of Network Cultures. http://www.stuartgeiger.com/lives-of-bots-wikipedia-cpov.pdf
Download Paper

Participation in Wikipedia’s Article Deletion Processes

Published in Proceedings of WikiSym, 2011

This paper investigates Wikipedia's article deletion processes, finding that it is heavily populated by specialists.

Recommended citation: Geiger, R. Stuart and Heather Ford. (2011) “Participation in Wikipedia’s Deletion Processes.” In Proceedings of the 7th International Symposium on Wikis and Open Collaboration (WikiSym 2011). New York: ACM Digital Library. http://www.stuartgeiger.com/papers/article-deletion-wikisym-geiger-ford.pdf
Download Paper

Trace Ethnography: Following Coordination through Documentary Practices

Published in Proceedings of HICSS , 2011

We detail the methodology of ‘trace ethnography’, which combines the richness of participant-observation with the wealth of data in logs so as to reconstruct patterns and practices of users in distributed sociotechnical systems

Recommended citation: Geiger, R. Stuart and David Ribes (2011). “Trace Ethnography: Following Coordination through Documentary Practices.” In Proceedings of the 44th Annual Hawaii International Conference on System Sciences (HICSS). http://www.stuartgeiger.com/trace-ethnography-hicss-geiger-ribes.pdf
Download Paper

The Work of Sustaining Order in Wikipedia: The Banning of a Vandal

Published in Proceedings of CSCW , 2010

This paper traces out a heterogeneous network of humans and non-humans involved in the identification and banning of a single vandal in Wikipedia.

Recommended citation: Geiger, R. Stuart and David Ribes (2010). “The Work of Sustaining Order in Wikipedia: The Banning of a Vandal.” In Proceedings of the 2010 ACM Conference on Computer-Supported Cooperative Work (CSCW 2012). New York: ACM Digital Library. http://www.stuartgeiger.com/papers/cscw-sustaining-order-wikipedia.pdf
Download Paper

The Social Roles of Bots and Assisted Editing Tools

Published in Proceedings of Wikisym, 2009

A short paper showing the recent explosive growth of automated editors (or bots) in Wikipedia, which have taken on many new tasks in administrative spaces.

Recommended citation: Geiger, R. Stuart (2009). “The Social Roles of Bots and Assisted Editing Tools.” In Proceedings of the 5th International Symposium on Wikis and Open Collaboration. New York: ACM Digital Library. http://www.stuartgeiger.com/papers/geiger-wikisym-bots.pdf
Download Paper

Does Habermas Understand the Internet? The Algorithmic Construction of the Blogo/Public Sphere

Published in Gnovis, 2009

Habermasians have been debating about the role of the Internet in the public sphere, but they have all taken for granted the highly-automated software infrastructures that mediate our knowledge of the blogosphere.

Recommended citation: Geiger, R. Stuart (2009). “Does Habermas Understand the Internet? The Algorithmic Construction of the Blogo/Public Sphere.” Gnovis: A Journal of Communication, Culture, and Technology. 10(1). http://www.stuartgeiger.com/papers/gnovis-habermas-blogopublic-sphere.pdf
Download Paper