Published in Proceedings of the 2020 ACM Conference on Fairness, Accountability, and Transparency (FAT* 2020), 2019
Many machine learning projects for new application areas involve teams of humans who label data for a particular purpose, from hiring crowdworkers to the paper’s authors labeling the data themselves. In this paper, we investigate to what extent a sample of machine learning application papers in social computing – specifically papers from ArXiv and traditional publications performing an ML classification task on Twitter data – give specific details about whether best practices in human annotation were followed.
Recommended citation: R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. "Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?" In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona, Spain. ACM, New York, NY, USA, 18 pages. https://stuartgeiger.com/papers/gigo-fat2020.pdf https://doi.org/10.1145/3351095.3372862
Download Paper
