extended-abstract

Who are the crowdworkers?: shifting demographics in mechanical turk

Authors:

M. Six Silberman,

Andrew Zaldivar,

Bill TomlinsonAuthors Info & Claims

CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

Pages 2863 - 2872

https://doi.org/10.1145/1753846.1753873

Published: 10 April 2010 Publication History

Abstract

Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous studies of the demographics and usage behaviors of MTurk workers. We describe how the worker population has changed over time, shifting from a primarily moderate-income, U.S.-based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers. This change in population points to how workers may treat Turking as a full-time job, which they rely on to make ends meet.

References

[1]

Amazon Mechanical Turk -- Participation Agreement. https://www.mturk.com/mturk/conditionsofuse.

[2]

von Ahn, L. Human computation. Pro. K-CAP, ACM (2007), 5--6.

Digital Library

[3]

von Ahn, L. and Dabbish, L. Designing games with a purpose. Commun. ACM 51, 8 (2008), 58--67.

Digital Library

[4]

Alonso, O., Rose, D.E., and Stewart, B. Crowdsourcing for relevance evaluation. SIGIR Forum 42, 2 (2008), 9--15.

Digital Library

[5]

Barley, S.R. and Kunda, G. Contracting: A new form of professional practice. Perspectives -- Academy of Management 20, 1 (2006), 45.

[6]

Barr, J. and Cabrera, L.F. AI gets a brain. Queue 4, 4 (2006), 24--29.

Digital Library

[7]

Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. Proc. CVPR 2009, IEEE (2009), 248--255.

[8]

Evans, B.M. and Chi, E.H. Towards a model of understanding social search. Proc. CSCW 2008, ACM (2008), 485--494.

Digital Library

[9]

Howe, J. The rise of crowdsourcing. Wired Magazine 14, 6 (2006).

[10]

Ipeirotis, P. Mechanical Turk: The Demographics. A Computer Scientist in a Business School, 2008. http://behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html.

[11]

Ipeirotis, P. Turker Demographics vs. Internet Demographics. A Computer Scientist in a Business School, 2009. http://behind-the-enemy-lines.blogspot.com/2009/03/turker-demographics-vs-internet.html.

[12]

Irani, L. and Silberman, M.S. "Agency and Exploitation in Mechanical Turk" at Internet as Playground and Factory Conference at The New School. 2009. http://www.slideshare.net/lirani/agency-and-exploitation-in-amazon-mechanical-turk.

[13]

Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc. CHI 2008, ACM (2008), 453--456.

Digital Library

[14]

Paolacci, G. and Massimo, W. Experimental Turk. http://experimentalturk.wordpress.com/.

[15]

Sala, M.C., Partridge, K., Jacobson, L., and Begole, J. An exploration into activity-informed physical advertising using pest. Proc. Pervasive Computing, (2007), 73.

Digital Library

[16]

Smith, V. New Forms of Work Organization. Annual Review of Sociology 23, 1 (1997), 315--339.

[17]

Snow, R., O'Connor, B., Jurafsky, D., and Ng, A.Y. Cheap and Fast-But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proc. EMNLP 2008, ACL (2008), 254--263.

Digital Library

[18]

Sorokin, A. and Forsyth, D. Utility data annotation with Amazon Mechanical Turk. Proc. CVPRW 2008, IEEE (2008), 1--8.

[19]

Zittrain, J. Ubiquitous human computing. Phil. Trans. R. Soc. A.: 366, 1881 (2008), 3813--3821.

Cited By

Trenge CGriffith J(2024)Master Turkers: An assessment of data qualityMeasurement Instruments for the Social Sciences10.5964/miss.136196Online publication date: 3-Jul-2024
https://doi.org/10.5964/miss.13619
Epperson RDiederich JGoeschl T(2024)How to Design the Ask? Funding Units vs. Giving MoneyManagement Science10.1287/mnsc.2021.00157Online publication date: 20-Jun-2024
https://doi.org/10.1287/mnsc.2021.00157
Ayanian AUluğ ÖRadke HZick A(2024)The social psychological predictors of men’s backlash responses to the #MeToo movementGroup Processes & Intergroup Relations10.1177/1368430223121049227:7(1680-1711)Online publication date: 30-Jan-2024
https://doi.org/10.1177/13684302231210492
Show More Cited By

Index Terms

Who are the crowdworkers?: shifting demographics in mechanical turk
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
  2. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Web-based interaction
2. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Computer supported cooperative work

Recommendations

How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
CrowdForge: crowdsourcing complex work
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing Systems

Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more ...
Pricing mechanisms for crowdsourcing markets
WWW '13: Proceedings of the 22nd international conference on World Wide Web

Every day millions of crowdsourcing tasks are performed in exchange for payments. Despite the important role pricing plays in crowdsourcing campaigns and the complexity of the market, most platforms do not provide requesters appropriate tools for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

April 2010

2219 pages

ISBN:9781605589305

DOI:10.1145/1753846

General Chairs:
Elizabeth Mynatt
Georgia Institute of Technology
,
Don Schoner
Georgia Institute of Technology
,
Program Chairs:
Geraldine Fitzpatrick
Vienna University of Technology
,
Scott Hudson
Carnegie Mellon University
,
Keith Edwards
Georgia Tech
,
Tom Rodden
University of Nottingham

Copyright � 2010 Copyright is held by the author/owner(s).

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 April 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Extended-abstract

Conference

CHI '10

Sponsor:

SIGCHI

CHI '10: CHI Conference on Human Factors in Computing Systems

April 10 - 15, 2010

Georgia, Atlanta, USA

Acceptance Rates

CHI EA '10 Paper Acceptance Rate 350 of 1,346 submissions, 26%;

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

605
Total Citations
View Citations
5,621
Total Downloads

Downloads (Last 12 months)174
Downloads (Last 6 weeks)13

Reflects downloads up to 21 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Trenge CGriffith J(2024)Master Turkers: An assessment of data qualityMeasurement Instruments for the Social Sciences10.5964/miss.136196Online publication date: 3-Jul-2024
https://doi.org/10.5964/miss.13619
Epperson RDiederich JGoeschl T(2024)How to Design the Ask? Funding Units vs. Giving MoneyManagement Science10.1287/mnsc.2021.00157Online publication date: 20-Jun-2024
https://doi.org/10.1287/mnsc.2021.00157
Ayanian AUluğ ÖRadke HZick A(2024)The social psychological predictors of men’s backlash responses to the #MeToo movementGroup Processes & Intergroup Relations10.1177/1368430223121049227:7(1680-1711)Online publication date: 30-Jan-2024
https://doi.org/10.1177/13684302231210492
Solomon MGenossar BGal AFekete JOmidvar-Tehrani BRong KShraga R(2024)CopycHats: Question Sequencing with Artificial AgentsProceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics10.1145/3665939.3665963(1-7)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3665939.3665963
Kingsley SSilberman MWang CLambeth RZhi JEslami MLi BBigham J(2024)‘Your Duties Are To Sweep A Floor Remotely’: Low Information Quality in Job Advertisements is a Barrier to Low-Income Job-Seekers’ Successful Use of Digital PlatformsProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663403(1-20)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663403
Cheon EKhovanskaya V(2024)Amazon Z to A: Speculative Design to Understand the Future of Labor-Intensive WorkplacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661561(314-327)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661561
Neshaei SRietsche RSu XWambsganss T(2024)Enhancing Peer Review with AI-Powered Suggestion Generation Assistance: Investigating the Design DynamicsProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645169(88-102)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645169
Munoz IKim PO'Neil CDunn MSawyer S(2024)Platformization of Inequality: Gender and Race in Digital Labor PlatformsProceedings of the ACM on Human-Computer Interaction10.1145/36373858:CSCW1(1-22)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637385
Nguyen TRazniewski SWeikum GSerra ESpezzano F(2024)Cultural Commonsense Knowledge for Intercultural DialoguesProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679768(1774-1784)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679768
G�ldi AWambsganss TNeshaei SRietsche R(2024)Intelligent Support Engages Writers Through Relevant Cognitive ProcessesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642549(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642549
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents