research-article

Being a turker

Authors:

Benjamin V. Hanrahan,

Neha GuptaAuthors Info & Claims

CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

Pages 224 - 235

https://doi.org/10.1145/2531602.2531663

Published: 15 February 2014 Publication History

Abstract

Crowdsourcing is a key current topic in CSCW. We build upon findings of a few qualitative studies of crowdworkers. We conducted an ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (AMT) users. Using forum data we provide novel depth and detail on how the Turker Nation members operate as economic actors, working out which Requesters and jobs are worthwhile to them. We show some of the key ways Turker Nation functions as a community and also look further into Turker-Requester relationships from the Turker perspective -- considering practical, emotional and moral aspects. Finally, following Star and Strauss [25] we analyse Turking as a form of invisible work. We do this to illustrate practical and ethical issues relating to working with Turkers and AMT, and to promote design directions to support Turkers and their relationships with Requesters.

References

[1]

Adda, G., Sagot, B., Fort, K., Mariani, J. Crowdsourcing for Language Resource Development. 5th Language and Technology Conference. (2011).

[2]

Antin, J. and Shaw, A. Social desirability bias and selfreports of motivation: A study of amazon mechanical turk in the us and india. In Proc. of CHI'12, ACM press (2012), 2925--2934.

Digital Library

[3]

Bederson, B. B. and Quinn, A. J. Web workers unite! addressing challenges of online laborers. In Proc. CHI EA'11. ACM (2011), 97--106.

Digital Library

[4]

Felstiner, A. Working the crowd: employment and labor law in the crowdsourcing industry. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=15 93853 (2011).

[5]

Felstiner, A. The weakness of crowds. Limn, Crowds and Clouds (2), 2013. http://limn.it/the-weakness-ofcrowds/.

[6]

Fort, K., Adda, G. and Bretonnel Cohen, K. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguisitcs, 37, 2 (2011).

Digital Library

[7]

Garfinkel, H., Studies in ethnomethodology. Englewood Cliffs, N.J.: Prentice-Hall, (1967)

[8]

Howe, J. Crowdsourcing: A definition. (2006). http://crowdsourcing.typepad.com/cs/2006/06/crowdso urcing_a.html

[9]

Hughes, J., Randall, D., Shapiro, D. Faltering from Ethnography to Design. In Proc of CSCW'92. ACM (1992), 115--122

Digital Library

[10]

Ipeirotis, P. Analyzing the Amazon Mechanical Turk marketplace. CeDER Working Papers No. CeDER-1004. (2010) http://hdl.handle.net/2451/29801

[11]

Ipeirotis, P. Demographics of Mechanical Turk. CeDER Working Papers No. CeDER-10-01. (2010) http://hdl.handle.net/2451/29585

[12]

Irani, L. and Silberman, M. S. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proc. CHI 2013, ACM Press (2013).

Digital Library

[13]

Kaufmann, N., Schulze, T., and Veit, D. More than fun and money. Worker Motivation in Crowdsourcing -- A Study on Mechanical Turk. In Proc. Of AMCIS (2011).

[14]

Kittur,A., Chi, E. H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. In Proc. of the CHI'08. ACM, (2008), 453--456

Digital Library

[15]

Kittur, A., Smus, B. and Kraut, R. E. CrowdForge: Crowdsourcing Complex Work. Technical Report Carnegie Mellon University-HCII-11--100, http://crowdresearch.org/blog/wpcontent/.../rowdForge IST 011.pdf

[16]

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M. and Horton, J. The future of crowd work. In Proc. of CSCW'13, ACM press, 1301--1318.

Digital Library

[17]

Kochhar, S., Mazzochi, S., Paritosh, P. The Anatomy of a Large-Scale Human Computation Engine. H OMP'10. A M press (2010). 10--17

Digital Library

[18]

Little, G., Chilton, L. B., Goldman, M. and Miller, R. C. Turkit: tools for iterative tasks on mechanical turk. In KDD-HCOMP'09, 2009.

Digital Library

[19]

O'Neill, J. & Martin, D. Text hat In Action. In Proc. Group'03, ACM Press (2003), 40--49.

Digital Library

[20]

Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., and Tomlinson, B. Who are the crowdworkers?: shifting demographics in mechanical turk. In Proc. of CHI EA'10, ACM (2010), 2863--2872.

Digital Library

[21]

Sacks, H. Lectures on conversation. (2 vols). Gail Jefferson (Ed.). Oxford: Basil Blackwell, (1992).

[22]

Silberman, M. S. What's fair? Rational action and its residuals in an electronic market. Unpublished manuscript, (2010). http://www.scribd.com/doc/86592724/Whats-Fair

[23]

Silberman, M. S., Irani, L., and Ross, J. Ethics and Tactics of Professional Crowdwork. ACM XRDS, 17, 2 (2010)

Digital Library

[24]

Silberman, M., Ross, J., Irani, L. and Tomlinson, B. Sellers' problems in human computation markets. In Proc. of HCOMP'10. ACM Press (2010), 18--21.

Digital Library

[25]

Star, S.L. and Strauss, A.L. Layers of silence, arenas of voice: The ecology of visible and invisible work'. CSCW Journal, 8: 9--30. (1998)

Digital Library

[26]

Suchman, L. Plans and situated action: the problem of human-machine communication. Cambridge: Cambridge University Press (1987)

Digital Library

[27]

Watson, R., Analysing Practical and Professional Texts; A Naturalistic Approach. Farnham, Surrey: Ashgate, (2009).

[28]

Worstall, T. On The New York Times Stupidity over Amazon's Mechanical Turk. Forbes. http://www.forbes.com/sites/timworstall/2013/03/19/on-the-new-york-times-stupidity-over-amazonsmechanical-turk/ The columns on the last page should be of approximately equal length. Remove these two lines from your final version.

Cited By

Dunn KShklovski IBjørn P(2024)What Research through Art can bring to CSCW: exploring ambiguous futures of worki-com10.1515/icom-2023-003823:1(33-55)Online publication date: 22-Apr-2024
https://doi.org/10.1515/icom-2023-0038
Kingsley SSilberman MWang CLambeth RZhi JEslami MLi BBigham J(2024)‘Your Duties Are To Sweep A Floor Remotely’: Low Information Quality in Job Advertisements is a Barrier to Low-Income Job-Seekers’ Successful Use of Digital PlatformsProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663403(1-20)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663403
Cheon EXu S(2024)Creative Precarity in Motion: Revealing the Hidden Labor Behind Animating Virtual CharactersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661545(3471-3484)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661545
Show More Cited By

Index Terms

Being a turker
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

This paper aims to understand the self-organization phenomenon among the workers of Amazon Mechanical Turk (MTurk), a well-known crowdsourcing platform. Specifically, we explored 1) why MTurk workers self-organize into online communities (Turker ...
Turker Tales: Integrating Tangential Play into Crowd Work
DIS '19: Proceedings of the 2019 on Designing Interactive Systems Conference

While past work has admirably supported crowd workers in improving their work performance, we argue that there is also value in designing for enjoyment untied from work outcomes--- what we call "tangential play.'' To this end, we present Turker Tales, a ...
Efficient Crowd Exploration of Large Networks: The Case of Causal Attribution

Accurately and efficiently crowdsourcing complex, open-ended tasks can be difficult, as crowd participants tend to favor short, repetitive "microtasks". We study the crowdsourcing of large networks where the crowd provides the network topology via ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

February 2014

1600 pages

ISBN:9781450325400

DOI:10.1145/2531602

General Chairs:
Susan Fussell
Cornell University
,
Wayne Lutters
University of Maryland, Baltimore County
,
Program Chairs:
Meredith Ringel Morris
Microsoft Research
,
Madhu Reddy
Penn State University

Copyright � 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CSCW'14

Sponsor:

SIGCHI

CSCW'14: Computer Supported Cooperative Work

February 15 - 19, 2014

Maryland, Baltimore, USA

Acceptance Rates

CSCW '14 Paper Acceptance Rate 134 of 497 submissions, 27%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

215
Total Citations
View Citations
2,097
Total Downloads

Downloads (Last 12 months)142
Downloads (Last 6 weeks)4

Reflects downloads up to 21 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dunn KShklovski IBj�rn P(2024)What Research through Art can bring to CSCW: exploring ambiguous futures of worki-com10.1515/icom-2023-003823:1(33-55)Online publication date: 22-Apr-2024
https://doi.org/10.1515/icom-2023-0038
Kingsley SSilberman MWang CLambeth RZhi JEslami MLi BBigham J(2024)‘Your Duties Are To Sweep A Floor Remotely’: Low Information Quality in Job Advertisements is a Barrier to Low-Income Job-Seekers’ Successful Use of Digital PlatformsProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663403(1-20)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663403
Cheon EXu S(2024)Creative Precarity in Motion: Revealing the Hidden Labor Behind Animating Virtual CharactersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661545(3471-3484)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661545
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Chandhiramowuli STaylor AHeitlinger SWang D(2024)Making Data Work CountProceedings of the ACM on Human-Computer Interaction10.1145/36373678:CSCW1(1-26)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637367
Chong AYoo JCheshire C(2024)Perceptions of Fairness in Technology-Mediated MarketplacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642678(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642678
de Groot EGadiraju U(2024)"Are we all in the same boat?" Customizable and Evolving Avatars to Improve Worker Engagement and Foster a Sense of Community in Online Crowd WorkProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642429(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642429
Li YXiang B(2024)Reducing organizational inequalities associated with algorithmic controlsDiscover Artificial Intelligence10.1007/s44163-024-00137-04:1Online publication date: 15-May-2024
https://doi.org/10.1007/s44163-024-00137-0
Shaikh RMcGregor MBrown BLampinen A(2024)Fleeting Alliances and Frugal Collaboration in Piecework: A Video-Analysis of Food Delivery Work in IndiaComputer Supported Cooperative Work (CSCW)10.1007/s10606-024-09501-1Online publication date: 21-Jun-2024
https://doi.org/10.1007/s10606-024-09501-1
Maffie M(2023)Adversaries or Cross-Organization Co-workers? Exploring the Relationship between Gig Workers and Conventional EmployeesILR Review10.1177/0019793923119425477:1(3-31)Online publication date: 1-Sep-2023
https://doi.org/10.1177/00197939231194254
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents