research-article

Expert crowdsourcing with flash teams

Authors:

Daniela Retelny,

S�bastien Robaszkiewicz,

Walter S. Lasecki,

Melissa Valentine,

Michael S. BernsteinAuthors Info & Claims

UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

Pages 75 - 85

https://doi.org/10.1145/2642918.2647409

Published: 05 October 2014 Publication History

Abstract

We introduce flash teams, a framework for dynamically assembling and managing paid experts from the crowd. Flash teams advance a vision of expert crowd work that accomplishes complex, interdependent goals such as engineering and design. These teams consist of sequences of linked modular tasks and handoffs that can be computationally managed. Interactive systems reason about and manipulate these teams' structures: for example, flash teams can be recombined to form larger organizations and authored automatically in response to a user's request. Flash teams can also hire more people elastically in reaction to task needs, and pipeline intermediate output to accelerate completion times. To enable flash teams, we present Foundry, an end-user authoring platform and runtime manager. Foundry allows users to author modular tasks, then manages teams through handoffs of intermediate work. We demonstrate that Foundry and flash teams enable crowdsourcing of a broad class of goals including design prototyping, course development, and film animation, in half the work time of traditional self-managed teams.

Supplementary Material

suppl.mov (uistf4003-file3.mp4)

Supplemental video

Download
50.31 MB

References

[1]

oDesk. URL www.odesk.com.

[2]

Ahmad, S., Battle, A., Malkani, Z., and Kamvar, S. The jabberwocky programming environment for structured social computing. Proc. UIST '11, 2011.

Digital Library

[3]

Antin, J. and Shaw, A. Social desirability bias and self-reports of motivation: a cross-cultural study of Amazon Mechanical Turk in the US and India. Proc. CHI '12, 2012.

Digital Library

[4]

Baldwin, C.Y. and Clark, K.B. Managing in an age of modularity. Harvard Business Review, 75(5):84--93, 1997.

[5]

Bechky, B.A. Gaffers, gofers, and grips: role-based coordination in temporary organizations. Organization Science, 17(1):3--21, 2006.

Digital Library

[6]

Bernstein, M.S., et al. Soylent: a word processor with a crowd inside. Proc. UIST '10, 2010.

Digital Library

[7]

Bigham, J.P., et al. VizWiz: nearly real-time answers to visual questions. Proc. UIST '10, 2010.

Digital Library

[8]

Bunderson, J.S. and Boumgarden, P. Structure and learning in self-managed teams: why bureaucratic teams can be better learners. Organization Science, 21(3):609--624, 2010.

Digital Library

[9]

Chilton, L.B., et al. Cascade: crowdsourcing taxonomy creation. Proc. CHI '13, 2013.

Digital Library

[10]

Cooper, S., et al. Predicting protein structures with a multiplayer online game. Nature, 466(7307):756--760, 2010.

[11]

Cramton, C.D. The mutual knowledge problem and its consequences for dispersed collaboration. Organization Science, 12(3):346--371, 2001.

Digital Library

[12]

Cranshaw, J. and Kittur, A. The polymath project: lessons from a successful online collaboration in mathematics. Proc. CHI '11, 2011.

Digital Library

[13]

Dai, P., Mausam, and Weld, D. Decision-theoretic control of crowd-sourced workflows. Proc. AAAI '10, 2010.

[14]

Dourish, P. and Bellotti, V. Awareness and coordination in shared workspaces. Proc. CSCW '92, 1992.

Digital Library

[15]

Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. Shepherding the crowd yields better work. Proc. CSCW '12, 2012.

Digital Library

[16]

Dow, S.P., et al. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. TOCHI, 17(4):18, 2010.

Digital Library

[17]

Fikes, R.E. and Nilsson, N.J. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3):189--208, 1972.

[18]

Heath, C. and Staudenmayer, N. Coordination neglect: how lay theories of organizing complicate coordination in organizations. Research in Organizational Behavior, 22:153--191, 2000.

[19]

Hinds, P., Liu, L., and Lyon, J. Putting the global in global work: an intercultural lens on the practice of cross-national collaboration. The Academy of Management Annals, 5(1):135--188, 2011.

[20]

Hollan, J. and Stornetta, S. Beyond being there. Proc. CHI '92, 1992.

Digital Library

[21]

Hu, C., Bederson, B.B., Resnik, P., and Kronrod, Y. Monotrans2: A new human computation system to support monolingual translation. Proc. CHI '11, 2011.

Digital Library

[22]

Huckman, R.S., Staats, B.R., and Upton, D.M. Team familiarity, role experience, and performance: evidence from Indian software services. Management Science, 55(1):85--100, 2009.

Digital Library

[23]

Kautz, H.A., Selman, B., and Others. Planning as satisfiability. Proc. ECAI '92, 1992.

Digital Library

[24]

Kittur, A., Khamkar, S., Andre, P., and Kraut, R.E. CrowdWeaver: visually managing complex crowd work. Proc. CSCW '12, 2012.

Digital Library

[25]

Kittur, A., Smus, B., Khamkar, S., and Kraut, R.E. Crowdforge: crowdsourcing complex work. Proc. UIST '11, 2011.

Digital Library

[26]

Kittur, A., et al. The future of crowd work. Proc. CSCW '13, 2013.

Digital Library

[27]

Kulkarni, A., Can, M., and Hartmann, B. Collaboratively crowdsourcing workflows with turkomatic. Proc. CSCW '12, 2011.

Digital Library

[28]

Lasecki, W., et al. Real-time captioning by groups of non-experts. Proc. UIST '12, 2012.

Digital Library

[29]

Lasecki, W.S., et al. Real-time crowd control of existing interfaces. Proc. UIST '11, 2011.

Digital Library

[30]

Lasecki, W.S., et al. Chorus: a crowd-powered conversational assistant. Proc. UIST '13, 2013.

Digital Library

[31]

Lashinsky, A. Inside Apple. Hachette Book Group, New York, 2012.

Digital Library

[32]

Little, G., Chilton, L., Goldman, M., and Miller, R.C. Exploring iterative and parallel human computation processes. Proc. HCOMP '10, 2010.

Digital Library

[33]

Luther, K., Fiesler, C., and Bruckman, A. Redistributing leadership in online creative collaboration. Proc. CSCW '13, 2013.

Digital Library

[34]

Malone, T.W., Crowston, K., and Herman, G.A. Organizing Business Knowledge: The MIT Process Handbook. MIT Press, Cambridge, MA, USA, 2003.

Digital Library

[35]

Mark, G. Extreme collaboration. Communications of the ACM, 45(6):89--93, 2002.

Digital Library

[36]

Mathieu, J.E., et al. The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85(2):273--83, 2000.

[37]

Nardi, B.A. and Whittaker, S. The place of face-to-face communication in distributed work. In P.J. Hinds and S. Kiesler (editors), Distributed Work, pp. 83--113. MIT Press, Cambridge, 2002.

[38]

Noronha, J., Hysen, E., Zhang, H., and Gajos, K.Z. Platemate: crowdsourcing nutritional analysis from food photographs. Proc. UIST '11, 2011.

Digital Library

[39]

Olson, G. and Olson, J. Distance Matters. Human-Computer Interaction, 15(2):139--178, 2000.

Digital Library

[40]

Stol, K.J. and Fitzgerald, B. Researching crowdsourcing software development: perspectives and concerns. ICSE '14 Workshop on Crowdsourcing in Software Engineering, 2014.

Digital Library

[41]

Valentine, M. Team scaffolds: how minimal in-group structures support fast-paced teaming. Academy of Management Proceedings, 2012.

[42]

Yu, L. and Nickerson, J.V. Cooks or cobblers? Crowd creativity through combination. Proc. CHI '11, 2011.

Digital Library

[43]

Zhang, H., et al. Human computation tasks with global constraints. Proc. CHI '12, 2012.

Digital Library

Cited By

Long TGero KChilton L(2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661587
McInnis BPindus RKareem DGamboa SNebeker C(2024)Exploring the Future of Informed Consent: Applying a Service Design ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36373308:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637330
Fulker ZRiedl C(2024)Cooperation in the Gig Economy: Insights from Upwork FreelancersProceedings of the ACM on Human-Computer Interaction10.1145/36373148:CSCW1(1-20)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637314
Show More Cited By

Index Terms

Expert crowdsourcing with flash teams
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

This paper introduces flash organizations: crowds structured like organizations to achieve complex and open-ended goals. Microtask workflows, the dominant crowdsourcing structures today, only enable goals that are so simple and modular that their path ...
Expert Crowdsourcing with Flash Teams and Organizations
A workload-aware flash translation layer enhancing performance and lifespan of TLC/SLC dual-mode flash memory in embedded systems

Similar to traditional NAND flash memory, triple-level cell (TLC) flash memory is used as secondary storage to meet the fast growing demands on storage capacity. TLC flash memory exhibits attractive features such as shock resistance, high density, low ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

October 2014

722 pages

ISBN:9781450330695

DOI:10.1145/2642918

General Chair:
Hrvoje Benko
Microsoft Research, USA
,
Program Chairs:
Mira Dontcheva
Adobe, USA
,
Daniel Wigdor
University of Toronto, Canada

Copyright � 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Stanford University
National Science Foundation
The Hasso Plattner Design Thinking Research Program

Conference

UIST '14

Sponsor:

UIST '14: The 27th Annual ACM Symposium on User Interface Software and Technology

October 5 - 8, 2014

Hawaii, Honolulu, USA

Acceptance Rates

UIST '14 Paper Acceptance Rate 74 of 333 submissions, 22%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

156
Total Citations
View Citations
1,974
Total Downloads

Downloads (Last 12 months)131
Downloads (Last 6 weeks)62

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Long TGero KChilton L(2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661587
McInnis BPindus RKareem DGamboa SNebeker C(2024)Exploring the Future of Informed Consent: Applying a Service Design ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36373308:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637330
Fulker ZRiedl C(2024)Cooperation in the Gig Economy: Insights from Upwork FreelancersProceedings of the ACM on Human-Computer Interaction10.1145/36373148:CSCW1(1-20)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637314
Li YXiang B(2024)Reducing organizational inequalities associated with algorithmic controlsDiscover Artificial Intelligence10.1007/s44163-024-00137-04:1Online publication date: 15-May-2024
https://doi.org/10.1007/s44163-024-00137-0
Fang YJin LHan TZhang KDing X(2024)Context-Aware Automatic Splitting Method for�Structured Complex Crowdsourcing TasksComputer Supported Cooperative Work and Social Computing10.1007/978-981-99-9640-7_4(48-61)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-981-99-9640-7_4
Kim GYeo DLee JKim S(2023)Simulating Urban Element Design with Pedestrian Attention: Visual Saliency as Aid for More Visible Wayfinding DesignLand10.3390/land1202039412:2(394)Online publication date: 1-Feb-2023
https://doi.org/10.3390/land12020394
Dortheimer JYang SYang QSprecher A(2023)Conceptual Architectural Design at Scale: A Case Study of Community Participation Using CrowdsourcingBuildings10.3390/buildings1301022213:1(222)Online publication date: 13-Jan-2023
https://doi.org/10.3390/buildings13010222
Xiao ZLiao QZhou MGrandison TLi Y(2023)Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information AccessProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584031(2-18)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584031
Wang YPapangelis KLykourentzou ISaker MChamberlain AKhan VLiang HYue Y(2023)Tasks of a Different Color: How Crowdsourcing Practices Differ per Complex Task Type and Why This MattersProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581418(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581418
Gupta PNguyen TGonzalez CWoolley A(2023)Fostering Collective Intelligence in Human–AI Collaboration: Laying the Groundwork for COHUMAINTopics in Cognitive Science10.1111/tops.12679Online publication date: 29-Jun-2023
https://doi.org/10.1111/tops.12679
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents