research-article

Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction

Authors:

Maximilian Mackeprang,

Claudia M�ller-Birn,

Maximilian Timo StaussAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 3, Issue CSCW

Article No.: 195, Pages 1 - 30

https://doi.org/10.1145/3359297

Published: 07 November 2019 Publication History

Abstract

Interactive intelligent systems, i.e., interactive systems that employ AI technologies, are currently present in many parts of our social, public and political life. An issue reoccurring often in the development of these systems is the question regarding the level of appropriate human and computer contributions. Engineers and designers lack a way of systematically defining and delimiting possible options for designing such systems in terms of levels of automation. In this paper, we propose, apply and reflect on a method for human-computer configuration design. It supports the systematic investigation of the design space for developing an interactive intelligent system. We illustrate our method with a use case in the context of collaborative ideation. Here, we developed a tool for information extraction from idea content. A challenge was to find the right level of algorithmic support, whereby the quality of the information extraction should be as high as possible, but, at the same time, the human effort should be low. Such contradicting goals are often an issue in system development; thus, our method proposed helped us to conceptualize and explore the design space. Based on a critical reflection on our method application, we want to offer a complementary perspective to the value-centered design of interactive intelligent systems. Our overarching goal is to contribute to the design of so-called hybrid systems where humans and computers are partners.

References

[1]

Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine, Vol. 35, 4 (2014), 105--120.

Digital Library

[2]

Bruce W Arden. 1983. What Can Be Automated?: Computer Science and Engineering Research Study .MiT Press.

[3]

Emily M Bender and Batya Friedman. 2018. Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science . (March 2018).

[4]

Osvald M Bjelland and Robert Chapman Wood. 2008. An inside view of IBM's' Innovation Jam'. MIT Sloan management review, Vol. 50, 1 (2008), 32--40.

[5]

Andre Breitenfeld, Maximilian Mackeprang, and Ming-Tung Hong. 2017. Enabling Structured Data Generation by Nontechnical Experts. Mensch und Computer 2017-Tagungsband: Spielend einfach interagieren, Vol. 17 (Sept. 2017), 181--192.

[6]

Joel Chan, Pao Siangliulue, Denisa Qori McDonald, Ruixue Liu, Reza Moradinezhad, Safa Aman, Erin T Solovey, Krzysztof Z Gajos, and Steven P Dow. 2017. Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. ACM, 93--105.

Digital Library

[7]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, Vol. 20, 1 (1960), 37--46.

[8]

Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving Efficiency and Accuracy in Multilingual Entity Extraction. In Proceedings of the 9th International Conference on Semantic Systems (I-Semantics) .

[9]

John Danaher. 2016. The Threat of Algocracy: Reality, Resistance and Accommodation . Philosophy & Technology, Vol. 29, 3 (Jan. 2016), 245--268.

[10]

Ewart J de Visser, Richard Pak, and Tyler H Shaw. 2018. From textquoteleftautomationtextquoteright to textquoteleftautonomytextquoteright: the importance of trust repair in humantextendashmachine interaction . Ergonomics, Vol. 61, 10 (April 2018), 1409--1427.

[11]

Sidney W. A. Dekker and David D. Woods. 2002. MABA-MABA or Abracadabra? Progress on HumantextendashAutomation Coordination ., Vol. 4, 4 (Nov. 2002), 240--244.

[12]

Gianluca Demartini, Djellel Eddine Difallah, Ujwal Gadiraju, and Michele Catasta. 2017. An introduction to hybrid human-machine information systems. Foundations and Trends� in Web Science, Vol. 7, 1 (2017), 1--87.

[13]

Mica R Endsley and Esin O Kiris. 1995. The Out-of-the-Loop Performance Problem and Level of Control in Automation. Human Factors, Vol. 37, 2 (1995), 381--394.

[14]

Douglas C Engelbart. 1962. Augmenting Human Intellect: A Conceptual Framework. Technical Report SRI Project No. 3578. Stanford Research Institute, Menlo Park, Carlifornia.

[15]

Umer Farooq and Jonathan Grudin. 2016. Human-computer integration . interactions, Vol. 23, 6 (Oct. 2016), 26--32.

[16]

Paul M Fitts. 1951. Human engineering for an effective air-navigation and traffic-control system. (1951).

[17]

Terrence Fong, Charles Thorpe, and Charles Baur. 2001. Collaborative control: A robot-centric model for vehicle teleoperation.

[18]

Batya Friedman. 1996. Value-sensitive design . interactions, Vol. 3, 6 (Dec. 1996), 16--23.

[19]

Batya Friedman, David G Hendry, and Alan Borning. 2017. A Survey of Value Sensitive Design Methods. Foundations and Trends in Human-Computer Interaction, Vol. 11, 2 (2017), 63--125.

Digital Library

[20]

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume� III, and Kate Crawford. 2018. Datasheets for Datasets. (2018).

[21]

R. Stuart Geiger and Aaron Halfaker. 2017. Operationalizing Conflict and Cooperation Between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight'. Proc. ACM Hum.-Comput. Interact., Vol. 1, CSCW, Article 49 (Dec. 2017), bibinfonumpages33 pages.

[22]

R. Stuart Geiger and David Ribes. 2010. The Work of Sustaining Order in Wikipedia: The Banning of a Vandal. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA, 117--126.

Digital Library

[23]

Karni Gilon, Joel Chan, Felicia Y Ng, Hila Liifshitz-Assaf, Aniket Kittur, and Dafna Shahaf. 2018. Analogy mining for specific design needs. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI 2018). ACM, 121.

Digital Library

[24]

Victor Girotto, Erin Walker, and Winslow Burleson. 2017. The effect of peripheral micro-tasks on crowd ideation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1843--1854.

Digital Library

[25]

Marco Grassi, Christian Morbidoni, Michele Nucci, Simone Fonda, and Francesco Piazza. 2013. Pundit: augmenting web contents with semantics. Literary and linguistic computing, Vol. 28, 4 (2013), 640--659.

[26]

Jonathan Grudin and Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 209, bibinfonumpages11 pages.

Digital Library

[27]

Aaron Halfaker and John Riedl. 2012. Bots and Cyborgs: Wikipedia's Immune System. IEEE Computer, Vol. 45, 3 (2012), 79--82.

[28]

Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology . Vol. 52. Elsevier, 139--183.

[29]

Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The Dataset Nutrition Label - A Framework To Drive Higher Data Quality Standards. (2018).

[30]

Eric Horvitz. 1999 a. Principles of Mixed-initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 159--166.

Digital Library

[31]

Eric Horvitz. 1999 b. Uncertainty, Action, and Interaction: In Pursuit of Mixed-Initiative Computing . Intelligent Systems 6 (Sept. 1999), 17--20.

[32]

Charles L Isbell and Jeffrey S Pierce. 2005. An IP continuum for adaptive interface design . In HCI International .

[33]

Anthony Jameson and John Riedl. 2011. Introduction to the Transactions on Interactive Intelligent Systems. TiiS, Vol. 1, 1 (2011), 1--6.

Digital Library

[34]

Matthew Johnson, Jeffrey M Bradshaw, and Paul J Feltovich. 2017. Tomorrowtextquoterights HumantextendashMachine Design Tools: From Levels of Automation to Interdependencies: . Journal of Cognitive Engineering and Decision Making, Vol. 12, 1 (Oct. 2017), 77--82.

[35]

Matthew Johnson, Jeffrey M Bradshaw, Paul J Feltovich, Robert R Hoffman, Catholijn M Jonker, Birna van Riemsdijk, and Maarten Sierhuis. 2011. Beyond Cooperative Robotics - The Central Role of Interdependence in Coactive Design. IEEE Intelligent Systems, Vol. 26, 3 (2011), 81--88.

Digital Library

[36]

Nehemiah Jordan. 1963. Allocation of functions between man and machines in automated systems. Journal of applied psychology, Vol. 47, 3 (1963), 161.

[37]

David B Kaber and Mica R Endsley. 1997. The Combined Effect of Level of Automation and Adaptive Automation on Human Performance with Complex, Dynamic Control Systems . Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 41, 1 (1997), 205--209.

[38]

Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS '12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467--474.

Digital Library

[39]

Bruno Latour. 1988. Mixing Humans and Nonhumans Together: The Sociology of a Door-Closer . Social Problems, Vol. 35, 3 (June 1988), 298--310.

[40]

Claire Liang, Julia Proft, Erik Andersen, and Ross A. Knepper. 2019. Implicit Communication of Actionable Information in Human-AI Teams. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 95, bibinfonumpages13 pages.

[41]

Joseph Carl Robnett Licklider. 1960. Man-Computer Symbiosis . IRE transactions on human factors in electronics, Vol. HFE-1, 1 (March 1960), 4--11.

[42]

Maximilian Mackeprang, Abderrahmane Khiat, and Claudia M�ller-Birn. 2018a. Concept Validation During Collaborative Ideation and Its Effect on Ideation Outcome. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW033, bibinfonumpages6 pages.

[43]

Maximilian Mackeprang, Abderrahmane Khiat, and Claudia M�ller-Birn. 2018b. Innovonto: An Enhanced Crowd Ideation Platform with Semantic Annotation (Hallway Test) (FU Technical Reports Serie B), Vol. TR-B-18-02. Berlin.

[44]

Jose L Martinez-Rodriguez, Aidan Hogan, and Ivan Lopez-Arevalo. 2018. Information extraction meets the semantic web: a survey. Semantic Web Preprint (2018), 1--81.

[45]

Pablo N Mendes, Max Jakob, Andr�s Garc'ia-Silva, and Christian Bizer. 2011. DBpedia spotlight: shedding light on the web of documents. In Proceedings of the 7th international conference on semantic systems. ACM, 1--8.

Digital Library

[46]

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 220--229.

Digital Library

[47]

Judith S Olson and Wendy A Kellogg. 2014. Ways of Knowing in HCI . Vol. 2. Springer.

[48]

Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. 2000. A model for types and levels of human interaction with automation. IEEE Trans. Systems, Man, and Cybernetics, Part A, Vol. 30, 3 (2000), 286--297.

Digital Library

[49]

Raja Parasuraman and Christopher D Wickens. 2008. Humans: still vital after all these years of automation. Human Factors, Vol. 50, 3 (June 2008), 511--520.

[50]

Heiko Paulheim. 2017. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web, Vol. 8, 3 (2017), 489--508.

[51]

Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. ACM, 157--164.

Digital Library

[52]

Thomas B Sheridan and William L Verplank. 1978. Human and Computer Control of Undersea Teleoperators. Technical Report.

[53]

Ben Shneiderman and Pattie Maes. 1997. Direct Manipulation vs. Interface Agents . interactions, Vol. 4, 6 (Nov. 1997), 42--61.

[54]

Pao Siangliulue, Kenneth C Arnold, Krzysztof Z Gajos, and Steven P Dow. 2015a. Toward collaborative ideation at scale: Leveraging ideas from others to generate more creative and diverse ideas. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 937--945.

Digital Library

[55]

Pao Siangliulue, Joel Chan, Steven P Dow, and Krzysztof Z Gajos. 2016. IdeaHound: improving large-scale collaborative ideation with crowd-powered real-time semantic modeling. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 609--624.

Digital Library

[56]

Pao Siangliulue, Joel Chan, Krzysztof Z Gajos, and Steven P Dow. 2015b. Providing timely examples improves the quantity and quality of generated ideas. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition. ACM, 83--92.

Digital Library

[57]

Matti Tedre. 2008. FEATURE: What Should Be Automated? Interactions, Vol. 15, 5 (Sept. 2008), 47--49.

Digital Library

[58]

C. J. van Rijsbergen. 1979. Information Retrieval .Butterworth.

[59]

Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences, Vol. 39, 2 (2008), 273--315.

[60]

Toby Walsh. 2016. Turing's red flag . Commun. ACM, Vol. 59, 7 (2016), 34--37.

Digital Library

[61]

Antoine Widl�cher and Yann Mathet. 2012. The glozz platform: A corpus annotation and mining tool. In Proceedings of the 2012 ACM symposium on Document engineering. ACM, 171--180.

Digital Library

[62]

Langdon Winner. 1980. Do artifacts have politics? Daedalus (1980), 121--136.

[63]

Donghee Yoo, Keunho Choi, Hanjun Lee, and Yongmoo Suh. 2015. An Ontology-based Co-creation Enhancing System for Idea Recommendation in an Online Community. ACM SIGMIS Database: the DATABASE for Advances in Information Systems, Vol. 46, 3 (2015), 9--22.

Digital Library

[64]

Daisy Yoo, Alina Huldtgren, Jill Palzkill Woelfer, David G Hendry, and Batya Friedman. 2013. A value sensitive action-reflection model - evolving a co-design space with stakeholder and designer prompts. Proceedings of the SIGCHI conference on human factors in computing systems (2013), 419--428.

Digital Library

[65]

Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018. Value-Sensitive Algorithm Design: Method, Case Study, and Lessons . Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (Nov. 2018), 194--23.

Digital Library

Cited By

Park JWisniewski PSingh V(2024)Leveraging Large Language Models (LLMs) to Support Collaborative Human-AI Online Risk Data AnnotationSSRN Electronic Journal10.2139/ssrn.4774771Online publication date: 2024
https://doi.org/10.2139/ssrn.4774771
Joshi STolloczko CWenaas SHolm OLangved M(2024)Investigating How Generative AI Affects Decision-Making in Participatory Design: Shifting the space to make design choicesProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685384(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685384
Butcher BZilka MHron JCook DWeller A(2024)Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text DocumentsACM Journal on Responsible Computing10.1145/36525911:2(1-27)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3652591
Show More Cited By

Index Terms

Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interactive systems and tools

Recommendations

Human-Computer Collaborative Visual Design Creation Assisted by Artificial Intelligence
With the support and promotion of big data and cloud computing, AI has penetrated into every field of people's lives more and more deeply, with its characteristics of sustainable work, extremely fast computing speed, and a certain intelligence. This is an ...
Human-Computer Collaborative Interaction Design of Intelligent Vehicle—A Case Study of HMI of Adaptive Cruise Control
HCI in Mobility, Transport, and Automotive Systems
Abstract
With the rapid development of the intelligent vehicle industry, in order to improve the efficiency and usability of systems, the interface design of the Human-Computer Interaction of intelligent vehicles has received great attention. In this ...
Design-oriented human-computer interaction
CHI '03: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We argue that HCI has emerged as a design-oriented field of research, directed at large towards innovation, design, and construction of new kinds of information and interaction technology. But the understanding of such an attitude to research in terms ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 3, Issue CSCW

November 2019

5026 pages

EISSN:2573-0142

DOI:10.1145/3371885

Editors:
Airi Lampinen
Stockholm University, Sweden
,
Darren Gergle
Northwestern University, USA
,
David A. Shamma
FXPAL, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2019

Published in�PACMHCI�Volume 3, Issue CSCW

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
533
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)14

Reflects downloads up to 22 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Park JWisniewski PSingh V(2024)Leveraging Large Language Models (LLMs) to Support Collaborative Human-AI Online Risk Data AnnotationSSRN Electronic Journal10.2139/ssrn.4774771Online publication date: 2024
https://doi.org/10.2139/ssrn.4774771
Joshi STolloczko CWenaas SHolm OLangved M(2024)Investigating How Generative AI Affects Decision-Making in Participatory Design: Shifting the space to make design choicesProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685384(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685384
Butcher BZilka MHron JCook DWeller A(2024)Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text DocumentsACM Journal on Responsible Computing10.1145/36525911:2(1-27)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3652591
Wan QHu SZhang YWang PWen BLu Z(2024)"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language ModelsProceedings of the ACM on Human-Computer Interaction10.1145/36373618:CSCW1(1-26)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637361
Kim DVegt NVisch VBos-De Vos M(2024)How Much Decision Power Should (A)I Have?: Investigating Patients’ Preferences Towards AI Autonomy in Healthcare Decision MakingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642883(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642883
Feng LYen RYou YFan MZhao JLu Z(2024)CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language ProgrammingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642212(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642212
Kuang ELi MFan MShinohara K(2024)Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and TimingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642168(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642168
Ismail AThakkar DMadhiwalla NKumar N(2023)Public Health Calls for/with AI: An Ethnographic PerspectiveProceedings of the ACM on Human-Computer Interaction10.1145/36102037:CSCW2(1-26)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610203
Meier SGlinka K(2023)To Classify is to Interpret: Building Taxonomies from Heterogeneous Data through Human-AI CollaborationProceedings of Mensch und Computer 202310.1145/3603555.3608532(395-401)Online publication date: 3-Sep-2023
https://dl.acm.org/doi/10.1145/3603555.3608532
Shi CHu YWang SMa SZheng CMa XLuo Q(2023)RetroLens: A Human-AI Collaborative System for Multi-step Retrosynthetic Route PlanningProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581469(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581469
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents