Export Citations
Save this search
Please login to be able to save your searches and receive alerts for new content matching your search criteria.
- research-articleOctober 2024
XCrowd: Combining Explainability and Crowdsourcing to Diagnose Models in Relation Extraction
CIKM '24: Proceedings of the 33rd ACM International Conference on Information and Knowledge ManagementPages 2097–2107https://doi.org/10.1145/3627673.3679777Relation extraction methods are currently dominated by deep neural models, which capture complex statistical patterns while being brittle and vulnerable to perturbations in data and distribution. Explainability techniques offer a means for understanding ...
- research-articleMay 2024
APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT
WWW '24: Proceedings of the ACM Web Conference 2024Pages 245–255https://doi.org/10.1145/3589334.3645642Recent research has highlighted the potential of LLMs, like ChatGPT, for performing label annotation on social computing data. However, it is already well known that performance hinges on the quality of the input prompts. To address this, there has been ...
- research-articleMarch 2024
Enhancing Human-in-the-Loop Ontology Curation Results through Task Design
Journal of Data and Information Quality (JDIQ), Volume 16, Issue 1Article No.: 4, Pages 1–25https://doi.org/10.1145/3626960The success of artificial intelligence (AI) applications is heavily dependent on the quality of data they rely on. Thus, data curation, dealing with cleaning, organising, and managing data, has become a significant research area to be addressed. ...
- research-articleAugust 2023
Mitigating Voter Attribute Bias for Fair Opinion Aggregation
AIES '23: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and SocietyPages 170–180https://doi.org/10.1145/3600211.3604660The aggregation of multiple opinions plays a crucial role in decision-making, such as in hiring and loan review, and in labeling data for supervised learning. Although majority voting and existing opinion aggregation models are effective for simple ...
- research-articleApril 2022
Ready Player One! Eliciting Diverse Knowledge Using A Configurable Game
WWW '22: Proceedings of the ACM Web Conference 2022Pages 1709–1719https://doi.org/10.1145/3485447.3512241Access to commonsense knowledge is receiving renewed interest for developing neuro-symbolic AI systems, or debugging deep learning models. Little is currently understood about the types of knowledge that can be gathered using existing knowledge ...
-
- research-articleJanuary 2022
Flud: A Hybrid Crowd–Algorithm Approach for Visualizing Biological Networks
ACM Transactions on Computer-Human Interaction (TOCHI), Volume 29, Issue 1Article No.: 8, Pages 1–53https://doi.org/10.1145/3479196Modern experiments in many disciplines generate large quantities of network (graph) data. Researchers require aesthetic layouts of these networks that clearly convey the domain knowledge and meaning. However, the problem remains challenging due to ...
- research-articleOctober 2021
Skyline in Crowdsourcing with Imprecise Comparisons
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge ManagementPages 37–46https://doi.org/10.1145/3459637.3482479Given an input of a set of objects each one represented as a vector of features in a feature space, the problem of finding the skyline is the problem of determining the subset of objects that are not dominated by any other input object. An example of an ...
- research-articleOctober 2021
The platform belongs to those who work on it! Co-designing worker-centric task distribution models
OpenSym '21: Proceedings of the 17th International Symposium on Open CollaborationArticle No.: 1, Pages 1–12https://doi.org/10.1145/3479986.3479987Today, digital platforms are increasingly mediating our day-to-day work and crowdsourced forms of labour are progressively gaining importance (e.g. Amazon Mechanical Turk, Universal Human Relevance System, TaskRabbit). In many popular cases of ...
- research-articleOctober 2021
Applying Rapid Crowdsourced Playtesting to a Human Computation Game
FDG '21: Proceedings of the 16th International Conference on the Foundations of Digital GamesArticle No.: 50, Pages 1–7https://doi.org/10.1145/3472538.3472626Player engagement and task effectiveness are crucial factors in human computation games. However, collecting data and making design changes towards these goals can be time-consuming. In this work, we incorporate rapid crowdsourced playtesting via the ...
- extended-abstractMay 2021
Task Design for Crowdsourcing Complex Cognitive Skills
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsArticle No.: 51, Pages 1–7https://doi.org/10.1145/3411763.3443447Task design for crowdsourcing is a key factor limiting the quality of crowd-sourced results. This case study presents our design process for a complex cognitive task: generating Dimension/Values for categorizing ideas. Conveying the task to workers was ...
- research-articleMay 2021
The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsArticle No.: 341, Pages 1–14https://doi.org/10.1145/3411764.3445092An estimated 100,000 people work today as commercial content moderators. These moderators are often exposed to disturbing content, which can lead to lasting psychological and emotional distress. This literature review investigates moderators’ ...
- research-articleApril 2021
Can Crowds Customize Instructional Materials with Minimal Expert Guidance?: Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor
Proceedings of the ACM on Human-Computer Interaction (PACMHCI), Volume 5, Issue CSCW1Article No.: 119, Pages 1–24https://doi.org/10.1145/3449193AI-based educational technologies may be most welcome in classrooms when they align with teachers' goals, preferences, and instructional practices. Teachers, however, have scarce time to make such customizations themselves. How might the crowd be ...
- research-articleJune 2021
What do You Mean? Interpreting Image Classification with Crowdsourced Concept Extraction and Analysis
WWW '21: Proceedings of the Web Conference 2021Pages 1937–1948https://doi.org/10.1145/3442381.3450069Global interpretability is a vital requirement for image classification applications. Existing interpretability methods mainly explain a model behavior by identifying salient image patches, which require manual efforts from users to make sense of, and ...
- short-paperNovember 2020
We Asked 100 People: How Would You Train Our Robot?
CHI PLAY '20: Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in PlayPages 335–339https://doi.org/10.1145/3383668.3419864While robotic proficiency excels in constrained environments, the demand for vast amounts of world knowledge to cover unforeseen circumstances, constellations and tasks prevents sufficiently robust real-world application. Human computation has shown to ...
- research-articleOctober 2020
CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing
Proceedings of the ACM on Human-Computer Interaction (PACMHCI), Volume 4, Issue CSCW2Article No.: 132, Pages 1–24https://doi.org/10.1145/3415203Paid micro-task crowdsourcing has gained in popularity partly due to the increasing need for large-scale manually labelled datasets which are often used to train and evaluate Artificial Intelligence systems. Modern paid crowdsourcing platforms use a ...
- research-articleMay 2020
C-Reference: Improving 2D to 3D Object Pose Estimation Accuracy via Crowdsourced Joint Object Estimation
Proceedings of the ACM on Human-Computer Interaction (PACMHCI), Volume 4, Issue CSCW1Article No.: 51, Pages 1–28https://doi.org/10.1145/3392858Converting widely-available 2D images and videos, captured using an RGB camera, to 3D can help accelerate the training of machine learning systems in spatial reasoning domains ranging from in-home assistive robots to augmented reality to autonomous ...
- research-articleApril 2020
Towards Hybrid Human-AI Workflows for Unknown Unknown Detection
WWW '20: Proceedings of The Web Conference 2020Pages 2432–2442https://doi.org/10.1145/3366423.3380306Predictive models are susceptible to errors called unknown unknowns, in which the model assigns incorrect labels to instances with high confidence. These commonly arise when training data does not represent variations of a class encountered at model ...
- research-articleOctober 2019
Characterising volunteers' task execution patterns across projects on multi-project citizen science platforms
IHC '19: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing SystemsArticle No.: 16, Pages 1–11https://doi.org/10.1145/3357155.3358441Citizen science projects engage people in activities that are part of a scientific research effort. On multi-project citizen science platforms, scientists can create projects consisting of tasks. Volunteers, in turn, participate in executing the project'...
- Work in ProgressOctober 2019
Can You Rely on Human Computation?: A Large-scale Analysis of Disruptive Behavior in Games With A Purpose
CHI PLAY '19 Extended Abstracts: Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended AbstractsPages 605–610https://doi.org/10.1145/3341215.3356297Outsourcing effortful problems as microtasks has been successfully implemented by various human computation serious games or GWAP. Still, most of the academic ap-proaches validate their results by conducting laboratory studies. While these have the ...
- research-articleOctober 2019
Designing Videogames to Crowdsource Accelerometer Data Annotation for Activity Recognition Research
CHI PLAY '19: Proceedings of the Annual Symposium on Computer-Human Interaction in PlayPages 135–147https://doi.org/10.1145/3311350.3347153Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces and interventions. However, developing valid algorithms that use accelerometer data to detect everyday ...