article

Performance issues and error analysis in an open-domain question answering system

Authors:

Sanda Harabagiu,

Mihai SurdeanuAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 21, Issue 2

Pages 133 - 154

https://doi.org/10.1145/763693.763694

Published: 01 April 2003 Publication History

Abstract

This paper presents an in-depth analysis of a state-of-the-art Question Answering system. Several scenarios are examined: (1) the performance of each module in a serial baseline system, (2) the impact of feedbacks and the insertion of a logic prover, and (3) the impact of various retrieval strategies and lexical resources. The main conclusion is that the overall performance depends on the depth of natural language processing resources and the tools used for answer finding.

References

[1]

Abney, S., Collins, M., and Singhal, A. 2000. Answer extraction. In Proceedings of the 6th Applied Natural Language Processing Conference (ANLP-2000, Seattle, WA). 296--301.

[2]

Breck, E., Burger, J., Ferro, L., Hirschman, L., House, D., Light, M., and Mani, I. 2000. How to evaluate your question answering system every day … and still get real work done. In Proceedings of the 2nd Conference on Language Resources and Evaluation (LREC-2000, Athens, Greece). 1495--1500.

[3]

Breck, E., Light, M., Mann, G., Riloff, E., Brown, B., Anand, P., Rooth, M., and Thelen, M. 2001. Looking under the hood: Tools for diagnosing your question answering engine. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics Workshop on Open-Domain Question Answering (ACL-01, Toulouse, France). 1--8.

[4]

Cardie, C., Ng, V., Pierce, D., and Buckley, C. 2000. Examining the role of statistical and linguistic knowledge sources in a general-knowledge question-answering system. In Proceedings of the 6th Applied Natural Language Processing Conference (ANLP-2000, Seattle, WA). 180--187.

[5]

Clarke, C., Cormack, G., Laszlo, M., Lynam, T., and Terra, E. 2002. The impact of corpus size on question answering performance. In Proceedings of the 25th ACM Conference on Research and Development in Information Retrieval, Poster session (SIGIR-2002, Tampere, Finland). ACM Press, New York, NY, 367--368.

[6]

Clarke, C., Cormack, G., and Lynam, T. 2001. Exploiting redundancy in question answering. In Proceedings of the 24th ACM Conference on Research and Development in Information Retrieval (SIGIR-2001, New Orleans, LA). ACM Press, New York, NY, 358--365.

[7]

Gaizauskas, R. and Humphreys, K. 2000. A combined IR/NLP approach to question answering against large text collections. In Proceedings of the 6th Content-Based Multimedia Information Access Conference (RIAO-2000, Paris, France). 1288--1304.

[8]

Harabagiu, S., Moldovan, D., Paşca, M., Surdeanu, M., Mihalcea, R., Gîrju, R., Rus, V., Morărescu, F. L. P., and Bunescu, R. 2001. Answering complex, list and context questions with lcc's question-answering server. In Proceedings of the 10th Text REtrieval Conference (TREC-2001). NIST, Gaithersburg, MD. 355--361.

[9]

Harabagiu, S., Paşca, M., and Maiorano, S. 2000. Experiments with open-domain textual question answering. In Proceedings of the 18th International Conference on Computational Linguistics (COLING-2000, Saarbrucken, Germany). 292--298.

[10]

Hovy, E., Gerber, L., Hermjakob, U., Lin, C., and Ravichandran, D. 2001. Toward semantics-based answer pinpointing. In Proceedings of the Human Language Technology Conference (HLT-2001, San Diego, CA). 339--345.

[11]

Ittycheriah, A., Franz, M., Zhu, W., and Ratnaparkhi, A. 2001. Question answering using maximum-entropy components. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL-2001, Pittsburgh, PA). 33--39.

[12]

Light, M., Mann, G., Riloff, E., and Breck, E. 2001. Analyses for elucidating current question answering technology. Nat. Lang. Eng. (Special Issue on Question Answering). 7, 4, 325--342.

[13]

Moldovan, D., Harabagiu, S., Gîrju, R., Morărescu, P., Lăcătuşu, F., Novischi, A., Bădulescu, A., and Bolohan, O. 2002. Lcc tools for question answering. In Proceedings of the 11th Text REtrieval Conference (TREC-2002). NIST, Gaithersburg, MD, 144--155.

[14]

Paşca, M. 2001. High-performance, open-domain question answering from large text collections. Ph.D. thesis, Southern Methodist University, Dallas, TX.

[15]

Paşca, M. and Harabagiu, S. 2001. The informative role of WordNet in open-domain question answering. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics, Workshop on WordNet and Other Lexical Resources: Applications, Extensions and Customizations. (NAACL-01, Pittsburgh, PA). 138--143.

[16]

Prager, J., Brown, E., Coden, A., and Radev, D. 2000. Question answering by predictive annotation. In Proceedings of the 23rd International Conference on Research and Development in Information Retrieval (SIGIR-2000, Athens, Greece). 184--191.

[17]

Salton, G., Allan, J., and Buckley, C. 1993. Approaches to passage retrieval in full text information systems. In Proceedings of the 16th ACM Conference on Research and Development in Information Retrieval (SIGIR-93, Pittsburgh, PA). 49--58.

[18]

Salton, G. and Buckley, C. 1988. Term-weighting approaches in automatic text retrieval. Informa. Proc. Manage. 24, 5, 513--523.

[19]

Salton, G. and McGill, M. 1983. Introduction to Modern Information Retrieval. McGraw-Hill, New York, NY.

[20]

Savoy, J. 1997. Ranking schemes in hybrid boolean systems: A new approach. J. Amer. Soc. Inform. Sci. 48, 3 (June), 235--253.

[21]

Srihari, R. and Li, W. 2000. A question answering system supported by information extraction. In Proceedings of the 6th Applied Natural Language Processing Conference (ANLP-2000, Seattle, WA). 166--172.

[22]

Voorhees, E. 1999. The TREC-8 Question Answering track report. In Proceedings of the 8th Text REtrieval Conference (TREC-8). NIST, Gaithersburg, MD, 77--82.

[23]

Voorhees, E. 2001. Overview of the TREC 2001 Question Answering track. In Proceedings of the 10th Text REtrieval Conference (TREC-2001). NIST, Gaithersburg, MD, 42--51.

[24]

Voorhees, E. and Tice, D. 2000. Building a question-answering test collection. In Proceedings of the 23rd International Conference on Research and Development in Information Retrieval (SIGIR-2000, Athens, Greece). 200--207.

Cited By

Setty VKlein MBen-David AJ�schke RKelly M(2024)Extreme Classification for Answer Type Prediction in Question AnsweringProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00041(232-236)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1109/JCDL57899.2023.00041
Pramanik SAlabi JRoy RWeikum G(2024)Uniqorn: Unified question answering over RDF knowledge graphs and natural language textJournal of Web Semantics10.1016/j.websem.2024.10083383(100833)Online publication date: Dec-2024
https://doi.org/10.1016/j.websem.2024.100833
Zhao XHuang JZhang JSong Y(2024)The Comprehensive Analysis of the Effect of Chinese Word Segmentation on Fuzzy-Based Classification Algorithms for Agricultural QuestionsInternational Journal of Fuzzy Systems10.1007/s40815-024-01724-0Online publication date: 20-May-2024
https://doi.org/10.1007/s40815-024-01724-0
Show More Cited By

Index Terms

Performance issues and error analysis in an open-domain question answering system

Recommendations

Performance issues and error analysis in an open-domain Question Answering system
ACL '02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics

This paper presents an in-depth analysis of a state-of-the-art Question Answering system. Several scenarios are examined: (1) the performance of each module in a serial baseline system, (2) the impact of feedbacks and the insertion of a logic prover, ...
Architecture and evaluation of BRUJA, a multilingual question answering system
Abstract
Given a user question, the goal of a Question Answering (QA) system is to retrieve answers rather than full documents or even best-matching passages, as most Information Retrieval systems currently do. In this paper, we present BRUJA, a QA system ...
Human question answering performance using an interactive document retrieval system
IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

Every day, people answer their questions by using document retrieval systems. Compared to document retrieval systems, question answering (QA) systems aim to speed the rate at which users find answers by retrieving answers rather than documents. To ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 21, Issue 2

April 2003

95 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/763693

Issue’s Table of Contents

Copyright © 2003 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2003

Published in TOIS Volume 21, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

127
Total Citations
View Citations
2,391
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)2

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Setty VKlein MBen-David AJ�schke RKelly M(2024)Extreme Classification for Answer Type Prediction in Question AnsweringProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00041(232-236)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1109/JCDL57899.2023.00041
Pramanik SAlabi JRoy RWeikum G(2024)Uniqorn: Unified question answering over RDF knowledge graphs and natural language textJournal of Web Semantics10.1016/j.websem.2024.10083383(100833)Online publication date: Dec-2024
https://doi.org/10.1016/j.websem.2024.100833
Zhao XHuang JZhang JSong Y(2024)The Comprehensive Analysis of the Effect of Chinese Word Segmentation on Fuzzy-Based Classification Algorithms for Agricultural QuestionsInternational Journal of Fuzzy Systems10.1007/s40815-024-01724-0Online publication date: 20-May-2024
https://doi.org/10.1007/s40815-024-01724-0
Mohasseb AKanavos A(2023)Grammar-Based Question Classification Using Ensemble Learning AlgorithmsWeb Information Systems and Technologies10.1007/978-3-031-43088-6_5(84-97)Online publication date: 29-Aug-2023
https://doi.org/10.1007/978-3-031-43088-6_5
Han DTohti THamdulla A(2022)Attention-Based Transformer-BiGRU for Question ClassificationInformation10.3390/info1305021413:5(214)Online publication date: 20-Apr-2022
https://doi.org/10.3390/info13050214
Zope BMishra SShaw KVora DKotecha KBidwe R(2022)Question Answer System: A State-of-Art Representation of Quantitative and Qualitative AnalysisBig Data and Cognitive Computing10.3390/bdcc60401096:4(109)Online publication date: 7-Oct-2022
https://doi.org/10.3390/bdcc6040109
Athanassoulis MTriantafillou PAppuswamy RBordawekar RChandramouli BCheng XManolescu IPapakonstantinou YTatbul N(2022)Artifacts Availability & Reproducibility (VLDB 2021 Round Table)ACM SIGMOD Record10.1145/3552490.355251151:2(74-77)Online publication date: 29-Jul-2022
https://dl.acm.org/doi/10.1145/3552490.3552511
Amer-Yahia SAmsterdamer YBhowmick SBonifati ABonnet PBorovica-Gajic RCatania BCerquitelli TChiusano SChrysanthis PCurino CDarmont JEl Abbadi AFloratou AFreire JJindal AKalogeraki VKoutrika GKumar AMaiyya SMeliou AMohanty MNaumann FNoack N�zcan FPeterfreund LRahayu WTan WTian YT�z�n PVargas-Solar GYadwadkar NZhang M(2022)Diversity and Inclusion Activities in Database ConferencesACM SIGMOD Record10.1145/3552490.355251051:2(69-73)Online publication date: 29-Jul-2022
https://dl.acm.org/doi/10.1145/3552490.3552510
Psallidas FZhu YKarlas BHenkel JInterlandi MKrishnan SKroth BEmani VWu WZhang CWeimer MFloratou ACurino CKaranasos K(2022)Data Science Through the Looking GlassACM SIGMOD Record10.1145/3552490.355249651:2(30-37)Online publication date: 29-Jul-2022
https://dl.acm.org/doi/10.1145/3552490.3552496
Hamza AEn-Nahnahi NEl Mahdaouy AEl Alaoui Ouatik S(2022)Embedding arabic questions by feature-level fusion of word representations for questions classificationJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2022.03.01534:9(6583-6594)Online publication date: 1-Oct-2022
https://dl.acm.org/doi/10.1016/j.jksuci.2022.03.015
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents