article

Evaluating the effectiveness of explanations for recommender systems

Authors:

Judith MasthoffAuthors Info & Claims

User Modeling and User-Adapted Interaction, Volume 22, Issue 4-5

Pages 399 - 439

https://doi.org/10.1007/s11257-011-9117-5

Published: 01 October 2012 Publication History

Abstract

When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distribution when evaluating effectiveness.

References

[1]

Ahn, J.W., Brusilovsky, P., Grady, J., He, D., Syn, S.Y.: Open user profiles for adaptive news systems: help or harm? In: Proceedings of the 16th International Conference on World Wide Web, pp. 11-20. Banff, Alberta, Canada (2007).

[2]

Ardissono, L., Goy, A., Petrone, G., Segnan, M., Torasso, P.: INTRIGUE: personalized recommendation of tourist attractions for desktop and handheld devices. Appl. Artif. Intell. 17, 687-714 (2003).

[3]

Bilgic, M., Mooney, R.J.: Explaining recommendations: satisfaction vs. promotion. In: Proceedings of the Workshop Beyond Personalization, in Conjunction with the International Conference on Intelligent User Interfaces, pp. 13-18. San Diego, CA (2005).

[4]

Billsus, D., Pazzani, M.J.: A personal news agent that talks, learns, and explains. In: Proceedings of the Third International Conference on Autonomous Agents, pp. 268-275. Seattle, WA (1999).

Digital Library

[5]

Carenini, G., Moore, D.J.: An empirical study of the influence of user tailoring on evaluative argument effectiveness. In: Proceedings of the 17th International Joint Conference on Artificial Intelligence, pp. 1307-1314. Seattle, WA (2001).

Digital Library

[6]

Chen, L., Pu, P.: Hybrid critiquing-based recommender systems. In: International Conference on Intelligent User Interfaces, pp. 22-31. Honolulu, HI, USA (2007).

[7]

Cho, Y., Im, I., Hiltz, J.F.S.R.: The impact of product category on customer dissatisfaction in cyberspace. Bus. Process Manag. J. 9(5), 635-651 (2003).

[8]

Cramer, H., Evers, V., Someren, M.V., Ramlal, S., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.: The effects of transparency on perceived and actual competence of a content-based recommender. In: Semantic Web User Interaction Workshop in Conjuction with the International Conference on Human Factors in Computing Systems, pp. 455-496. Florence, Italy (2008a).

[9]

Cramer, H.S.M., Evers, V., Ramlal, S., Someren, M. van, Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.J.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User Adapt. Interact. 18(5), 455-496 (2008b).

Digital Library

[10]

Czarkowski, M.: A scrutable adaptive hypertext. PhD thesis, University of Sydney (2006).

[11]

Dale, R.: Dynamic document delivery: generating natural language texts on demand. In: Proceedings of the 9th International Workshop on Database and Expert Systems Applications, DEXA '98, pp. 131-136. IEEE Computer Society, Vienna, Austria (1998).

[12]

Felfernig, A., Gula, B., Teppan, E.: User acceptance of knowledge-based recommenders. Mach. Percept. Artif. Intell. 70, 249-276 (2007).

[13]

Felfernigm, A., Gula, B., Letiner, G., Maier, M., Melcher, R., Schippel, S., Teppan, E.: A dominance model for the calculation of decoy products in recommendation environments. In: Symposium on Persuasive Technology in Conjuction with Artificial Intelligence and the Simulation of Behavior Convention, pp. 43-50. Aberdeen, Scotland (2008).

[14]

Guy, I., Ronen, I., Wilcox, E.: Do you know? Recommending people to invite into your social network. In: International Conference on Intelligent User Interfaces, pp. 77-86. Sanibel Island, FL, USA (2009a).

[15]

Guy, I., Zwerdling, N., Carmel, D., Ronen, I., Uziel, E., Yogev, S., Ofek-Koifman, S.: Personalized recommendation of social software items based on social relations. In: ACM Conference on Recommender systems, pp. 53-60. New York City, NY, USA (2009b).

[16]

H�ubl, G., Trifts, V.: Consumer decision making in online shopping environments: the effects of interactive decision aids. Market. Sci. 19, 4-21 (2000).

Digital Library

[17]

Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: ACM Conference on Computer Supported Cooperative Work, pp. 241-250. Philadelphia, PA, USA (2000).

[18]

Hingston, M.: User friendly recommender systems. Master's thesis, Sydney University, Australia (2006).

[19]

Laband, D.N.: An objective measure of search versus experience goods. Econ. Inq. 29(3), 497-509 (1991).

[20]

Masthoff, J.: The evaluation of adaptive systems. In: Patel, N. (ed.) Adaptive Evolutionary Information Systems, pp. 329-347. Idea group publishing, Hershey, PA (2002).

[21]

Masthoff, J.: Group modeling: selecting a sequence of television items to suit a group of viewers. User Model. User Adapt. Interact. 14, 37-85 (2004).

Digital Library

[22]

McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Thinking positively--explanatory feedback for conversational recommender systems. In: Explanation Workshop in Conjunction with the European Conference on Case-Based Reasoning, pp. 115-124. Madrid, Spain (2004).

[23]

McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: International Conference on Intelligent User Interfaces, pp. 175-182. San Diego, CA, USA (2005a).

[24]

McCarthy, K., Reilly, J., Smyth, B., Mcginty, L.: Generating diverse compound critiques. Artif. Intell. Rev. 24, 339-357 (2005b).

Digital Library

[25]

McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: International Conference on Human Factors in Computing Systems, pp. 1097-1101. Montreal, Canada (2006a).

[26]

McNee, S.M., Riedl, J., Konstan, J.A.: Making recommendations better: An analytic model for human-recommender interaction. In: Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006), pp. 1103-1108. Montreal, Canada (2006b).

[27]

McSherry, D.: Explanation in recommender systems. Artif. Intell. Rev. 24(2), 179-197 (2005).

Digital Library

[28]

Murphy, P.E., Enis, B.M.: Classifying products strategically. J. Market. 50, 24-42 (1986).

[29]

Oberlander, J., Mellish, C.: Final report on the ILEX project. online: http://www.hcrc.ed.ac.uk/ilex/final.html (1998).

[30]

Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20, 383-453 (2010).

Digital Library

[31]

Pommeranz, A., Broekens, J., Wiggers, P., Brinkman, W.P., Jonker, C.M.: Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process. User Model. User Adapt. Interact. 22 (2012).

[32]

Pu, P., Chen, L.: Trust building with explanation interfaces. In: International Conference on Intelligent User Interfaces, pp. 93-100. Sydney, Australia (2006).

[33]

Pu, P., Chen, L.: Trust-inspiring explanation interfaces for recommender systems. Knowl. Syst. 20, 542-556 (2007).

Digital Library

[34]

Pu, P., Chen, L., Hu, R.: Evaluating recommender systems from the user's perspective: survey of the state of the art. User Model. User Adapt. Interact. 22 (2012).

[35]

Rashid, A.M., Albert, I., Cosley, D., Lam, S.K., McNee, S.M., Konstan, J.A., Riedl, J.: Getting to know you: learning new user preferences in recommender systems. In: International Conference on Intelligent User Interfaces, pp. 127-134. San Francisco, CA, USA (2002).

[36]

Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Incremental critiquing. In: SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 143-151. Cambridge, UK (2004).

[37]

Ricci, F., Rokach, L., Shapira, B., Kantor, P. (eds.): Recommender Systems Handbook. Springer, Dordrecht (2010).

[38]

Roth-Berghofer, T., Schulz, S., Leake, D.B., Bahls, D.: Workshop on explanation-aware computing. In: European Conference on Artificial Intelligence, Patras, Greece (2008).

[39]

Roth-Berghofer, T., Tintarev, N., Leake, D.B.: Workshop on explanation-aware computing. In: International Joint Conference on Artificial Intelligence, Pasadena, CA, USA (2009).

[40]

Roth-Berghofer, T., Tintarev, N., Leake, D.B.: Workshop on explanation-aware computing. In: European Conference on Artificial Intelligence, Lisbon, Portugal (2010).

[41]

Shapiro, C.: Optimal pricing of experience goods. Bell J. Econ. 14(2), 497-507 (1983).

[42]

Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: Conference on Human Factors in Computing Systems, pp. 830-831. Minneapolis, MN, USA (2002).

[43]

Symeonidis, P., Nanopoulos, A., Manolopoulos, Y.: Justified recommendations based on content and rating data. In: Workshop on Web Mining and Web Usage Analysis in Conjunction with the International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA (2008).

[44]

Thompson, C.A., G�ker, M.H., Langley, P.: A personalized system for conversational recommendations. J. Artif. Intell. Res. 21, 393-428 (2004).

Digital Library

[45]

Tintarev, N., Masthoff, J.: Effective explanations of recommendations: user-centered design. In: Recommender Systems, pp. 153-156. Minneapolis, MN, USA (2007a).

[46]

Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: WPRSIUI Associated with ICDE'07, pp. 801-810. Istanbul, Turkey (2007b).

[47]

Tintarev, N., Masthoff, J.: Over- and underestimation in different product domains. In: Workshop on Recommender Systems in Conjunction with the European Conference on Artificial Intelligence, pp. 14-19. Patras, Greece (2008a).

[48]

Tintarev, N., Masthoff, J.: Personalizing movie explanations using commercial meta-data. In: International Conference on Adaptive Hypermedia, pp. 204-213. Hannover, Germany (2008b).

[49]

Tintarev, N., Masthoff, J.: Evaluating recommender explanations: Problems experienced and lessons learned for evaluation of adaptive systems. In: UCDEAS Workshop in Conjuction with UMAP, pp. 54-63, Trento, Italy (2009).

[50]

Tintarev, N., Masthoff, J. : Designing and evaluating explanations for recommender systems. In: Kantor, P.B., Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 479-510. Springer, Dordrecht (2010).

[51]

Vig, J., Sen, S., Riedl, J.: Tagsplanations: Explaining recommendations using tags. In: International Conference on Intelligent User Interfaces, pp. 47-56. Sanibel Island, FL, USA (2009).

[52]

Wang, W., Benbasat, I.: Recommendation agents for electronic commerce: effects of explanation facilities on trusting beliefs. J. Manag. Inf. Syst. 23, 217-246 (2007).

Digital Library

[53]

W�rnest�l, P.: User evaluation of a conversational recommender system. In: Workshop on Knowledge and Reasoning in Practical Dialogue Systems in Conjunction with the International Joint Conference on Artificial Intelligence, pp. 32-39. Edinburgh, Scotland (2005).

Cited By

Chatti MGuesmi MMuslim A(2024)Visualization for Recommendation Explainability: A Survey and New PerspectivesACM Transactions on Interactive Intelligent Systems10.1145/367227614:3(1-40)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3672276
Garcia KVontobel JMayer S(2024)A Digital Companion Architecture for Ambient IntelligenceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596108:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659610
Tran TPolat Erdeniz SFelfernig ALubos SEl Mansi MLe V(2024)Less is More: Towards Sustainability-Aware Persuasive Explanations in Recommender SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3691708(1108-1112)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3691708
Show More Cited By

Evaluating the effectiveness of explanations for recommender systems
1. Human-centered computing

Recommendations

LLM-generated Explanations for Recommender Systems
UMAP Adjunct '24: Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization

Users are often confronted with situations where they have to decide in favor or against an offered item, like a book, movie, or recipe. Those suggested items are commonly determined by a recommender system, which considers personal preferences to ...
Knowledgeable Explanations for Recommender Systems
WI-IAT '10: Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology - Volume 01

Recommender Systems (RS) serve online customers in identifying those items from a variety of choices that best match their needs and preferences. In this context explanations summarize the reasons why a specific item is proposed and strongly increase ...
A Novel Framework for Improving Recommender Diversity
International Workshop on Behavior and Social Informatics on Behavior and Social Computing - Volume 8178

Recommender systems are being used to assist users in finding relevant items from a large set of alternatives in many online applications. However, while most research up to this point has focused on improving the accuracy of recommender systems, other ...

Comments

Information & Contributors

Information

Published In

cover image User Modeling and User-Adapted Interaction

User Modeling and User-Adapted Interaction Volume 22, Issue 4-5

October 2012

190 pages

ISSN:0924-1868

Issue’s Table of Contents

Copyright © Copyright © 2012 Springer Science+Business Media B.V.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 October 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

141
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chatti MGuesmi MMuslim A(2024)Visualization for Recommendation Explainability: A Survey and New PerspectivesACM Transactions on Interactive Intelligent Systems10.1145/367227614:3(1-40)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3672276
Garcia KVontobel JMayer S(2024)A Digital Companion Architecture for Ambient IntelligenceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596108:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659610
Tran TPolat Erdeniz SFelfernig ALubos SEl Mansi MLe V(2024)Less is More: Towards Sustainability-Aware Persuasive Explanations in Recommender SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3691708(1108-1112)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3691708
Shashaani S(2024)Explainability in Music Recommender SystemProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688028(1395-1401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688028
Okoso AOtaki KKoide SBaba Y(2024)Toward Tone-Aware Explanations in Recommender SystemsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659572(261-266)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659572
Kotkov DMedlar ALiu YGlowacka DHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)On the Negative Perception of Cross-domain Recommendations and ExplanationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657735(2102-2113)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657735
Nam DMacvean AMyers BVasilescu B(2024)Understanding Documentation Use Through Log Analysis: A Case Study of Four Cloud ServicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642721(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642721
Silva ATambwekar PSchrum MGombolay MGrollman DBroadbent EJu WSoh HWilliams T(2024)Towards Balancing Preference and Performance through Adaptive Personalized ExplainabilityProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3635000(658-668)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3635000
Wang XLi QYu DLi QXu G(2024)Reinforced Path Reasoning for Counterfactual Explainable RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.335407736:7(3443-3459)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3354077
de Campos LFern�ndez-Luna JHuete J(2024)An explainable content-based approach for recommender systems: a case study in journal recommendation for paper submissionUser Modeling and User-Adapted Interaction10.1007/s11257-024-09400-634:4(1431-1465)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11257-024-09400-6
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents