Human Interest or Conflict? Leveraging LLMs for Automated Framing Analysis in TV Shows
Pages 157 - 167
Abstract
In the current media landscape, understanding the framing of information is crucial for critical consumption and informed decision making. Framing analysis is a valuable tool for identifying the underlying perspectives used to present information, and has been applied to a variety of media formats, including television programs. However, manual analysis of framing can be time-consuming and labor-intensive. This is where large language models (LLMs) can play a key role. In this paper, we propose a novel approach to use prompt-engineering to identify the framing of spoken content in television programs. Our findings indicate that prompt-engineering LLMs can be used as a support tool to identify frames, with agreement rates between human and machine reaching up to 43%. As LLMs are still under development, we believe that our approach has the potential to be refined and further improved. The potential of this technology for interactive media applications is vast, including the development of support tools for journalists, educational resources for students of journalism learning about framing and related concepts, and interactive media experiences for audiences.
References
[1]
Toril Aalberg and James Curran. 2012. How media inform democracy: A comparative approach. Routledge.
[2]
Mohammad Ali and Naeemul Hassan. 2022. A survey of computational framing analysis approaches. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 9335–9348.
[3]
Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2023. Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks. arxiv:2307.02179 [cs.CL]
[4]
David Alonso del Barrio and Daniel Gatica-Perez. 2023. Framing the News: From Human Perception to Large Language Model Inferences. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval (Thessaloniki, Greece) (ICMR ’23). Association for Computing Machinery, New York, NY, USA, 627–635. https://doi.org/10.1145/3591106.3592278
[5]
Nic Badullovich, Will J Grant, and Rebecca M Colvin. 2020. Framing climate change for effective communication: a systematic map. Environmental research letters 15, 12 (2020), 123002.
[6]
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023).
[7]
Vibhu Bhatia, Vidya Prasad Akavoor, Sejin Paik, Lei Guo, Mona Jalal, Alyssa Smith, David Assefa Tofu, Edward Edberg Halim, Yimeng Sun, Margrit Betke, 2021. OpenFraming: open-sourced tool for computational framing analysis of multilingual data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 242–250.
[8]
Gonzalo Espinoza Bianchini, Lisa Zanotti, and Carlos Meléndez. 2023. Using OpenAI models as a new tool for text analysis in political leaders’ unstructured discourse. (2023).
[9]
Dallas Card, Amber Boydstun, Justin Gross, Philip Resnik, and Noah Smith. 2015. The Media Frames Corpus: Annotations of Frames Across Issues. 2 (01 2015), 438–444. https://doi.org/10.3115/v1/P15-2072
[10]
Marcelo Carvalho Afonso, Pedro Almeida, Pedro Beça, Telmo Silva, and Iulia Covalenco. 2022. Usability Of Text-To-Speech Technology in Creating News Podcasts using Portuguese Of Portugal. In Proceedings of the 2022 ACM International Conference on Interactive Media Experiences (Aveiro, JB, Portugal) (IMX ’22). Association for Computing Machinery, New York, NY, USA, 363–368. https://doi.org/10.1145/3505284.3532968
[11]
Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. 2023. LLM-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306.14924 (2023).
[12]
Sarah Cohen, James T. Hamilton, and Fred Turner. 2011. Computational journalism. Commun. ACM 54, 10 (oct 2011), 66–71. https://doi.org/10.1145/2001269.2001288
[13]
Claes H. de Vreese. 2004. The Effects of Frames in Political Television News on Issue Interpretation and Frame Salience. Journalism & Mass Communication Quarterly 81, 1 (2004), 36–52. https://doi.org/10.1177/107769900408100104 arXiv:https://doi.org/10.1177/107769900408100104
[14]
Claes H De Vreese. 2005. News framing: Theory and typology. Information design journal+ document design 13, 1 (2005), 51–62.
[15]
Nicholas Diakopoulos. 2019. Automating the news: How algorithms are rewriting the media. Harvard University Press.
[16]
Astrid Dirikx and Dave Gelders. 2010. To frame is to explain: A deductive frame-analysis of Dutch and French climate change coverage during the annual UN Conferences of the Parties. Public understanding of science 19, 6 (2010), 732–742.
[17]
Robert M Entman. 1993. Framing: Towards clarification of a fractured paradigm. McQuail’s reader in mass communication theory 390 (1993), 397.
[18]
Robert M Entman, Jörg Matthes, and Lynn Pellicano. 2009. Nature, sources, and effects of news framing. The handbook of journalism studies (2009), 175–190.
[19]
Bahareh Fatemi, Fazle Rabbi, and Andreas L. Opdahl. 2023. Evaluating the Effectiveness of GPT Large Language Model for News Classification in the IPTC News Ontology. IEEE Access 11 (2023), 145386–145394. https://doi.org/10.1109/ACCESS.2023.3345414
[20]
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. arXiv preprint arXiv:2305.08283 (2023).
[21]
Fernando Filgueiras. 2023. Artificial intelligence and education governance. Education, Citizenship and Social Justice (2023), 17461979231160674.
[22]
Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. Large language models in education: Vision and opportunities. In 2023 IEEE International Conference on Big Data (BigData). IEEE, 4776–4785.
[23]
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing using Language Models. In Proceedings of the 2022 ACM Designing Interactive Systems Conference (, Virtual Event, Australia,) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1002–1019. https://doi.org/10.1145/3532106.3533533
[24]
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, 30 (July 2023). https://doi.org/10.1073/pnas.2305016120
[25]
Katerina Gorkovenko and Nick Taylor. 2019. Audience and Expert Perspectives on Second Screen Engagement with Political Debates. In Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video (Salford (Manchester), United Kingdom) (TVX ’19). Association for Computing Machinery, New York, NY, USA, 70–82. https://doi.org/10.1145/3317697.3323352
[26]
Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. 2023. AI and the transformation of social science research. Science 380, 6650 (2023), 1108–1109.
[27]
Muhammad Usman Hadi, qasem al tashi, Rizwan Qureshi, Abbas Shah, amgad muneer, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, and Seyedali Mirjalili. 2023. Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects. (Nov. 2023). https://doi.org/10.36227/techrxiv.23589741.v4
[28]
Bahareh R. Heravi. 2019. 3Ws of Data Journalism Education. Journalism Practice 13, 3 (2019), 349–366. https://doi.org/10.1080/17512786.2018.1463167 arXiv:https://doi.org/10.1080/17512786.2018.1463167
[29]
Shima Khanehzar, Andrew Turpin, and Gosia Mikołajczak. 2019. Modeling Political Framing Across Policy Issues and Contexts. In ALTA.
[30]
Soomin Kim, JongHwan Oh, and Joonhwan Lee. 2016. Automated News Generation for TV Program Ratings. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video (Chicago, Illinois, USA) (TVX ’16). Association for Computing Machinery, New York, NY, USA, 141–145. https://doi.org/10.1145/2932206.2933561
[31]
Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting frames in news headlines and its application to analyzing news framing trends surrounding US gun violence. In Proceedings of the 23rd conference on computational natural language learning (CoNLL).
[32]
Neil Maiden, Konstantinos Zachos, Amanda Brown, George Brock, Lars Nyre, Aleksander Nygård Tonheim, Dimitris Apsotolou, and Jeremy Evans. 2018. Making the news: Digital creativity support for journalists. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11.
[33]
Fiona Fui-Hoon Nah, Ruilin Zheng, Jingyuan Cai, Keng Siau, and Langtao Chen. 2023. Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of Information Technology Case and Application Research 25, 3 (2023), 277–304. https://doi.org/10.1080/15228053.2023.2233814 arXiv:https://doi.org/10.1080/15228053.2023.2233814
[34]
Amaya Sánchez Naoaín. 2022. Addressing the Impact of Artificial Intelligence on Journalism: The perception of experts, journalists and academics. (2022).
[35]
Savvas Petridis, Nicholas Diakopoulos, Kevin Crowston, Mark Hansen, Keren Henderson, Stan Jastrzebski, Jeffrey V Nickerson, and Lydia B Chilton. 2023. Anglekindling: Supporting journalistic angle ideation with large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–16.
[36]
Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). 2343–2361.
[37]
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The Kaldi Speech Recognition Toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding (Hilton Waikoloa Village, Big Island, Hawaii, US). IEEE Signal Processing Society. IEEE Catalog No.: CFP11SRW-USB.
[38]
Holli Semetko and Patti Valkenburg. 2000. Framing European Politics: A Content Analysis of Press and Television News. Journal of Communication 50 (06 2000), 93 – 109. https://doi.org/10.1111/j.1460-2466.2000.tb02843.x
[39]
Than Htut Soe, Frode Guribye, and Marija Slavkovik. 2021. Evaluating AI assisted subtitling. In Proceedings of the 2021 ACM International Conference on Interactive Media Experiences (Virtual Event, USA) (IMX ’21). Association for Computing Machinery, New York, NY, USA, 96–107. https://doi.org/10.1145/3452918.3458792
[40]
Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. From humans to machines: can ChatGPT-like LLMs effectively replace human annotators in NLP tasks. In Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media.
[41]
Sina Thäsler-Kordonouri and Kurt Barling. 2023. Automated Journalism in UK Local Newsrooms: Attitudes, Integration, Impact. Journalism Practice 0, 0 (2023), 1–18. https://doi.org/10.1080/17512786.2023.2184413 arXiv:https://doi.org/10.1080/17512786.2023.2184413
[42]
Maria Touri and Nelya Koteyko. 2015. Using corpus linguistic software in the extraction of news frames: towards a dynamic process of frame analysis in journalistic texts. International Journal of Social Research Methodology 18, 6 (2015), 601–616.
[43]
Baldwin Van Gorp and Tom Vercruysse. 2012. Frames and counter-frames giving meaning to dementia: A framing analysis of media content. Social science & medicine 74, 8 (2012), 1274–1281.
[44]
Dror Walter and Yotam Ophir. 2019. News frame analysis: An inductive mixed-method computational approach. Communication Methods and Measures 13, 4 (2019), 248–266.
[45]
Philipp Wicke and Marianna M Bolognesi. 2020. Framing COVID-19: How we conceptualize and discuss the pandemic on Twitter. PloS one 15, 9 (2020), e0240010.
[46]
Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Empowering LLM-based Machine Translation with Cultural Awareness. arXiv preprint arXiv:2305.14328 (2023).
[47]
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848 (2023).
Index Terms
- Human Interest or Conflict? Leveraging LLMs for Automated Framing Analysis in TV Shows
Recommendations
Framing the News: From Human Perception to Large Language Model Inferences
ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia RetrievalIdentifying the frames of news is important to understand the articles’ vision, intention, message to be conveyed, and which aspects of the news are emphasized. Framing is a widely studied concept in journalism, and has emerged as a new topic in ...
Interactive TV narratives: Opportunities, progress, and challenges
This article is motivated by the question whether television should do more than simply offer interactive services alongside (and separately from) traditional linear programs, in the context of its dominance being seriously challenged and threatened by ...
Designing attention for multi-screen TV experiences
British HCI '15: Proceedings of the 2015 British HCI ConferenceIn this Work-In-Progress we discuss our work on designing attention for multi-screen TV experiences. We first briey describe the current trends, and then progress to touch on two investigations we have conducted. In the first study we look at current ...
Comments
Information & Contributors
Information
Published In
June 2024
465 pages
ISBN:9798400705038
DOI:10.1145/3639701
Copyright � 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- H2020 Programme ICT-48-2020
Conference
IMX '24: ACM International Conference on Interactive Media Experiences
June 12 - 14, 2024
Stockholm, Sweden
Acceptance Rates
Overall Acceptance Rate 69 of 245 submissions, 28%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 102Total Downloads
- Downloads (Last 12 months)102
- Downloads (Last 6 weeks)26
Reflects downloads up to 21 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format