CLARIN-Emo: Training Emotion Recognition Models Using Human Annotation and ChatGPT
Pages 365 - 379
Abstract
In this paper, we investigate whether it is possible to automatically annotate texts with ChatGPT or generate both artificial texts and annotations for them. We prepared three collections of texts annotated with emotions at the level of sentences and/or whole documents. CLARIN-Emo contains the opinions of real people, manually annotated by six linguists. Stockbrief-GPT consists of real human articles annotated by ChatGPT. ChatGPT-Emo is an artificial corpus created and annotated entirely by ChatGPT. We present an analysis of these corpora and the results of Transformer-based methods fine-tuned on these data. The results show that manual annotation can provide better-quality data, especially in building personalized models.
References
[1]
Acheampong FA, Wenyu C, and Nunoo-Mensah H Text-based emotion detection: advances, challenges, and opportunities Eng. Rep. 2020 2 7
[2]
Antaki, F., Touma, S., Milad, D., El-Khoury, J., Duval, R.: Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. medRxiv (2023)
[3]
Baran, J., Kocoń, J.: Linguistic knowledge application to neuro-symbolic transformers in sentiment analysis. In: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 395–402. IEEE (2022)
[4]
Bravo-Marquez, F., Frank, E., Mohammad, S.M., Pfahringer, B.: Determining word-emotion associations from tweets by multi-label classification. In: 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 536–539. IEEE (2016)
[5]
Chen, S.Y., Hsu, C.C., Kuo, C.C., Ku, L.W., et al.: EmotionLines: an emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379 (2018)
[6]
Chen, Y., Eger, S.: Transformers go for the LOLs: generating (humourous) titles from scientific abstracts end-to-end (2022)
[7]
Cohan, A., Beltagy, I., King, D., Dalvi, B., Weld, D.: Pretrained language models for sequential sentence classification. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3693–3699. Association for Computational Linguistics (2019)
[8]
Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. CoRR abs/1911.02116 (2019)
[9]
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: GoEmotions: a dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547 (2020)
[10]
Desai, S., Caragea, C., Li, J.J.: Detecting perceived emotions in hurricane disasters. arXiv preprint arXiv:2004.14299 (2020)
[11]
Ekman P Are there basic emotions? Psychol. Rev. 1992 99 3 550-553
[12]
Gao, C.A., et al.: Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv (2022)
[13]
Kazienko, P., et al.: Human-centred neural reasoning for subjective content processing: hate speech, emotions, and humor. Inf. Fusion (2023)
[14]
Kim Y, Jeong SR, and Ghani I Text opinion mining to analyze news for stock market prediction Int. J. Adv. Soft Comput. Appl. 2014 6 1 2074-8523
[15]
Kocoń J et al. Groen D, de Mulatier C, Paszynski M, Krzhizhanovskaya VV, Dongarra JJ, Sloot PMA, et al. Neuro-symbolic models for sentiment analysis Computational Science 2022 Cham Springer 667-681
[16]
Kocoń, J., et al.: ChatGPT: jack of all trades, master of none. arXiv preprint arXiv:2302.10724 (2023)
[17]
Kocoń, J., et al.: Learning personal human biases and representations for subjective tasks in natural language processing. In: 2021 IEEE International Conference on Data Mining (ICDM), pp. 1168–1173. IEEE (2021)
[18]
Kocoń J, Miłkowski P, and Kanclerz K Paszynski M, Kranzlmüller D, Krzhizhanovskaya VV, Dongarra JJ, and Sloot PMA MultiEmo: multilingual, multilevel, multidomain sentiment analysis corpus of consumer reviews Computational Science – ICCS 2021 2021 Cham Springer 297-312
[19]
Kocoń, J., Miłkowski, P., Zaśko-Zielińska, M.: Multi-level sentiment analysis of PolEmo 2.0: extended corpus of multi-domain consumer reviews. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 980–991 (2019)
[20]
Korczyński, W., Kocoń, J.: Compression methods for transformers in multidomain sentiment analysis. In: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 419–426. IEEE (2022)
[21]
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach (2019)
[22]
Maynard, D.G., Greenwood, M.A.: Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis. In: LREC 2014 Proceedings. ELRA (2014)
[23]
Miłkowski P, Gruza M, Kazienko P, Szołomicka J, Woźniak S, and Kocoń J Nguyen NT, Manolopoulos Y, Chbeir R, Kozierkiewicz A, and Trawiński B Multi-model analysis of language-agnostic sentiment classification on multiemo data Computational Collective Intelligence 2022 Cham Springer 163-175
[24]
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., Gao, J.: Deep learning based text classification: a comprehensive review (2020)
[25]
Mirzaee, H., Peymanfard, J., Moshtaghin, H.H., Zeinali, H.: ArmanEmo: a Persian dataset for text-based emotion detection. arXiv preprint arXiv:2207.11808 (2022)
[26]
Ngo, A., Candri, A., Ferdinan, T., Kocoń, J., Korczynski, W.: StudEmo: a non-aggregated review dataset for personalized emotion recognition. In: Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pp. 46–55 (2022)
[27]
Paul, M.J., et al.: Social media mining for public health monitoring and surveillance. In: Biocomputing 2016: Proceedings of the Pacific Symposium, pp. 468–479. World Scientific (2016)
[28]
Păvăloaia VD, Teodor EM, Fotache D, and Danileţ M Opinion mining on social media data: sentiment analysis of user preferences Sustainability 2019 11 16 4459
[29]
Phillips, T., Saleh, A., Glazewski, K.D., Hmelo-Silver, C.E., Mott, B., Lester, J.C.: Exploring the use of GPT-3 as a tool for evaluating text-based collaborative discourse. In: Companion Proceedings of the 12th, p. 54 (2022)
[30]
Plutchik, R.: A general psychoevolutionary theory of emotion. In: Theories of Emotion, pp. 3–33. Elsevier (1980)
[31]
Rashkin, H., Bosselut, A., Sap, M., Knight, K., Choi, Y.: Modeling Naive psychology of characters in simple commonsense stories. arXiv preprint arXiv:1805.06533 (2018)
[32]
Reimers, N., Gurevych, I.: Making monolingual sentence embeddings multilingual using knowledge distillation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (2020)
[33]
Rotsztejn, J., Hollenstein, N., Zhang, C.: ETH-DS3Lab at SemEval-2018 task 7: effectively combining recurrent and convolutional neural networks for relation classification and extraction. In: Proceedings of the 12th International Workshop on Semantic Evaluation, New Orleans, Louisiana, pp. 689–696. Association for Computational Linguistics (2018)
[34]
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019)
[35]
Sprugnoli, R., et al.: Multiemotions-it: a new dataset for opinion polarity and emotion analysis for Italian. In: Proceedings of the Seventh Italian Conference on Computational Linguistics (CLiC-it 2020), pp. 402–408. Accademia University Press (2020)
[36]
Susnjak, T.: ChatGPT: the end of online exam integrity? arXiv preprint arXiv:2212.09292 (2022)
[37]
Szołomicka, J., Kocon, J.: MultiAspectEmo: multilingual and language-agnostic aspect-based sentiment analysis. In: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 443–450. IEEE (2022)
[38]
Tabone, W., de Winter, J.: Using ChatGPT for human-computer interaction research: a primer. Manuscript Submitted for Publication (2023)
Recommendations
AUTOMATIC ANNOTATION OF AMBIGUOUS PERSONAL NAMES ON THE WEB
Personal name disambiguation is an important task in social network extraction, evaluation and integration of ontologies, information retrieval, cross-document coreference resolution and word sense disambiguation. We propose an unsupervised method to ...
Building the�ArabNER Corpus for�Arabic Named Entity Recognition Using ChatGPT and�Bard
Intelligent Information and Database SystemsAbstractThe ArabNER corpus marks a significant advancement in the field of Arabic Named Entity Recognition (NER) due to its novel methodology that combines the capabilities of ChatGPT and Bard. In the data generation phase, ChatGPT plays a pivotal role by ...
Automated annotation of microbial proteomes in SWISS-PROT
Large-scale sequencing of prokaryotic genomes demands the automation of certain annotation tasks currently manually performed in the production of the SWISS-PROT protein knowledgebase. The HAMAP project, or 'High-quality Automated and Manual Annotation ...
Comments
Information & Contributors
Information
Published In
Jul 2023
718 pages
ISBN:978-3-031-35994-1
DOI:10.1007/978-3-031-35995-8
� The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 03 July 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 21 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in