Assessing LLM-assisted genre analysis beyond the English context: The case of Italian CEO statements

Abstract

Recent studies have demonstrated the potential of using large language models (LLMs) in analyzing the rhetorical structure of specific genres. However, these investigations have primarily focused on English-language contexts. The current study extends this line of research by exploring LLMs’ capabilities in analyzing non-English texts, examining Italian CEO statements as a case study. The experimental data comprises 30 Italian CEO statements from corporate social responsibility (CSR) reports, which were subdivided into S1, S2, and S3 to be used in three iterative stages: a) prompt design; b) initial tests and prompt refinement; and c) follow-up tests. Results indicate that Claude 3.5 Sonnet outperformed GPT-4o in annotating moves in Italian texts in all tests. Moreover, the inclusion of additional examples in the prompt can help improve the performance of the LLMs, especially GPT-4o. When integrating human verification on inconsistent cases between the two models, the annotation accuracies exceeded 90% in all tests. The results highlight the potential for using LLMs to automate the creation of multilingual corpora annotated with moves, which could serve as valuable resources for genre-based writing instruction.