Digital Humanities Working Group: David Kinney
For the first session of the Digital Humanities Working Group of Fall 2025, Prof. David Kinney (Philosophy) will give a presentation titled:
“Propositional Cultural Knowledge versus Culturally Fluency in LLM Generations”
Description: It is well-known that text from WEIRD (Western, Educated, Industrialized, Rich, and Democratic) sources is over-represented in the training data used to build contemporary generative language models. Thus, the outputs of language models tend to display greater awareness of the artifacts, norms, and customs of WEIRD cultures as compared to non-WEIRD cultures. In this talk, I will present the results of several experiments that test different strategies for using retrieval augmented generation (RAG) to improve LLMs' apparent understanding of a broader range of global cultures. These experiments find that while RAG strategies succeed in improving LLMs' performance on multiple-choice benchmarks meant to test cultural knowledge, they fail to improve performance on stereotype avoidance and story-telling tasks meant to measure LLMs' cultural fluency. I argue that these results demonstrate that we must distinguish between outputs that are consistent with propositional knowledge about a culture and outputs that demonstrate genuine cultural fluency when evaluating the cultural awareness of LLM outputs.
The session will take place on Friday September 26th, from 10.30-12, in the Olin Library, Room 142 (Level 1). The presentation will be followed by a Q&A. Lunch will be provided.
If you plan to attend this session, please RSVP and provide your lunch order here.
The Digital Humanities Working Group working is a space for faculty and advanced graduate students to present works-in-progress for feedback before submitting their work to an external conference, journal or grant body. We also aim to create a regular community gathering space for researchers in the digital humanities across disciplines in Arts and Sciences. Scholars interested in any of the subfields of the digital humanities, including but not limited to humanities data analytics, cultural analytics, media studies, critical digital studies, critical data studies, and history of science and technology, are welcome to attend. The group consists of monthly meetings in which one or two faculty or grad students will present a current project. The working group is a cross-disciplinary intitative sponsored by the Transdisciplinary Institute in Applied Data Sciences and the Humanities Digital Workshop, with the support of Olin Library Data Services.
If you wish to be added to the general mailing list for the DH working group, please fill out this form.
If you have any questions, or if you are interested in presenting to the group, please email Claudia Carroll (claudiac@wustl.edu).