Thomas Schmidt, Elena Frick (IDS Mannheim): Reading, listening to and watching concordances of audiovisual interaction corpora

Join us for the RC21 Project Symposium, where invited speakers and project team members, Poster Presenters will present their work on methodology and applications of concordance analysis!

Thomas Schmidt, Elena Frick (IDS Mannheim):»Reading, listening to and watching concordances of audiovisual interaction corpora«

Abstract:

KWIC (Key Word in Context) concordancers have become a standard feature of every corpus analysis platform. Our contribution is concerned with extensions of the standard KWIC that are useful or even necessary for working with audiovisual interaction corpora. We do this from the perspective of research paradigms such as conversation analysis, which strive to integrate or supplement their established qualitative micro-analysis methods with corpus linguistic approaches.

Audiovisual interaction corpora consist of audio and/or video recordings, their transcriptions (possibly with additional annotations), and metadata about participants and the interaction situation. In a first approximation, corpus queries can be carried out on the transcript text and visualised as a KWIC with basic elements like highlighted hits, left and right context, and links to metadata. As an overview of current platforms providing access to interaction corpora (Frick/Schmidt, submitted) reveals, this is the minimum functionality that all platforms cater forin some form or other.

However, researchers working with audiovisual interaction data often require additional features that are owing both to the nature of the data themselves, and to the way they are typically analysed. Most importantly, it must be possible:

- to display the search result represented in a KWIC line in a larger transcript context. This feature is fundamental for understanding the conversational context.

- to playback the corresponding part of the audio or video recording underlying a line in the KWIC. This is vital for accessing information not represented in the transcript, e.g.prosodic cues that help interpret spoken utterances.

- to manually select or deselect individual lines of the KWIC. This is important because queries on interaction corpora are often initially over-inclusive (“fuzzy”) and consequently may contain false positives that need to be removed.

- to manually add annotations to KWICs in order to further categorise search results analytically.

- to access, in addition to metadata, supplementary materials (such as a diagram that speakers refer to) that may be crucial for interpreting an utterance.

Last but not least, multimodal analyses of video data will profit from methods for integrating visual information (e.g. still images, see Figure 1) into a KWIC in a compact manner, allowing researchers to get an idea of regularities and peculiarities of visible features of interaction in the same way that a “classical” KWIC enables the discovery of patterns in verbal behaviour.

Based on our experience in developing EXAKT1 (EXMARaLDA Analysis and Concordance Tool, Schmidt/Wörner 2014), DGD2 (Database for Spoken German, Schmidt 2014), and ZuMult3 (Framework for object-oriented architecture ofspoken corpora, Fandrych et al. 2023), our poster will provide an overview of the desiderata for KWIC concordances for research on audiovisual interaction data. It will also facilitate discussions on future developments, such as collaborative work with KWIC results, and address challenges such as how to display KWIC when hits involve multiword sequences realized by more than one speaker.

References:

Fandrych, Christian, Thomas Schmidt, Franziska Wallner and Kai Wörter (eds.) (2023): Korpora Deutsch als Fremdsprache 3(1). Themenschwerpunkt: Zugänge zu mündlichen Korpora für DaF und DaZ: Das ZuMult-Projekt. Darmstadt: KorDaF, 2023.

Frick, Elena and Thomas Schmidt (submitted): Querying spoken language data. In: Piotr Bański, Ulrich Heid and Laura Herzberg (eds.), Standards for language data and infrastructures. Digital Linguistics. De Gryuter.

Schmidt, Thomas (2014): The Database for Spoken German – DGD2. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). European Language Resources Association (ELRA), pp. 1451-1457.

Schmidt, Thomas and Kai Wörner (2014): EXMARaLDA. In: Durand, Jacques, Ulrike Gut and Gjert Kristoffersen (eds.), The Oxford Handbook of Corpus Phonology, Oxford: OUP, pp. 402-419.