Hungarian Academy of Sciences Comparison of the nonverbal ways of marking the discourse relation of concession and the functions of lexical search and approximation in Hungarian dialogues – Relations and correspondances between discourse-pragmatic functions (concession versus lexical search) and nonverbal features of the Hungarian DM mondjuk (’let’s say’; ’although’) Ágnes Abuczki Hungarian Academy of Sciences February 2015

Goals, outline Empirical study on a multifunctional (polysemous) Hungarian lexical item (mondjuk, meaning „let’s say” or „although, it must added that”) that operates at discourse level and marks relationshop between discourse unit as well as the attitude of the speaker --> discourse marker (henceforth: DM) 2 salient functions of the item will be described by multimodal features in order to enhance its meaning disambiguation Questions: Is there a significant relation/correspondance between the discourse- pragmatic function of a DM and Manual gesticulation Facial expression Gaze direction of the speaker Duration of the DM Pause preceding the DM?

Material Multimodal HuComTech corpus (Hungarian only) 22 eaf files (involving audio, video, multimodal pragmatic and automatic prosodic annotation): 22 informal conversations 22 interviewees Number of tokens of DMs segmented: Mondjuk (~let’s say): 208

Most salient functions of the selected DM 2 most salient functions of mondjuk ('let’s say'): LXS APPR= lexical search + approximation (46 tokens); CON= concession (41 tokens).

Most salient functions of mondjuk ('let’s say') - markers of lexical search + approximation (abbreviated as LXS; can be glossed as about, like): „gyorsan megy a motorom mondjuk 120–140-nel” (‘my bike is really fast, it can do DMmondjuk 120–140 kmphs’) (HuComTech, 017_I)   - markers of concession (abbreviated as CON; can be glossed as although, but): „szeretek a belvárosban élni mondjuk elég nagy a szmog” (‘I like living in the city centre DMmondjuk the air is polluted’) (HuComTech, 019_I)

Methods Segmentation of the selected word Tagging functions Low-level prosodic features and temporal features (durations and preceding pauses) were extracted from the segmented sound files (.wav) using Praat and Prosogram scripts. The nonverbal-visual features (gaze direction, facial expression, hand gestures) of the speaker’s behaviour were extracted from the manually-performed video annotations of the recordings and can be automatically queried using the ELAN software. The queries on the relation of each function and each nonverbal feature were run separately and were ultimately joined in contingency tables for statistical analysis.

Methods: Segmentation and annotation in ELAN

The relation of function and manual gesticulation Prior to queries, I expected to find correspondences between discourse functions and hand movements. I considered manual gesticulation: any handshape type annotated other than the default handshape of the actual speaker (most common default type: half-open-flat hands) any handshape change during uttering a DM I queried the relation of hand gesticulation and each of the salient functions of DM one by one in separate queries (with the ‘Find overlapping labels’ command), and then joined them in contingency tables for statistical analysis in SPSS 19. 0. 8

The relation of pragmatic functions and hand gestures (mondjuk – let’s say) initial observation and hypothesis: lexical search and gesticulation contrary to expectations  Significant (2(1)=12,442, p<0,01)

The relation of pragmatic functions and gaze direction (mondjuk – let’s say) typical marker of concession: averted gaze direction type during; not significant (p>0,05) typical marker of lexical search and approximation: upwards gaze direction type; not significant (p>0,05)

The relation of pragmatic functions and facial expressions (mondjuk – let’s say) Recalling affect display during mondjuk_LXS_APPR

Comparison of the durations of a DM expressing different functions Why was it analyzed? I expected that different functions are realized in different durations. Method: Queries were run by a Praat script in order to measure the duration of the individual DM tokens performing the two most salient functions, and save them in a spreadsheet file. Representation of results: box-and-whiskers plots - it shows the median and variation of duration.

Duration and pragmatic function My hypothesis about the duration of the various functions of this DM: Mondjuk (let’s say) expressing lexical search and approximation is expected to be realized longer than mondjuk expressing concession

Distribution of the duration of DMS with different functions iindependent samples t-test on mondjuk (say): significant iindependent samples t-test on ugye (is that so?): not significant

Silence annotation in Praat Silence annotation was performed following the segmentation of DMs with the aim to test the hypothesis if DMs are predominantly separated by pauses (as they are often described in the literature). The phonetic parameters set for automatic silence annotation were as follows: minimum pitch: 100 Hz (subtract mean) time step: automatic (0,01 s) silence threshold: - 45 dB minimum silent interval duration: 0,15 s minimum sounding interval duration: 0,05 s As a result, annotation segmented the recordings into sounding and silent segments:

Silence annotation in Praat The difference between the two categories has not been found significant by Pearson’s Chi-Square test (p>0,05).

Multiple layer searches in ELAN

Conclusions: prototypical sets of features of the canonical uses of mondjuk (say) performing its two different functions Lexical search, approximation Concession HAND GESTURES no yes GAZE DIRECTION upwards other than upwards FACIAL EXPRESSION recall other than recall DURATION > 250 ms < 250 ms PRECEDING PAUSE < 150 ms > 150 ms

References L. Hunyadi, Szekrényes I., Borbély A., Kiss H., Annotation of spoken syntax in relation to prosody and multimodal pragmatics. In: Proceedings of 3rd Cognitive Infocommunications Conference. Kosice: IEEE Conference Publications, 2012, 537–541. B. Fraser, “Topic orientation markers,” Journal of Pragmatics, 41, 2009, pp. 892–898. W. Chafe, “Consciousness and language,” Cognition and Pragmatics (Handbook of Pragmatics Highlights), D. Sandra, J. Östman, J. Verschueren, Eds. Amsterdam/Philadelphia: John Benjamins, 2009, pp. 135–145. Boersma P., Weenink, D, 2007. Praat: doing phonetics by computer 5.0.02. University of Amsterdam: Institute of Phonetic Sciences. http://www.praat.org http://www.physics.csbsju.edu/stats/exact_NROW_NCOLUMN_form.html This research was supported by the European Union and the State of Hungary, co-financed by the European Social Fund in the framework of TÁMOP 4.2.4. A/2-11-1-2012-0001 ‘National Excellence Program’.

Thank you for your attention.