Hearing voices: Supporting online questionnaires with Text-to-Speech technology Natalia D. Kieruj Joris Mulder Arnaud Wijnant Salima Douhou (CentERdata) Frederick Conrad (university of Michigan) 12/10/2018

Main goal of research Can Text-to-Speech be a valuable tool for online questionnaires? Why use Text-to-Speech? 1.5 million (8.9%) of Dutch population have trouble reading. The largest group are those with low education, a dyslectic handicap, the elderly, or people who have poor eyesight Focus of research Feasibility of Text-to-Speech software Methodological implications 12/10/2018

Feasibility (1) Software we used: ReadSpeaker Example: Several male & female voices & several languages Server side speech processing of dynamic webpages Variable reading pace and highlighting Cross-browser / cross-device Example: 12/10/2018

Feasibility (2) Some practical issues, e.g. grid vs. single questions 12/10/2018

Feasibility (3) Some practical issues, e.g. grid vs. single questions 12/10/2018

Feasibility (4) Our experiences using Text-to-Speech: ReadSpeaker license (approx. 250 euro p/m) Most browsers and devices are supported, but testing is needed Extra time needed to implement Not everything is read out in the most logical order Completion time of questionnaire for respondent is longer Not suitable for all respondents 12/10/2018

Methodological implications (1) Protect data quality when using Text-to-Speech Does implementing Text-to-Speech have an effect on response behavior? Experimental design: 2 (reading disabilities vs. no reading disabilities) x 3 (text only vs. text & female computer voice vs. text & male computer voice) text only text & female computer voice text & male computer voice reading limitations no reading limitations 12/10/2018

Methodological implications (1) Focus on two main types of potential bias in online questionnaires: 1) Primacy & recency effects Primacy effects with text (e.g. CAWI) Recency effects with audio (e.g. CATI)  So….. what happens to response behavior when text and audio are combined due to T2S? 2) Interviewer effect: social desirable answering Sense of a social presence due to voice? 12/10/2018

Methodological implications (3) Questionnaire: Instructions, audio-check & practice questions Primacy / recency 2 ranking lists of 9 items (EVS, 2008) Randomized to prevent order effects Social desirability 5 questions from Balanced Inventory of Desirable Responding (BIDR; Paulhus, 1989) 6 sensitive case questions (based on Coutts and Jann, 2011) Control & evaluation questions Did Rs leave sound on? Did Rs listen? Did Rs find questionnaire easier or harder with T2S? 12/10/2018

Results (1) No significant effects between female voice condition and male voice condition  Therefore, we pooled the speech conditions: text only text & text-to-speech m/f (pooled) reading limitations no reading limitations 12/10/2018

Results (2) Primacy / recency effects: No significant primacy or recency response effects Between: text condition vs. text & voice condition 12/10/2018

Results (3) Social desirability: Significant response effects on 6 sensitive case questions Between: text condition vs. text & voice condition Rs in Text-to-Speech condition tend to respond in a more social desirable manner (t=-2.25, p<.05) N M SD Only text 931 25.68 3.23 Text & speech 690 26.05 3.26 12/10/2018

Results (4) Did respondents find Text-to-Speech pleasant or annoying? “What did you think of the text being read out?” Answering scale ranging from “pleasant”(1) to “annoying”(5) Rs in target group (potential reading disabilities) found it pleasant that the text was read out to them. (t=9.29, p<.001) N M SD Control group 314 3.78 1.28 Target group 376 2.82 1.44 12/10/2018

Results (5) Did respondents find Text-to-Speech a helpful tool for answering questions? “Was it easier or harder to fill out questions with Text-to-speech?” Answering scale ranging from “easier”(1) to “harder”(5) R’s in target group (potential reading disabilities) found it easier to answer questions with text being read out to them. (t=5.40, p<.001) N M SD Control group 314 3.3 1.03 Target group 376 2.85 1.13 12/10/2018

Results (6) What do respondents think of Text-to-Speech? (comments at the end of questionnaire) To ‘computerized’: voice emphasis & intonation not always right Reading pace to slow Reading order not always logical (grid) Not always helpfull, sometimes the opposite 12/10/2018

Conclusions Combining text and audio in this experiment does not result in significant primacy or recency effects Text-to-speech software does induce an interviewer effect in this experiment: Rs did answer in a more social desirable manner Rs who have trouble reading find it helpful, but Rs without trouble reading do not like it, and even find it annoying sometimes 12/10/2018

Future reseach Less controlled  variable reading pace answering allowed before all text is read out highlighting text Human voices (as opposed to computerized) (August 2013) 12/10/2018

Interested in doing your own research. Interested in free data. www Interested in doing your own research? Interested in free data? www.lissdata.nl