2. Hearing voices: Supporting online questionnaires with Text-to-Speech technology
Natalia D. Kieruj
Joris Mulder
Arnaud Wijnant
Salima Douhou
(CentERdata)
Frederick Conrad
(university of Michigan)
12/10/2018

3. Main goal of research
Can Text-to-Speech be a valuable tool for online questionnaires?
Why use Text-to-Speech?
1.5 million (8.9%) of Dutch population have trouble reading. The largest group are those with low education, a dyslectic handicap, the elderly, or people who have poor eyesight
Focus of research
Feasibility of Text-to-Speech software
Methodological implications
12/10/2018

4. Feasibility (1) Software we used: ReadSpeaker Example:
Several male & female voices & several languages
Server side speech processing of dynamic webpages
Variable reading pace and highlighting
Cross-browser / cross-device
Example:
12/10/2018

5. Feasibility (2) Some practical issues, e.g. grid vs. single questions
12/10/2018

6. Feasibility (3) Some practical issues, e.g. grid vs. single questions
12/10/2018

7. Feasibility (4) Our experiences using Text-to-Speech:
ReadSpeaker license (approx. 250 euro p/m)
Most browsers and devices are supported, but testing is needed
Extra time needed to implement
Not everything is read out in the most logical order
Completion time of questionnaire for respondent is longer
Not suitable for all respondents
12/10/2018

8. Methodological implications (1)
Protect data quality when using Text-to-Speech
Does implementing Text-to-Speech have an effect on response behavior?
Experimental design:
2 (reading disabilities vs. no reading disabilities) x 3 (text only vs. text & female computer voice vs. text & male computer voice)
text only
text & female computer voice
text & male computer voice
reading limitations
no reading limitations
12/10/2018

9. Methodological implications (1)
Focus on two main types of potential bias in online questionnaires:
1) Primacy & recency effects
Primacy effects with text (e.g. CAWI)
Recency effects with audio (e.g. CATI)
 So….. what happens to response behavior when text and audio are combined due to T2S?
2) Interviewer effect: social desirable answering
Sense of a social presence due to voice?
12/10/2018

10. Methodological implications (3)
Questionnaire:
Instructions, audio-check & practice questions
Primacy / recency
2 ranking lists of 9 items (EVS, 2008)
Randomized to prevent order effects
Social desirability
5 questions from Balanced Inventory of Desirable Responding (BIDR; Paulhus, 1989)
6 sensitive case questions (based on Coutts and Jann, 2011)
Control & evaluation questions
Did Rs leave sound on?
Did Rs listen?
Did Rs find questionnaire easier or harder with T2S?
12/10/2018

11. Results (1)
No significant effects between female voice condition and male voice condition
 Therefore, we pooled the speech conditions:
text only
text & text-to-speech
m/f (pooled)
reading limitations
no reading limitations
12/10/2018

12. Results (2) Primacy / recency effects:
No significant primacy or recency response effects
Between: text condition vs. text & voice condition
12/10/2018

13. Results (3) Social desirability:
Significant response effects on 6 sensitive case questions
Between: text condition vs. text & voice condition
Rs in Text-to-Speech condition tend to respond in a more social desirable manner
(t=-2.25, p<.05) N M SD
Only text
931
25.68
3.23
Text & speech
690
26.05
3.26
12/10/2018

14. Results (4) Did respondents find Text-to-Speech pleasant or annoying?
“What did you think of the text being read out?”
Answering scale ranging from “pleasant”(1) to “annoying”(5)
Rs in target group (potential reading disabilities) found it pleasant that the text was read out to them. (t=9.29, p<.001) N M SD
Control group
314
3.78
1.28
Target group
376
2.82
1.44
12/10/2018

15. Results (5)
Did respondents find Text-to-Speech a helpful tool for answering questions?
“Was it easier or harder to fill out questions with Text-to-speech?”
Answering scale ranging from “easier”(1) to “harder”(5)
R’s in target group (potential reading disabilities) found it easier to answer questions with text being read out to them. (t=5.40, p<.001) N M SD
Control group
314
3.3
1.03
Target group
376
2.85
1.13
12/10/2018

16. Results (6)
What do respondents think of Text-to-Speech? (comments at the end of questionnaire)
To ‘computerized’: voice emphasis & intonation not always right
Reading pace to slow
Reading order not always logical (grid)
Not always helpfull, sometimes the opposite
12/10/2018

17. Conclusions
Combining text and audio in this experiment does not result in significant primacy or recency effects
Text-to-speech software does induce an interviewer effect in this experiment: Rs did answer in a more social desirable manner
Rs who have trouble reading find it helpful, but Rs without trouble reading do not like it, and even find it annoying sometimes
12/10/2018

18. Future reseach Less controlled 
variable reading pace
answering allowed before all text is read out
highlighting text
Human voices (as opposed to computerized) (August 2013)
12/10/2018

19. Interested in doing your own research. Interested in free data. www
Interested in doing your own research? Interested in free data?