The Thematic Baseline Technique as a Means of Improving the Sensitivity of Self-report Scales David Nunez & Edwin Blake Collaborative Visual Computing Laboratory Department of Computer Science University Of Cape Town

2 The CVC Lab, UCT Computer Graphics HCI & Usability Virtual Presence VR Applications VR Authoring for novices & non-programmers We are looking for European research partners!

3 Self-report scales in presence Not wildly popular but widely used Not wildly popular but widely used Many problems – objectivity, sensitivity, validity, etc.Many problems – objectivity, sensitivity, validity, etc. We have found them to have low validity We have found them to have low validity Especially in relation to realism/display quality criterion, although this is a well known effectEspecially in relation to realism/display quality criterion, although this is a well known effect

4 Scale failure: Johns et al, (2000) Tried to manipulate presence as an independent variable Tried to manipulate presence as an independent variable Two VEs – ‘high presence’ and ‘low presence’Two VEs – ‘high presence’ and ‘low presence’ Manipulation: 9 out of 17 factors listed by Witmer & Singer (1998) Manipulation: 9 out of 17 factors listed by Witmer & Singer (1998) On the W&S scale, still no difference with 30 subjects per groupOn the W&S scale, still no difference with 30 subjects per group

5 Other scales, similar results Several failures to detect difference between VE and reality Several failures to detect difference between VE and reality Mania (2001) [included five conditions]Mania (2001) [included five conditions] Usoh et al (2000)Usoh et al (2000) Construct validity of scales seems to be weak Construct validity of scales seems to be weak Either not capturing the construct (unlikely) or extremely insensitiveEither not capturing the construct (unlikely) or extremely insensitive

6 What is it like to have your presence measured? Many items require an implicit comparison of VE experiences to some familiar experiences (eg. SUS scale) Many items require an implicit comparison of VE experiences to some familiar experiences (eg. SUS scale) To what extent were there times when the VE was the reality for you?To what extent were there times when the VE was the reality for you? Please rate your sense of being in the VE on a scale of 1 to 7, where 7 represents your normal sense of being in a placePlease rate your sense of being in the VE on a scale of 1 to 7, where 7 represents your normal sense of being in a place

7 How do you compare to that? Which experience does one compare to? Which experience does one compare to? Not defined by the scale!Not defined by the scale! Extremely common, below the level of awarenessExtremely common, below the level of awareness The demand characteristics will lead subjects to provide an answer The demand characteristics will lead subjects to provide an answer Each will make the comparison differentlyEach will make the comparison differently This will increase the error variance of the scale, and reduce its sensitivityThis will increase the error variance of the scale, and reduce its sensitivity

8 Providing a baseline Problem could be alleviated by providing a baseline Problem could be alleviated by providing a baseline Something for the subjects to compare toSomething for the subjects to compare to Extremely problematic Extremely problematic How do we avoid experimenter effects?How do we avoid experimenter effects? How do we capture individual differences?How do we capture individual differences? How do we avoid confusing presence and immersion?How do we avoid confusing presence and immersion?

9 Natural Expectation Have users compare to what they expect to experience in the VE Have users compare to what they expect to experience in the VE Use schemata and scriptsUse schemata and scripts Schemata encode relationships between concepts, and link concepts together with behaviours Schemata encode relationships between concepts, and link concepts together with behaviours Great deal of communality, but allow for individual differenceGreat deal of communality, but allow for individual difference Abstraction of top-down processingAbstraction of top-down processing

10 Watch me prime you

11 Which is it?

12 Study: 2x2 Factorial ANOVA (n=103) Variables: Variables: Independent: Priming, Stimulus QualityIndependent: Priming, Stimulus Quality Dependent: SUS, Witmer & Singer’s PQDependent: SUS, Witmer & Singer’s PQ Priming manipulation Priming manipulation VE Relevant – Related in theme to the VEVE Relevant – Related in theme to the VE VE Irrelevant – Unrelated in theme to the VEVE Irrelevant – Unrelated in theme to the VE StimQual manipulation StimQual manipulation High quality – Radiosity, sound, texture-mappingHigh quality – Radiosity, sound, texture-mapping Low quality – Flat shading, no soundLow quality – Flat shading, no sound

13 Priming manipulation - Relevant

14 Priming manipulation - Irrelevant

15 Findings StimQual manipulation effective StimQual manipulation effective SUS:F(1, 99) = 9.68 p<0.003SUS:F(1, 99) = 9.68 p<0.003 PQ:F(1, 99) = 5.99 p<0.02PQ:F(1, 99) = 5.99 p<0.02 Interaction on priming x StimQual Interaction on priming x StimQual SUS:F(1, 99) = p<0.002SUS:F(1, 99) = p<0.002 PQ:F(1, 99) = 4.23 p<0.04PQ:F(1, 99) = 4.23 p<0.04

16 Means plot: SUS

17 Means plot: PQ

18 Discussion Priming magnifies differences Priming magnifies differences In relevant priming condition, H-L difference significant; in Irrelevant condition, not significantIn relevant priming condition, H-L difference significant; in Irrelevant condition, not significant Priming allowed subjects to express presence more easily in questionnaires Priming allowed subjects to express presence more easily in questionnaires Or perhaps magnified responsesOr perhaps magnified responses

19 Discussion Providing a thematic baseline could improve the sensitivity of self-report scales Providing a thematic baseline could improve the sensitivity of self-report scales Highlight differences; not sure of effects on absolute scoresHighlight differences; not sure of effects on absolute scores Suggests that higher-level semantic processes involved in presence Suggests that higher-level semantic processes involved in presence