Evaluating VR Systems

Scenario You determine that while looking around virtual worlds is natural and well supported in VR, moving about them is a difficult problem. You address this problem by developing a new locomotion technique for virtual worlds. What now? – Prove that your design is better than alternatives – What’s “better” and how do you “prove” it?

Better how? Usability – Intuitiveness, flexibility, functionality Presence/Copresence – Pi, Psi Performance – Accuracy, precision, efficiency Effectiveness – Training, therapy, distraction

Proof Compare system with alternative(s) by conducting human subjects experiments (user studies) How do we go about this “scientifically”? – Design an experiment to identify and potentially magnify differences you expect to exist between systems? – Is the experiment a good (valid one)?

Classic VR Experiment Recruit participants from the local population (population sample) Randomly divide that sample into two groups, control and intervention. Control group gets normal VR, and intervention group gets new VR Compare observations of the two groups to determine if they are significantly (non- randomly) different

Experimental Design Validity Internal Validity – Does your design properly address possible bias factors that may lead to incorrect interpretation of observations? E.g. Selection bias – Can you definitively establish a cause and effect relationship? Correlation is not causation External Validity – Generalization of results to other settings

Observations (Measures) Constructs must be operationalized as measures (metrics). Data produced by the metric proportional to hypothetical value of the construct Construct Validity – “Are you measuring what you think you’re measuring?” – Very difficult to establish (Think Pi, Psi) – Requires community to believe you – Comes after having reliability, predictive validity

How do you compare the data? Assume (Hypothesize) that all of the data comes from the same exact population. – Null Hypothesis – Alternative Hypothesis (your real belief) Find the likelihood of that seeing your particular distribution with random samples of that data If the probability of seeing your particular distribution is less than some pre-determined value, REJECT your hypothesis.

Statistical Tests Most common in VR, by far, is the Student’s T- Test (built into most spreadsheet software) – Can be used to determine the probability that a sample population has a specific mean – Can be used to determine if two samples have the same mean Compute T-Value for your case Determine probability of seeing this or greater T value in the T distribution

Example Suppose you have Metric M, which yields the following values for group 1 and group 2 – [1,7,5,6,4,1,8,5,6,3,7,5,6,4,6] – [2,1,6,1,3,2,1,7,3,1,5,3,8,4,1] Is there a significant difference between the two groups?