1. Measuring User Satisfaction in Virtual Environment Maciej A. Orzechowski Design System and Urban Planning Group @ TU/e Workshop Mass Customisation 26.06.2003

2. Plan Introduction VR System (brief description) Belief Networks (introduction) Results of the experiment (Benchmark) MuseV3 in action – Live Demo

3. The user is asked to modify that design according to his/her needs and desires. General Idea of Measuring User’s Preferences The Virtual Environment (VE) is used to present an architectural design to a user. Behind that visual system there is a statistical model to estimate and predict respondent’s preferences based on applied modifications.

4. MuseV – VR System MuseV3 – a virtual reality (VR) application with functionality of a simple CAD system for non-designers. Two categories of modifications: Structural modifications (change of layout) Textural modifications (change of visual impression)

5. Structural Modifications The most important from the point of view of estimation of user’s preferences. Change of internal and external layout Direct impact on overall costs Expressed in simple and direct commands: create/resize/divide space; insert openings

6. Textural Modifications Secondary modifications (visual impact), mainly used to check proportions, dimensions (inserting furniture) and to decorate (applying finishes). Not included in the preference model No influence on costs

7. MuseV3 in Desktop CAVE

9. Belief Network Searching for new, flexible method to access user’s preferences.Criteria: Interaction with the model during the time of preferences estimation Possibility to find weak points (where the knowledge about preferences is the worst) Improve data collection by direct feedback Incremental learning

10. Short explanation of BN What it is? Belief network (BN) also known as a Bayesian network or probabilistic causal network BN captures believed relations (which may be uncertain, stochastic, or imprecise) between a set of variables which are relevant to some problem (e.g. coefficients and choices). How does it work? After the belief network is constructed, it may be applied to a particular case. For each variable you know the value of, you enter that value into its node as a finding (also known as “evidence”). Then Netica does probabilistic inference to find beliefs for all the other variables. Incremental learning. After the beliefs are found (post priori) MuseV updates the network, so they become a’ priori for the next respondent.

11. Step 0Step 1Step 5Step 15Step 64

12. BN - Model In our proposal the network (model) is learning  while a user is modifying a design! To improve the quality of collected data and the knowledge about design attributes, the system, (based on beliefs), can post a question to user.

13. Experiment & Results

14. Experiment Types There are in total four experiment types (FMVR, OEVR, MECA, VECA). Two in each of two groups (VR and CA). Each respondent had to complete two random tasks (one from each group), however each combination of tasks should be presented approximately equal number of times.

15. Experiment Types – cont. VR ExperimentCA Experiment TypeFree ModificationPreset OptionsMultimedia Presentation Verbal Description Software (Mean of presentation) MuseV3 FMMuseV3 OEMuseV3 SCWeb Pages Collection Method Interaction with 3D environment Questionnaire TaskModification of architectural design Respond to pre designed options Choice from between three design alternatives Interactivity with 3D model Restrained to design constrains Finishes and furniture Walk ThroughN/a Feedback from the system yes none Estimation method Belief Network MNL Model

16. Results

17. Respondents The truth about the respondents: We sent 1,600 letters in total !!!! The preparations to send those letters took 2,5 days for two people (Vincent and Maciek) Within 2 weeks we received 96 positive conformations. At the end of the experiment we end up with solid number of 64 respondents that have completed the both appointed to them tasks! 5 of the 64 respondents would not buy the house that they have designed. 4 respondents did not completed second task (as the design was not relevant to them) 2 respondents did not started the experiment for the same reason!

18. The most preferred system

19. The difficultness and pleasure of the tasks.

21. External validity Real Life Data – Overall (CA, BN) G-O-F of REAL LIFE (RL) PREDICTION (Rho 2 calculation based on log likelihood) Calculations based on BETAS GOF (CA) = -0.0384 GOF (BN) = 0.1128 External validity Real Life Data – BN (FMVR, OEVR) G-O-F of REAL LIFE (RL) PREDICTION (Rho 2 calculation based on log likelihood) Calculations based on BETAS GOF (FMVR) = 0.13096 GOF (OEVR) = 0.01979

22. External validity Real Data – BN based on probability distribution of each option Option Type Lounge Ext. First Floor Ext. Garage Ext. Extra Kitchen BedroomsDormer Window Real life situation0.867 0.6670.067 0.133 Belief Network includes both subtypes (all respondents) 0.810.350.710.320.2080.519 Subtype: Free Modification0.830.3340.7810.4910.1680.391 Subtype: Preset Options0.810.3940.6330.1560.2440.666 The table illustrates ratio (percentage) of choosing certain design option. In case of real life - based on numbers of subjects buying certain option. Ri = Ni / N, where Ri – ratio for option i, Ni – number of subjects choosing option i, N – all subjects In case of BN based on beliefs read from the network.

23. Summary The majority of the respondents prefer the VR environment to the traditional. Respondents highly valued the freedom in modifying the architectural design. Due to learning and understanding the software - VR is slightly difficult. The traditional method was find as the most difficult (due to problems related with imagining the description of the house)

24. Summary Direct observation of respondent's engagement (created designs and the time spent on the process) into the VR - indicates that people prefer to work with 3D models rather then with textual description. The possibility of experiencing with the not existing house reinsure users' decision, raise questions and provokes discussions. The numerical analyses showed that working with virtual reality helps respondents to understand the design and improve their decision consistency.

25. MuseV3 in Action ! DEMO