1. How to remove an out layer tester Lucjan Janowski Faculty of Electrical Engineering, Automatics, Computer Science and Electronics Department of Telecommunications

2. Agenda Can a tester be an out layer? The detecting philosophy Latent variables Rasch model WinSteps The final decision Conclusion 2008 I 05-072

3. Can a tester be an out layer? 2008 I 05-073

4. What would we like to model? Why do we use testers? A tester represents human perception that is difficult to model People are different and so are our users/clients. Our goal is to take such difference into account Some of us are critical and others are uncritical A tester can be tired or not focused enough and therefore his/her answer can be random 2008 I 05-074

5. A tired tester problem A user can be tired too. Should we remove all tired testers? Can a tester score randomly? What are the consequences? Note that detecting that a tester scores a picture differently than the average score does not mean that it is a random tester We have to be very careful with testers removal since our goal is to build a model of the average user not the proper user 2008 I 05-075

6. Why are some scores different? Different effects can affect tester’s judgement differently (e.g. motion intensity, color, etc.) Testers have different experience (e.g. watching mainly youtube or films on a DVD set) Each of us is more or less critic to anything that he/she judges The words describing the opinion scale can be understood differently (in Poland OK is good in England OK is fair) 2008 I 05-076

7. What can we do? We have to detect random scores A tester that scores randomly often should be removed from the model building An answer that differs from the average score is not necessarily a random one therefore we have to consider the average score but corrected by a tester individualism We need a mathematic model of a user behavior that takes into account those properties 2008 I 05-077

8. Latent variable OS This is what a tester sees Any distortion that influences QoE 2008 I 05-078

9. Latent variable OS Latent variable This is what a tester sees Any distortion that influences QoE 2008 I 05-079

10. Latent variable manifestation 2008 I 05-0710 54321 54321 54 3 21 54321

11. An example 2008 I 05-0711 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

12. Non extreme values testers 2008 I 05-0712 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

13. Wide range for 10 and 1 2008 I 05-0713 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

14. Critical tester 2008 I 05-0714 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

15. Are the answers random? 2008 I 05-0715 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

16. Rasch model We assume that a latent variable is the variable that is really scored by testers We assume that the opinion score probability is a logit function of the model parameters The function has parameters describing: –a tester “criticism” factor –a film/picture/… quality –an average threshold value for particular score 2008 I 05-0716

17. Rasch model equation n the tester number i the object number (what is scored) x the opinion score value (1-5, 0-10, …) 2008 I 05-0717

18. 2008 I 05-0718 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

19. 2008 I 05-0719 Tester ID Video ID (increasing distortion) 012345678910 147 109 74254211 148 109854313211 149 810927431101 150 99956532522 151 87876652532 152 109787433211 153 36433333321

20. Rasch model We assume that Rasch model is correct and the data that do not fit this model are incorrect [sic] Note that without any assumption we are not able to detect randomly scoring testers 2008 I 05-0720 Data Model values Observed values

21. OMS (Outfit Mean Square) Knowing the model probability and the user answer we can estimate how far is a tester from the model A tester’s accuracy or quality is based on the OMS (Outfit Mean Square) Rasch model can be computed by WinSteps software (http://www.winsteps.com/)http://www.winsteps.com/ The OMS can be interpreted on the basis of heuristically obtained ranges 2008 I 05-0721

22. Results interpretation 2008 I 05-0722 A tes ter is not rel ev ant an d he/ sh e sh oul d be re mo ve d 2<OMS We should be suspicious 1.5<OMS<2 Correct tester 0.5<OMS<1.5 A tester fits the model too well OMS<0.5

23. An example results 2008 I 05-0723 Tester ID Video ID (increasing distortion) OMS 012345678910 147 109 74254211 1.78 148 109854313211 1.23 149 810927431101 2.81 150 99956532522 0.90 151 87876652532 0.76 152 109787433211 1.36 153 36433333321 0.67

24. Rasch model disadvantages It is more accurate for more data. It is difficult to have lots of results since the tests are expensive Not all type of correct testers’ behavior can be modeled The algorithms are not implemented in Matlab therefore it is difficult to implement it in an automatic analysis made in Matlab 2008 I 05-0724

25. Conclusion A tester’s answers make it possible to model human perception but not all his/her answers are correct Out layers should be removed Rasch model helps to detect not relevant testers The final decision should be checked since not all correct behaviors can be modeled by Rasch model 2008 I 05-0725

26. 2008 I 05-0726