1. 1 Statistics and Image Evaluation Oleh Tretiak Medical Imaging Systems Fall, 2002

2. 2 Which Image Is Better?

3. 3 Which Image is Better?

4. 4 Overview Why measure image quality? How to measure image quality Statistical variation and probability theory Some results in probability theory Some results in statistics Experimental design Subjective quality measurement ROC theory and estimation

5. 5 Why Measure Image Quality Market studies (sell films) Market studies (sell equimpent) Test if equipment is working up to specification Measure effect of equipment on radiologists performance Measure the ability to perform diagnosis

6. 6 Types of ‘Quality’ Viewer preference –Relevant for entertainment, home viewing Technical quality –Process control (equipment maintenance) Utility –Ability to perform diagnosis, drive a remote vehicle, locate enemy weapons

7. 7 How to Measure Image Quality Viewer preference –Viewer trials Technical quality –Phantoms, resolution targets, expert viewers Utility –Viewer trials with expert viewers

8. 8 Variation and Probability Theory Sources of variation –Input: Different patients are different –Equipment: Different equipment is different, same equipment is different at different times –Subjective: Different viewers report different opinions, same viewers report different findings at different times

9. 9 Probability Theory Probability theory used almost universally Models –Independent trials –Dependency effects Same films viewed by different viewers Same patients imaged by different modalities Same patients viewed by several radiologists, multiple reading

10. 10 Example Goal: evaluate quality of television set Method: Ask viewers to report quality of image, standard viewing conditions Viewers report a quality number 1-5 –1-Terrible, 2-Bad, 3-So-so, 4-Good, 5- Excellent Evaluation: Compute average response

11. 11 Probabilistic Model Reported values are iid random variables. –Set {1, 2, 3, 4, 5} –Probabilities p(1), p(2), p(3), p(4), p(5) –Probabilities are unknown! –Example: 100 observation, computed average value is ?

12. 12 Breakout Excel visq.xls Blackboard

13. 13 Conclusions Results depend on who you ask Average result measured from a sample varies from sample to sample Prob. Theory tells us that with large samples, the average is equal to expected value Why does this matter?

14. 14 Some Results from Prob. Theory Mean (Expected value) Variance, Standard Deviation Sample mean, sample variance Law of large numbers Central limit theorem

15. 15 Some Results in Statistics Statistical problems –Estimation –Hypothesis testing –Confidence level

16. 16 Estimate of the Mean For a Gaussian (Normal) random variable with known standard deviation, ~  is the confidence level

17. 17 Practical Issues Can use for non-normal distributions (CLT) If n is large and st. dev. Is not known, use sample st. dev. Small sample, unknown st. dev. — use the Student t statistic.

18. 18 Experiment Quality 5 Quality 1

19. 19 Case A

20. 20 Case B

21. 21 More Statistics Estimate of variance – chi-squared (ki-squared) –Depends on number of observations (degrees of freedom) Estimate ratio of independent normal variances –Fisher f distribution –Most important