Student Research Week 2006 Image-based Evaluation of Video-acquired Research Skills Unmil Karadkar, Marlo Nordt Richard Furuta Cody Lee Christopher Quick Texas A&M University Center for the Study of Digital Libraries The Department of Computer Science Michael E. DeBakey Institute Veterinary Physiology & Pharmacology

Student Research Week 2006 Outline Cardiovascular Research Research Skill Testing Interfaces Design of the Evaluation Results and Analysis Future Work

Student Research Week 2006 Outline Cardiovascular Research Research Skill Testing Interfaces Design of the Evaluation Results and Analysis Future Work

Student Research Week 2006 Cardiovascular Research Related to the blood circulation system Involves live bats –Non-destructive, in-vivo studies –Pallid Bats – transparent wings Observe bat wings under a microscope Use these observations for research –Normal conditions –Modified conditions (occlusion, pressure)

Student Research Week 2006 Research Skills Arteries and Veins Order of blood vessel –Vein –Venule –Capillary Lymphatic vessels

Student Research Week 2006 Working with videos Single layout of images Montage –Facilitates comparison of consecutive images Motion cues But experienced researchers often don’t need these cues

Student Research Week 2006 Still Images for Testing Skills Validated by a pilot subject Properties of image layouts Spatial organization Temporal presentation Constrain or facilitate contextual information Comparison of images –Images of blood vessels at different locations –Images of a specific location over time

Student Research Week 2006 Outline Cardiovascular Research Research Skill Testing Interfaces Design of the Evaluation Results and Analysis Future Work

Student Research Week 2006 Montage Layout Spatial Dimensions (0) Context (user dependent) Temporal Dimension (used) Slowed down video

Student Research Week 2006 Scrolling Layout Spatial Dimensions (1) Context (4 images) Temporal Dimension (used) Smooth scrolling

Student Research Week 2006 Thumbnail Layout Spatial Dimensions (2) Context (all images in set) Temporal Dimension (unused)

Student Research Week 2006 Outline Cardiovascular Research Research Skill Testing Interfaces Design of the Evaluation Results and Analysis Future Work

Student Research Week 2006 Experimental Design 1 3 Layouts –Thumbnail –Scrolling –Montage –16 images in a set AEL Time/image3s4s2s Answersper image per set Sets223 3 Tasks –Artery/Vein recognition –Size Estimation –Lymphatic Vessel Wall Identification

Student Research Week 2006 Experimental Design 2 Balanced across subjects –Ordering of tasks (AEL, ELA, LAE) –Ordering of image layouts (TMS, MST, STM) Powerpoint for presenting layouts Feedback –Verbal –Powerpoint’s Pointer feature Validation of answers –Panel of experts –Accurate measurements

Student Research Week 2006 Subjects Trained researchers in the bat lab –About 30 subjects available world-wide 9 experts –Research experience of more than 2 semesters 6 novices –Research experience of 1 semester only Age group 18 to 35 4 female, 11 male

Student Research Week 2006 Data Collection Answers for the tasks Demographic Questionnaire –Self evaluation of expertise Task Questionnaire –For each task Most preferred layout Perceived best performance Short interview

Student Research Week 2006 Outline Cardiovascular Research Research Skill Testing Interfaces Design of the Evaluation Results and Analysis Future Work

Student Research Week 2006 N=16, x=9.75 σ 2 =9.73 No learning Constrained by knowledge Preferred layout T N=16, x=13.59 σ 2 =2.13 Learning (p=0.0004) Constrained by layout Preferred layout T Artery/Vein Recognition (p=0.005) 89.6% 91.4% Validated conventional wisdom

Student Research Week 2006 Size Estimation N=16, x=9.03 σ 2 =14.11 Preferred layout T N=16, x=4.06 σ 2 =12.51 Preferred layout T General understanding of the order of blood vessels does not translate into accurate estimation of size (p<0.002)

Student Research Week 2006 Lymphatic Vessel Wall Identification Worked best with layout M (p=0.035) Preferred layout M (p=0.03) 87% 55% Experts and Novices use different strategies for applying contextual information Worked best with layout T Preferred layout M

Student Research Week 2006 Outline Cardiovascular Research Research Skill Testing Interfaces Design of the Evaluation Results and Analysis Future Work

Student Research Week 2006 Future Work Train subjects on new interfaces –Subjects preferred familiar interfaces but performed well with the unfamiliar ones as well Investigate mechanisms for conveying size of objects within images –Subjects could not accurately estimate the size of blood vessels in spite of possessing the required knowledge Explore other image layouts

Student Research Week Center for the Study of Digital Libraries Micahel E. DeBakey Institute Further information Unmil Karadkar Marlo Nordt Richard Furuta Christopher