1. Browser Evaluation Test …A Trial Run Pierre Wellner & Mike Flynn, IDIAP Fribourg Nov 26, 2004 Mike Flynn, Pierre Wellner IDIAP Simon Tucker, Steve Whittaker University of Sheffield

2. Outline Reminder of BET Trial Run Results Analysis Future work

3. Reminder What is a Browser for? “Browsing a meeting recording is an attempt to find a maximum number of observations of interest in a minimum amount of time.” “Observations of Interest” –Pairs of complementary statements about the meeting –Of interest to… the participants, or to people who missed the meeting. Observers –Unlimited access –No time limit actually 4½ x meeting time (on average) Subjects –Answer as many Questions as possible –Time limit: ½ meeting time –Questions are observation pairs, without indication

4. The BET Process

5. Trial Run: Observers Needed native English speakers –University of Sheffield –Students, researchers, lecturers Meetings1 x 44 minutes Observers6 Observations294 (only 255 used)

6. Observer’s Screen Shot

7. Observations… about the observations Examples: Agnes thinks having the sofa along the whiteboard is a good idea. Agnes thinks the sofa will be in the way if under the whiteboard. Martin wants to put the coffee machine along the left wall. Martin wants to put the coffee machine along the right wall. Mainly about what was said, not done Participants names all in top ten words –Others: the, of, to, at, is, that 283/294 (83%) use participant by name Observation density…

8. Observation Density Graph

9. Trial Run: Subjects 11f + 13m = 24 total University of Sheffield Three conditions: “Guess”- no media whatsoever “Base”- same media as Observers “F 1 ”- Ferret with Brno ASR transcript + slides + speaker segmentations

10. Guess Condition Screen Shot

11. Base Condition Screen Shot

12. F1 Condition Screen Shot

13. Results: Guess Condition SubjectAnswersCorrectIncorrectScore A125514211355.7% A22201239755.9% A3135815460.0% Total61034626456.7%

14. Results: Base Condition SubjectAnswersCorrectIncorrectScore B12214863% B22517868% B3127558% B4880100% B552340% B631233% B7128466% B854180% B983537% B1022121054% B11440100% Base Total126804663.5%

15. Results: F 1 Condition SubjectAnswersCorrectIncorrectScore C12011955% C263350% C31817194% C42112957% C51811761% C6117463% C7660100% C81410471% C91211191% C1072528% F 1 Total133904367.7%

16. Details Scores by time Media time-difference Speed versus accuracy

17. Results by time, overlaid Scores by Time

18. Media time difference histogram Proximity of Answers to Questions

19. Speed versus Accuracy graph Speed versus Accuracy

20. BET scores ConditionSpeedAccuracy Guess27.756.7% Base5.763.5% F 1 6.067.7%

21. Future work AMI recording 100 hour corpus More observations More subjects –reduce confidence interval (~18% wide) Design, test & compare browser improvements