1. D O B ATCH AND U SER E VALUATIONS G IVE THE S AME R ESULTS ? William Hersh, Andrew Turpin, Susan Price, Benjamin Chan, Dale Kraemer, Lynetta Sacherek, Daniel Olson Hande Adıgüzel Hayrettin Erdem

2. B ATCH EXPERIMENTS VS TREC I NTERACTIVE T RACK Measuring recall and precision in the noninteractive laboratory setting Interaction is the key element of successful retrieval system use, and relevance-based measures do not capture the complete picture of user performance. The TREC Interactive Track instructs human users to tag the relevant documents for different topics. These results are used for independent relevance judgment. 2

3. C ONTRIBUTION OF THE PAPER To find whether IR approaches achieving better performance in the batch environment could translate that effectiveness to real users. 3

4. E XPERIMENT S TEPS 1. Establishment of the best weighting approach for batch searching experiments. 2. User experiments to determine if those measures give comparable results with human searchers. 3. Verification that the new TREC interactive track data gives comparable batch searching results for the chosen weighting schemes. 4

5. Finding an effective weighting scheme for experimental system using TREC 6-7 5

6. Interactive searching to assess weighting scheme with real users 6 Not statistically significant.

7. Interactive searching to assess weighting scheme with real users 7 All of the difference between the systems occurred in just one query, 414i.

8. V ERIFYING WEIGHTING SCHEME WITH TREC 8 This experiment is to verify that the improvements in batch evaluation detected with TREC-6 and TREC-7 data held with TREC-8 data. 8