1. “Do users do what they think they do?” – a comparative study of user perceived and actual information searching behaviour in the National electronic Library of Infection (NeLI) Anjana Roy 1, 2, Patty Kostkova 1, Mike Catchpole 2 & Ewart Carson 1 1 City ehealth Research Centre and Centre for Health Informatics, City University, Northamption Square, London. UK Anjana.Roy@hpa.org.uk 2 Health Protection Agency 61 Colindale Avenue Colindale London NW9 5EQ

2. 1. Introduction – importance of evaluation 2. Research questions 3. Methodology a. Website & navigation structure b. Study design 4. Results 5. Limitations of study 6. Conclusions & implementations Outline of presentation

3. eHealth professionals Challenging issues & different responsibilities Demand of regular evaluation Understanding of underlying issues of user satisfaction Content Healthcare websites - importance of evaluation Medical websites provide Evidence based information Informed decision Ease of navigation

4. Research Questions: Difference between perceived (reported) and actual (observed) searching behavior? Which navigation strategy most popular? Within each navigation strategy most popular options? Did users find answers to their queries? Was there an information seeking behavior pattern?

5. Web site used as test bed – National electronic Library of Infection The aim of NeLI is to provide a single information gateway in a portal of evidence based information. Information related to investigation, diagnosis treatment and management of infection and communicable diseases.

6. NeLI Navigation Structure Snap shot of “Browse” options available on NeLI during study period

7. Snap shot of “Search” options available on NeLI during study period NeLI Navigation Structure

8. On-line questionnaires Weblog report IP addresses of respondents were used to track their navigation. Answers were compared with the weblogs. Study design

9. 1. Was there any difference between perceived (reported) and actual (observed) searching behavior? Questionnaire Weblog Browse (%) Search (%) Combination (%) Browse 207 13 Search 040 0 Combination 0713 Questionnaires and weblog answers compared by kappa measurement of agreement. The mean of the agreement was 0.59 - moderate agreement between answers in the questionnaire and actual behavior recorded on weblog

10. Cases where the users reported and navigation behaviour did not match Users were ‘observed’ used combination of navigational options while reporting only one option. Possible explanation: The most likely explanation is web space disorientation. Recommendation: Make the navigation access points clearer and simpler to understand Users reported using a different navigation technique e.g. they reported using ‘search’ but used ‘Browse option’.

11. 2. How did the users seek their information – searching or browsing? Questionnaire WeblogBrowse (%)Search(%) Combination(%)Total (%) Browse 207 13 40 Search 040 0 Combination 07 13 20 Total 2053.3 26.7 100 According to weblogs both search and browse are equally. Respondents are more likely to report or recall searching activity rather than browsing.

12. Options Questionnaire (percent) Web logs (percent) Browse 1. A – Z listing of pages on NeLI 713 2. Top 10 Topics1320 3. Factsheets listed on NeLI13 4. Guidelines listed on NeLI 13 5. Antimicrobials Resistance00 6. List of Infectious Disease Society Websites77 Search 1. Pull down Menu6760 2. Free Text Search00 3. Within each navigation strategy most popular options?  Overall, most users reported their methods of navigation accurately.  “Browse” - no significant difference in the choice of choice of options available.  “Search” – the ‘Free Text Search’ was not used at all

13. Questionnaire : All the users reported that they were able to find the answers to their questions. One exception ………………………. 4. Were the users able to find the answers to the queries? Web logs: indicated that all respondents did access a page of their interest as reported in their questionnaire. Users understand and are able to navigate the website as expected by the developers. Information on diverse aspects of infectious diseases available on NeLI.

14. 5. Was there any information seeking behavior pattern typical of certain users? 80% of the respondents visited other pages of interest after submitting questionnaire. Web logs: 93% of those who visited these other sites used “browse”. 27% used “Top 10 Topics”. Recommendations: Need to identify “Top 25 Topics” in infectious diseases.

15. Limitations of the study External web site when accessed from cannot be tracked using the available web logs. Questionnaire has excluded “how users would surf NeLI”. Sample size limited.

16. According to web logs both search and browse are used equally. Respondents are more likely to report or recall searching activity rather than browsing. Moderate agreement between answers in the questionnaire and actual behavior recorded on web log. Users understand and are able to navigate the website retrieve information on ‘question of interest’. When surfing though NeLI their preference was to browse, using “Top 10 Topics. Conclusions

17. Navigation access points clearer and simpler to understanding NeLI now has a list of 30 Infectious Diseases as an independent navigation item. Implementation of recommendations to actions: