1. Development of a Self-Assessment Method for Patients to Evaluate Internet-based Health Information

2. AMIA Fall 1999Josette Jones Josette Jones, RNc, Licentiate MIS, Licentiate Nursing, Doctoral Student School of Nursing University of Wisconsin - Madison Academic Advisor: Patricia Flatley Brennan, RN, Ph.D., FAAN, FACMI

3. AMIA Fall 1999Josette Jones Purpose of the study Development of a Self-Assessment Method for patients to evaluate quality and relevance of health care oriented websites Validation of the Stability and Consistency the Self-Assessment Method for website evaluation the use of the tool the conceptualization

4. AMIA Fall 1999Josette Jones Background and Significance The Internet hosts a large number of accurate health-oriented websites with endless opportunities to inform, teach and connect patients. Health care consumers need a way to judge the quality and relevance of the information provided on the Internet.

5. AMIA Fall 1999Josette Jones Background and Significance Quality as perceived by the “general” users relates not only to accuracy of content but rather to presentation, perceived trust, clear credentials, and other markers that tend to give information 'weight'. Patients will accept or use a WWW page when the information is perceived as relevant to his/her situation.

6. AMIA Fall 1999Josette Jones Helping patients determine the quality and relevance of health information found on the Internet is a key responsibility for clinicians who want to use network technologies to promote the health of patients and provide them with clinical service.

7. Development of a Self- Assessment Method for Evaluation of Websites

8. AMIA Fall 1999Josette Jones Study Design Summary of criteria/guidelines for evaluating WebPages, published in journal articles and on-line publications Categorization of the criteria through lexical and contextual analysis

9. AMIA Fall 1999Josette Jones Study Design Comparison to the criteria to what the “general” user of the WWW perceives as quality and/or relevance. Conceptualization of four criteria considered as indicative of quality and relevance as perceived by the “general” user of information on the Internet.

10. AMIA Fall 1999Josette Jones “Quality and Relevance” Criteria Content Topicality Depth Accuracy Quantity Clarity Design Attractiveness Audience appeal Writing style Layout Graphics & multimedia Communication Readability Understandable Easily navigated Credibility Assesses author’s authority and qualifications Information currency & functional feedback

11. AMIA Fall 1999Josette Jones The proposed Self-Assessment Method for Evaluating Websites consists of nine unrelated questions

12. AMIA Fall 1999Josette Jones 1.Match of interest 2.Clear and sufficient information 3.Novel info 4.Compelling case 5.Support from graphics and pictures 6.Ease of reading and understanding 7.Up-to-date information 8.Familiarity with publisher 9.Facilitate behavior change Prompts to Patients

13. AMIA Fall 1999Josette Jones

14. AMIA Fall 1999Josette Jones

15. Validation of the Stability and Consistency the Self- Assessment Method for Website Evaluation

16. AMIA Fall 1999Josette Jones Sample 16 students from a small Midwestern university participated in the testing of the method –Female, Caucasian –Average age: 22 year –Medical/clinical knowledge: none to basic –Internet skills varying from none to good

17. AMIA Fall 1999Josette Jones Procedure Participants were asked to list 3 topics related to health and to search the web for information on these topics The test was repeated after 3 weeks

18. AMIA Fall 1999Josette Jones Identified a web site Answered each item on the questionnaire with “yes” or “no”. Testing the Use of the Self- Assessment Method

19. Summative scores for each answer per item on the questionnaire Time 1 (N=48) Time 2 (N=48)** YesNoUndYesNoUnd Match of interest41614080 Clear and sufficient info3312337110 Novel info38914251 Compelling case40613692 Support from graphics & pictures141319201411 Ease of reading & understanding39634431 Up-to-date info47104341 Familiarity with publisher1136112360 Facilitate behavior change2514924186

20. AMIA Fall 1999Josette Jones Validating the Conceptualized Criteria Each of the four criteria were validated for “quality and relevance” using a 4- point Likert scale. 1 =not relevant 2=somewhat relevant 3=quite relevant 4=very relevant

21. AMIA Fall 1999Josette Jones “Quality and Relevance” Criteria Content Topicality Depth Accuracy Quantity Clarity Design Attractiveness Audience appeal Writing style Layout Graphics & multimedia Communication Readability Understandable Easily navigated Credibility Assesses author’s authority and qualifications Information currency & functional feedback

22. AMIA Fall 1999Josette Jones Validation of the criteria indicative of quality and relevance: modal score (frequency) Time 1 N=16 Time 2 N=16 Overall N=32 Credibility4 (63%) Communication4 (38%)4 (50%)4 (44%) Content3 (50%) Design2 (50%)3 (44%)2 (41%)

23. AMIA Fall 1999Josette Jones Matching Perceptions of “Quality and Relevance” to Questionnaire Each questionnaire item was validated as useful to determine the “quality and relevance” using a 4-point Likert scale. 1 =not relevant 2=somewhat relevant 3=quite relevant 4=very relevant

24. Perception of Quality and Relevance: Ranked by Modal Score - Frequency N=32Modal Score Frequency % 1. Match of interest463 6. Ease of reading &understanding463 2. Clear and sufficient info453 7. Up-to-date info444 3. Novel info363 4. Compelling case350 8. Familiarity with publisher334 9. Facilitate behavior change241 5. Support from graphics & pictures238

25. AMIA Fall 1999Josette Jones Discussion The study suggests that: Consumers evaluate quality and information against a simple set of indirect criteria Design (Q5) and facilitation of behavior change (Q9) are rated less helpful in evaluating the quality and relevance of health related websites than the other questions.

26. AMIA Fall 1999Josette Jones Limitations of the study Limited size of the convenience sample Demographics of participants No data available from patient sample. No elicitation of unrecognized needs No measurement of the number of steps it may take to locate the information

27. AMIA Fall 1999Josette Jones Acknowledgments Dr. Patricia Flatley Brennan The HeartCare team in Madison and Cleveland University of Wisconsin, Eau Claire Dr. M. Oleson - and - Gyda Bjornsdottir RN, MSN

28. AMIA Fall 1999Josette Jones Josette Jones, RNC, Licentiate Nursing, Licentiate MIS wouterjf@uwec.edu Academic Advisor: Patricia Flatley Brennan, RN, Ph.D., FAAN, FACMI This study was supported by the HeartCare Project funded through NIH grant R01-LM06249 funded through NIH grant R01-LM06249