Development of a Self-Assessment Method for Patients to Evaluate Internet-based Health Information

AMIA Fall 1999Josette Jones Josette Jones, RNc, Licentiate MIS, Licentiate Nursing, Doctoral Student School of Nursing University of Wisconsin - Madison Academic Advisor: Patricia Flatley Brennan, RN, Ph.D., FAAN, FACMI

AMIA Fall 1999Josette Jones Purpose of the study Development of a Self-Assessment Method for patients to evaluate quality and relevance of health care oriented websites Validation of the Stability and Consistency the Self-Assessment Method for website evaluation the use of the tool the conceptualization

AMIA Fall 1999Josette Jones Background and Significance The Internet hosts a large number of accurate health-oriented websites with endless opportunities to inform, teach and connect patients. Health care consumers need a way to judge the quality and relevance of the information provided on the Internet.

AMIA Fall 1999Josette Jones Background and Significance Quality as perceived by the “general” users relates not only to accuracy of content but rather to presentation, perceived trust, clear credentials, and other markers that tend to give information 'weight'. Patients will accept or use a WWW page when the information is perceived as relevant to his/her situation.

AMIA Fall 1999Josette Jones Helping patients determine the quality and relevance of health information found on the Internet is a key responsibility for clinicians who want to use network technologies to promote the health of patients and provide them with clinical service.

Development of a Self- Assessment Method for Evaluation of Websites

AMIA Fall 1999Josette Jones Study Design Summary of criteria/guidelines for evaluating WebPages, published in journal articles and on-line publications Categorization of the criteria through lexical and contextual analysis

AMIA Fall 1999Josette Jones Study Design Comparison to the criteria to what the “general” user of the WWW perceives as quality and/or relevance. Conceptualization of four criteria considered as indicative of quality and relevance as perceived by the “general” user of information on the Internet.

AMIA Fall 1999Josette Jones “Quality and Relevance” Criteria Content Topicality Depth Accuracy Quantity Clarity Design Attractiveness Audience appeal Writing style Layout Graphics & multimedia Communication Readability Understandable Easily navigated Credibility Assesses author’s authority and qualifications Information currency & functional feedback

AMIA Fall 1999Josette Jones The proposed Self-Assessment Method for Evaluating Websites consists of nine unrelated questions

AMIA Fall 1999Josette Jones 1.Match of interest 2.Clear and sufficient information 3.Novel info 4.Compelling case 5.Support from graphics and pictures 6.Ease of reading and understanding 7.Up-to-date information 8.Familiarity with publisher 9.Facilitate behavior change Prompts to Patients

AMIA Fall 1999Josette Jones

AMIA Fall 1999Josette Jones

Validation of the Stability and Consistency the Self- Assessment Method for Website Evaluation

AMIA Fall 1999Josette Jones Sample 16 students from a small Midwestern university participated in the testing of the method –Female, Caucasian –Average age: 22 year –Medical/clinical knowledge: none to basic –Internet skills varying from none to good

AMIA Fall 1999Josette Jones Procedure Participants were asked to list 3 topics related to health and to search the web for information on these topics The test was repeated after 3 weeks

AMIA Fall 1999Josette Jones Identified a web site Answered each item on the questionnaire with “yes” or “no”. Testing the Use of the Self- Assessment Method

Summative scores for each answer per item on the questionnaire Time 1 (N=48) Time 2 (N=48)** YesNoUndYesNoUnd Match of interest Clear and sufficient info Novel info Compelling case Support from graphics & pictures Ease of reading & understanding Up-to-date info Familiarity with publisher Facilitate behavior change

AMIA Fall 1999Josette Jones Validating the Conceptualized Criteria Each of the four criteria were validated for “quality and relevance” using a 4- point Likert scale. 1 =not relevant 2=somewhat relevant 3=quite relevant 4=very relevant

AMIA Fall 1999Josette Jones “Quality and Relevance” Criteria Content Topicality Depth Accuracy Quantity Clarity Design Attractiveness Audience appeal Writing style Layout Graphics & multimedia Communication Readability Understandable Easily navigated Credibility Assesses author’s authority and qualifications Information currency & functional feedback

AMIA Fall 1999Josette Jones Validation of the criteria indicative of quality and relevance: modal score (frequency) Time 1 N=16 Time 2 N=16 Overall N=32 Credibility4 (63%) Communication4 (38%)4 (50%)4 (44%) Content3 (50%) Design2 (50%)3 (44%)2 (41%)

AMIA Fall 1999Josette Jones Matching Perceptions of “Quality and Relevance” to Questionnaire Each questionnaire item was validated as useful to determine the “quality and relevance” using a 4-point Likert scale. 1 =not relevant 2=somewhat relevant 3=quite relevant 4=very relevant

Perception of Quality and Relevance: Ranked by Modal Score - Frequency N=32Modal Score Frequency % 1. Match of interest Ease of reading &understanding Clear and sufficient info Up-to-date info Novel info Compelling case Familiarity with publisher Facilitate behavior change Support from graphics & pictures238

AMIA Fall 1999Josette Jones Discussion The study suggests that: Consumers evaluate quality and information against a simple set of indirect criteria Design (Q5) and facilitation of behavior change (Q9) are rated less helpful in evaluating the quality and relevance of health related websites than the other questions.

AMIA Fall 1999Josette Jones Limitations of the study Limited size of the convenience sample Demographics of participants No data available from patient sample. No elicitation of unrecognized needs No measurement of the number of steps it may take to locate the information

AMIA Fall 1999Josette Jones Acknowledgments Dr. Patricia Flatley Brennan The HeartCare team in Madison and Cleveland University of Wisconsin, Eau Claire Dr. M. Oleson - and - Gyda Bjornsdottir RN, MSN

AMIA Fall 1999Josette Jones Josette Jones, RNC, Licentiate Nursing, Licentiate MIS Academic Advisor: Patricia Flatley Brennan, RN, Ph.D., FAAN, FACMI This study was supported by the HeartCare Project funded through NIH grant R01-LM06249 funded through NIH grant R01-LM06249