A Reusable Template for Evaluating Point-of-Care Information Products Patricia Weiss Friedman Andrea M. Ketchum Health Sciences Library System University of Pittsburgh

How often were specific products mentioned? (Jan-Nov 2003) Discussion of EBM Products on MEDLIB-L How many questions have there been about EBM resources in recent years?

Characteristics of Point-of-Care (POC) Products Provide access to succinct evidence-based diagnosis and treatment information supporting care of individual patients Oriented to…  primary care physician  specialist seeking information outside his/her own specialty

Characteristics of POC Products The Core Document Core document is a topic review presented in a product-specific, standardized structure. Core document is evidence-based. Core document content is subdivided for quicker access. In addition to the core document, product may incorporate other resources (journals, textbooks, drug information, news or alerts, calculators).

6 WEIGHTED SCORING scoring instrument 4 product/ attribute grid INTERVIEW product reps 3 7 ADAPT  Test products  Score using different criteria  Result: Second set of weighted scores template 5 LITERATURE REVIEW Wish List 1 IDENTIFY products of interest Steps in This Study 2 DEVELOP (structured question format) template attribute checklist

POC Products in this Study Product selection criteria  Ostensibly, all have characteristics listed above.  Product or vendor known directly or by reputation  Willing to be evaluated or tested against other products Products evaluated  Micromedex’s DISEASEDEX  InfoRetriever  Ovid’s Clineguide (since subsumed into Skolar)  MD Consult’s PDxMD (since renamed FirstConsult)

Development of Evaluation Template Topic sources  Informal brainstorming among project librarians  Brief exposure to some of the products  Comparison with known products Format  Questionnaire  Structured and hierarchical  Reusable in further or additional product evaluations

Template Topic Areas 1.Content  Scope  Methodology and core document structure  Editorial practices 2.Architecture/Navigation  Content storage and organization  User experience 3.Integration with other environments  PDAs  Local clinical information systems 4.Niche (self-perceived product strength)

Interviews Questionnaire was used in scheduled vendor interviews (3 interviews in person, 1 by phone). Group situation: 1-2 product developers or representatives were interviewed by 2-3 librarians. One vendor followed up by with additional information. Product/attribute comparison grid was created in Excel and populated as interviews were completed.

Literature Review Challenges identified  Information needs  Information-seeking behaviors  Barriers to implementation Solutions proposed  Evidence Cart (Sackett & Strauss, 1998) Remaining questions  Impact on clinical practice  Product comparison

Broad scope Help with queries and search strategies Rx recommendations Drug information Practice guidelines with automatic EBM updates Synthesis of evidence Patient education modules On demand at point of care Linked directly to relevant literature Customized for local use Flexible decision-making models Fast PDA compatible Conformity to hardware and software standards Wish List Based on Literature Review: What would the ideal product be like?

Product Comparison Scoring Instrument Scores for 6 categories (4 in original questionnaire) 1. Content 2. Audience 3. Integration 4. Architecture/Navigation 5. Retrieval (originally under Content) 6. Quality Control (originally under Architecture/Navigation) Scores weighted to favor Wish List compliance and non-proprietary standards Scoring instrument can be tailored to local preferences.  Different weights  different scoring outcomes

Scoring Scheme Attributes were listed for each category. For every product, each attribute was assigned a score.  Default: 1 if attribute present, 0 if not present  Scores weighted for attributes of particular interest 2 points for presence of each Wish List attribute 0 points for presence of an attribute if it involved a proprietary standard Points for scores in all categories were totaled.

Example Scoring Instrument for Attributes Associated with Content

Example One Scoring Outcome (Total Points)

Example One Scoring Outcome (Points by Category)

Limitations of This Work Interview data were varied and contained gaps. There was either no or limited hands-on use of products themselves. Outcome depends on scoring scheme; ours may not be optimal for other researchers or libraries. This methodological approach disregards real world constraints such as:  Product cost  IP restrictions and other implementation issues

Recap 1.Experience with familiar products can be used to generate a structured, hierarchical template for use in product evaluation. 2.Requirements for point-of-care products can be distilled from a literature review. 3.Product characteristics can be scored using a locally weighted instrument. 4.Template can be adapted and reused with additional products or for further evaluation based on product testing.