1 Development of Valid and Reliable Case Studies for Teaching, Diagnostic Reasoning, and Other Purposes Margaret Lunney, RN, PhD Professor College of Staten Island, The City University of New York

2 What Are Case Studies? Simulations of patients’ stories: Written, Video-taped, Computer-based Designed for specific purpose(s) Variety of designs: Length Content Style: Linear, branching

3 When Are Case Studies Used? Practice e.g., orientation of new nurses, discussion of complex cases Education e.g., teaching aids, testing Research e.g., measurement of accuracy, evaluation of knowledge

4 Advantages: Simulations of Patients’ Stories Standardized Represent important, usual, familiar, & challenging situations Restricts the complexity of practice User “gets involved” Reasonable cost

5 Why Is It Difficult to Develop Good Case Studies? Case studies are tools Quality= Validity & Reliability Development takes time, energy and commitment

6 Case Studies as Tools Goal Strengthen the link between knowledge and application How Operationalize complex abstract concepts and “real” patients’ stories, e.g., Hope Caregiver Stress Ineffective Self Health Management

7 Case Studies as Tools to Measure Nursing Concepts Nursing Concepts nursing diagnoses Capture key elements of a real patient situation Use principles of measurement Instrumentation-Method to measure concepts

8 Case Studies as Tools (cont.) Measurement: Definition See Waltz, Strickland & Lenz, 2005 Importance of conceptual frameworks Challenges of measurement

9 Case Studies as Tools (cont.) The dilemma & challenge: Patients’ stories are complex Overlapping variables Biases in interpretation Use of heuristics Many types of thinking Goal Reduce ambiguous, abstract ideas to concrete behavioral indicators

10 Measurement Frameworks Norm-referenced Evaluate performance relative to the performance of others Criterion-referenced Evaluate performance based on pre-determined standards

11 Case Studies as Criterion- Referenced Tools: Reliability Definition: Consistency of scoring Range of variability is reduced, use nonparametric procedures Test-Retest Parallel forms Interrater & intrarater agreement

12 Case studies as Criterion- Referenced Tools: Validity Definition: Measures what was intended, systematic error is reduced Types: Content validity Criterion-related validity Construct validity

13 Case Studies as Criterion- Referenced Tools: Tasks Precisely specify target behaviors Identify standards for target behavior Discriminate who did and did not acquire the target behavior Compare subjects’ performance to the standards

14 Case Studies as Tools (cont.) Types of written simulations Linear Branching Free branching Modified free branching Forced branching

15 Linear Technique Subjects follow same sequence One or more sections If more than one section, students receive specific instructions, and Sections are appropriate for all subjects One section does not influence other sections Each section samples inquiries or actions

16 Guidelines for Case Study Development: Linear Format 1.Identify the overall purposes:  What is the general topic? What kind of problem solving will be represented? How important are the responses, e.g., learning process, grade, research data?

17 Guidelines (cont.) 2. Specify objectives, e.g., to Measure accuracy of diagnosis Illustrate relation of cues to inferences Facilitate planning Orient new nurses Teach sequential decision making

18 Guidelines (cont.) 3.Decide the complexity: Length Number of diagnoses, interventions, and/or outcomes Amounts of high, moderate & low relevance data Types of data

19 Guidelines (cont.) 4. Obtain literature sources & nurse experts to support validity: Select authoritative sources: actual cases, literature sources Decide importance of each source Identify expert judges, e.g., may ask colleagues Use well-known experts for research tools

20 Guidelines (cont.) 5. Formulate case studies, directions, & scoring manual Directions must be explicit, comprehensive, and clearly written Scoring manual provides scores for possible answers

21 A) Create Blueprint Choose a general problem area What should this exercise teach or test? What kinds of problems need to be solved? What kinds of knowledge needs to be used? Define the objectives to be sampled Select concrete problem situation(s) Outline and diagram the exercise

22 B) Prepare Specifications Method of administration Proportion of content for each objective Style of writing case(s) Restrictions, e.g. time Describe scoring procedures

23 C) Construct Case Studies Develop pool of data/items to match objectives Review to determine content validity and appropriateness Edit, delete, change as indicated Assemble the case(s)

24 D) Set Standards or Cut Scores What types of answers are “correct” What answers/scores are acceptable or unacceptable Classification of various answers, # of points toward a grade

25 Guidelines (cont.) 6. Obtain content validity ratings from experts: Select content validity method Item-objective congruence Interrater agreement Average congruency percentage  Send with explicit instructions  Be prepared to make changes

26 Guidelines (cont.) 7. Evaluate with a pilot test: Consider demographics, e.g., Experience Knowledge  Evaluate directions, restrictions, scoring methods, interrater reliability

27 Conclusion Case studies are fun to use The effort is “worth it” New case study feature in IJNTC, send submissions to Margaret Lunney Suggested website: