Practice- Based Evaluation Research Week 4 Day 2 DIE 4564 Research Methods

Chapter Outline Topics Appropriate for Evaluation Research Formulating the Problem: Issues of Measurement Types of Evaluation Research Designs The Social Context Social Indicators Research Ethics and Evaluation Research Quick Quiz

Evaluation Research Evaluation Research – research undertaken for the purpose of determining the impact of some intervention, such as a program aimed at solving a problem.

Program Evaluation Program evaluation includes a variety of approaches for provide feedback about what is working well and what can and should be improved The evaluation approach must match the goals of the assessment After gathering evidence from a variety of sources, practical suggestions are made based on the conclusions of the assessment

Topics Appropriate for Evaluation Research Needs assessment studies – studies that aim to determine the existence and extent of problems, typically among a segment of the population.

Topics Appropriate for Evaluation Research Cost-Benefit Studies – studies that determine whether the results of a program can be justified by its expense (both financial and other).

Topics Appropriate for Evaluation Research Monitoring Studies – studies that provide a steady flow of information about something of interest, such as medical error rates or the outbreak of an epidemic.  Six Sigma – management philosophy that uses data to eliminating mistakes, waste, and re-work.  Sigma = standard deviation  Six = goal level of perfection  Based on measuring, analyzing, improving, and controlling

Topics Appropriate for Evaluation Research Program Evaluation Studies – the determination of whether an intervention is producing the intended result.  Also called Outcome Assessment Studies

Topics Appropriate for Evaluation Research Review Question  A researcher wants to determine if changing what is stocked in school vending machines will improve nutrition intake. This is an example of ______ research. A. needs-assessment B. cost-benefit C. outcomes assessment D. survey

Topics Appropriate for Evaluation Research Review Question  A researcher wants to determine if changing is what stocked in school vending machines will improve nutrition intake. This is an example of ______ research.  The correct answer is C. outcomes assessment

PICO Strategy for Practice-based clinical research Questions* P – Patient or problem I – Intervention C – Control or comparison O - Outcome or expected result

Formulating the Problem: Issues of Measurement Specifying Outcomes Measuring Experimental Contexts Specifying Interventions Specifying the Population New versus Existing Measures Operationalization Success/Failure

Formulating the Problem: Issues of Measurement Review Question  Which one of the below is not one of the measurement issues in evaluation research? A. Defining the units of analysis B. Measuring the outcome C. Measuring the stimulus or intervention D. Defining success E. Defining failure

Formulating the Problem: Issues of Measurement Review Question  Which one of the below is not one of the measurement issues in evaluation research?  The correct answer is A. Defining the units of analysis

NUTRITION SCREENING & INTERVENTION Townsville Nursing Home Project Interim Report

Goal To implement a nutrition screening and intervention program into Townsville Nursing Home… to improve the nutrition status among nursing home residents.

Project Plan Complete a literature search for validated nutrition assessment tools and nutrition intervention strategies. Complete a literature search for validated nutrition assessment tools and nutrition intervention strategies. Select and test assessment tools appropriate for The Townsville Nursing Home (TNH) resident population. Select and test assessment tools appropriate for The Townsville Nursing Home (TNH) resident population. Assess the current incidence of malnutrition among TNH residents. Assess the current incidence of malnutrition among TNH residents.

Project Plan continued With key stakeholders, develop nutrition policies and procedures. With key stakeholders, develop nutrition policies and procedures. Educate / train staff on use of the nutrition assessment tool. Educate / train staff on use of the nutrition assessment tool. Validate with staff which residents are at risk of malnutrition & are to receive intervention. Validate with staff which residents are at risk of malnutrition & are to receive intervention.

Project Plan continued Implement nutrition intervention. Implement nutrition intervention. Monitor / review delivery of intervention. Monitor / review delivery of intervention. Problem-solve any barriers to new policy & procedure. Problem-solve any barriers to new policy & procedure. Evaluate project effectiveness by repeat of nutrition assessment & resident survey. Evaluate project effectiveness by repeat of nutrition assessment & resident survey.

Interim Report Two assessment tools have been validated among elderly Australians: Two assessment tools have been validated among elderly Australians: the Mini Nutrition Assessment (MNA) and the Mini Nutrition Assessment (MNA) and the Subjective Global Assessment (SGA). the Subjective Global Assessment (SGA).

Both MNA and SGA tools were tested… The MNA, which requires Body Mass Index (height and weight), mid-arm circumference, calf circumference, and patient interviewing, was not an effective tool for the TNH population. The MNA, which requires Body Mass Index (height and weight), mid-arm circumference, calf circumference, and patient interviewing, was not an effective tool for the TNH population. For over half of the residents, only an estimated MNA score could be generated. For over half of the residents, only an estimated MNA score could be generated.

Unforseen problems… Many residents could not respond to interview questions. Many residents could not respond to interview questions. Several of the residents did not have their weight monitored due to their physical or mental condition. Several of the residents did not have their weight monitored due to their physical or mental condition. Purchase or rental of a hoist scale was identified as needed. Purchase or rental of a hoist scale was identified as needed. Though arm span (demi-span) can be used to estimate height, many residents could not place their arm into correct position for measurement. Though arm span (demi-span) can be used to estimate height, many residents could not place their arm into correct position for measurement.

Results… The SGA provided a quick and reliable method for assessment, even when weight trend was not available. The SGA provided a quick and reliable method for assessment, even when weight trend was not available. SGA may be even more effective when used by staff who have an on-going relationship with the resident. SGA may be even more effective when used by staff who have an on-going relationship with the resident.

Of 40 residents assessed… 11 residents did not appear malnourished (SGA Category A). 11 residents did not appear malnourished (SGA Category A). 13 residents appeared to have mild to moderate depletion (SGA Category B). 13 residents appeared to have mild to moderate depletion (SGA Category B). 16 residents appeared to have evidence of severe depletion (SGA Category C). 16 residents appeared to have evidence of severe depletion (SGA Category C).

72.5% of TNH residents appear to be candidates for nutritional intervention.

INTERVENTION TwoCal HN Medication Pass has been validated for use in Long-Term Care facilities. TwoCal HN Medication Pass has been validated for use in Long-Term Care facilities. 60 ml distributed with medications four times a day provides almost 2000kJ. 60 ml distributed with medications four times a day provides almost 2000kJ.

What’s Next?… Review policy & procedure. Review policy & procedure. Train staff on SGA. Train staff on SGA. Verify who we think needs intervention. Verify who we think needs intervention. Order the TwoCal HN supplement. Order the TwoCal HN supplement. Implement the MedPass intervention. Implement the MedPass intervention. Identify barriers as we trial the new procedure. Identify barriers as we trial the new procedure. Evaluate & modify as needed. Evaluate & modify as needed.

Types of Evaluation Research Design Experimental Designs

Types of Evaluation Research Design Quasi-Experimental Designs – nonrigorous inquiries somewhat resembling controlled experiments but lacking key elements such as pre- and post-testing and/or control groups.

Types of Evaluation Research Design Quasi-Experimental Designs  Time-Series Design – a research design that involves measurement made over some period.  Nonequivalent Control Groups – a control group that is similar to the experimental group but is not created by the random assignment of subjects.  Multiple Time-Series Designs – the use of more than one set of data that were collected over time, so that comparisons can be made.

Types of Evaluation Research Design Time-Series Design

Types of Evaluation Research Design Time-Series Design

Types of Evaluation Research Design Time-Series Design

Types of Evaluation Research Design Review Question  True or False: Evaluation research requires quantitative analysis of collected information.

Types of Evaluation Research Design Review Question  False: Evaluation research can use both qualitative and quantitative analyses of collected information.

The Social Context Logistical Problems – getting subjects to do what they are supposed to do.  Evaluation research occurs within the context of real life. Discuss example: Motivating people with diabetes Discuss example: Administrative control of break-room snacks

Townsville Aboriginal and Islanders Health Services Ltd 59 Gorden Street, Garbutt, Queensland 4817 PROJECT TEAM: Diane Longstreet, MPH, RD, APD-pending, Dietitian, Project Officer Marlene Griffiths, RN, Medical Clinic Nurse Coordinator Deanne Heath, PhD, Researcher providing statistical assistance Emily Marshall, RN, Senior Indigenous Nurse Paulo Morisco, MD, DTM&H, MNHAA, FRACGP, Lead physician for project Nichelle Geary, Bmed, FRACCP, MPHTM, Doctor coordinating second audit Sarah Larkin, MBBS, BMedSci, FRACGP, MPH, Doctor coordinating first audit Kathryn Panaretto, MBBS, MPH, Senior Medical Officer providing project oversight Improving diabetes care in an urban Aboriginal medical centre

Townsville Aboriginal and Islanders Health Services Ltd A large community-controlled medical centre in Townsville, North Queensland. Primary Health Care focused Currently 720 patients on the Diabetes Register Approximately 80% of TAIHS clients are of Aboriginal or Torres Strait Islander descent

FIRST AUDIT (mid-2002) Staff initiated audit due to concerns about care of diabetic clients at TAIHS Poor results found in many areas, especially: care plans foot assessments diet assessments exercise assessments Received funding for 12 month NDIP National Diabetes Improvement Program

FIRST STRATEGY - TEAM MANAGEMENT Diabetes Team met fortnightly on issues to improve diabetes management Team consisted of: Nurses Health Workers Dietitian Doctors Eye Health Coordinator Practice Manager

SECOND STRATEGY – CLARIFY GOALS To increase opportunistic care of Diabetic patients To improve the completion of Diabetic annual cycle of care To improve all audit parameters

THIRD STRATEGY – ANNUAL CARE PLAN Goals: BP,BSL,weight Every visit ACR,Fasting Cholesterol6 monthly HbA1c3 monthly Foot Assessment6 monthly Eye Check2 yearly Dietitian reviewYearly Also negotiate patient goals for exercise, diet, and smoking cessation Refer to diabetes educators and specialists as needed Patient to sign and receive copy of care plan!

INITIAL PROBLEMS IDENTIFIED Not taking advantage of walk-ins Hard for diabetics to attend lots of appointments Long waiting times, especially for specialists Not enough diabetes education (literature) Continuity of care Never enough time!!!

Prioritizing identified problems Patient IssuesEducational Issues Health Care Provider Issues Staff education Patient education Culturally appropriate Long waiting times Endocrinologist access Dietitian access Limited staff trained Patchy recall system Too busy for walk in diabetic patients Poor diabetes control Lack of money Lack of knowledge No motivation Lack of transport Cultural foods Lack of exercise

THIRD STRATEGY – EXPAND SERVICES Increased number of Diabetes clinics per week Diabetes nurse available daily Eye care clinic weekly Foot checks by all nurses Dietitian available for every diabetes clinic Walkabout Together pedometer-based lifestyle program Diabetes support group Cooking classes and grocery tours Podiatrist clinic weekly Endocrinologist clinic monthly Recall system with monthly mail-outs Provide transportation Home visits if needed Community outreach

RESOURCES DEVELOPED Outline of discussion topics for education Fact sheets for each topic Booklet – “The Story of Diabetes” Outreach pamphlet on clinic services Link with Walkabout Together Program included pedometers, walking groups, monthly newsletters…

DIABETES CLINIC FLOW SHEET Mini chart audit (every scheduled visit) Diabetic Screening (include BSL, BP, & weight) Foot check on attendance to clinic After seeing doctor, as time permits, send to dietitian, eye health, nurse educator, and complete blood tests Schedule next appointments before leaving clinic

2003 AUDIT RESULTS Cycle of Care2002 (n=108) 2003 (n=99) Care plan complete18%72% Foot assessment18%65% Eyes examined70%81% Exercise assessment21%67% Diet assessment31%67% Meds assessment47%70%

2003 AUDIT RESULTS Cycle of Care cont.2002 (n=108) 2003 (n=99) GP review 6+ times38%64% Dietitian seen3%37% Podiatrist seen5%8% Endocrinologist seen2%3%

OUTCOME MEASURES 2002 (n=108) 2003 (n=99) Mean BP140/88127/76 Mean HbA1c8.8%8.6% HbA1c <722%25% ACR < 3.543%55% Mean BMI

MORE OUTCOMES * * * * * p<0.01

CONCLUSIONS Positive responses on Patient Satisfaction Survey Measured improvement of all audit parameters Patients achieving more holistic care Health outcomes beginning to improve Positive Progress!

Continuing Improvement Continue Diabetes Team monthly meetings Maintain specialist clinics (podiatry, endocrinology, ophthalmology) Create a dual purpose podiatry office Staff training in foot care, wound care Review and revise care plan form and procedure

Focus for the future… Integrate previous effort into Continuous Improvement Project Automate audits of the computerized medical records Implement multi-disciplinary case conferences Provide training for new staff and cross- training for all Diabetes team members

The Social Context Use of Research Results  Why evaluation research results are not always put into practice:  Implications may not be presented in a way that is understandable to the non-researcher.  Results may contradict deeply held beliefs.  Researchers may have a vested interest in the results.

The Social Context Review Question  ________ refers to getting research subjects to do what they’re supposed to do.

The Social Context Review Question  Logistics refers to getting research subjects to do what they’re supposed to do.

Social Indicators Research Social indicators – measurements that reflect the quality or nature of social life. Social indicators are often monitored to determine the nature of social change in a society.

Social Indicators Research Review Question  Which of the below would not be a type of social indicators research? A. Comparing child obesity rates across states with and without schools physical fitness programs B. Comparing infant mortality rates across countries with and without nationalized health care C. Comparing injury rates before and after passage of a employee safety education law D. Comparing a student’s test grades across different classes in a semester

Social Indicators Research Review Question  Which of the below would not be a type of social indicators research?  The correct answer is D.

Ethics and Evaluation Research Sometimes the social interventions being assessed in evaluation research themselves raise ethical issues. Evaluation research may entail added pressure to produce specific results, as desired by interested parties. Fraudulent research results in an evaluation study can have severer consequences than consequences produced by other types of research.

Quick Quiz

Chapter 12 Quiz Why has evaluation research grown in popularity? A. federal requirements demanding program evaluations B. the availability of research funds to fulfill federal requirements C. both of the above D. none of the above

Chapter 12 Quiz Answer: C. Evaluation research has grown in popularity because of federal requirements demanding program evaluations, and the availability of research funds to fulfill federal requirements.

Chapter 12 Quiz _____ aim to determine the existence and extent of problems. A. Longitudinal studies B. Needs assessment studies C. Cost-benefit studies D. Monitoring studies

Chapter 12 Quiz Answer: B. Needs assessment studies aim to determine the existence and extent of problems.

Chapter 12 Quiz _____ provide a steady flow of information about something of interest. A. Longitudinal studies B. Needs assessment studies C. Cost-benefit studies D. Monitoring studies

Chapter 12 Quiz Answer: D. Monitoring studies provide a steady flow of information about something of interest.

Chapter 12 Quiz Besides making measurements relevant to the outcomes of a program, research must measure the _____. A. primary cause B. program intervention C. tertiary effect D. all of the above E. none of the above

Chapter 12 Quiz Answer: B. Besides making measurements relevant to the outcomes of a program, research must measure the program intervention.

Chapter 12 Quiz A research design that involves measurements made over some period may be defined as A. time series designs. B. quasi experiments. C. multiple time series designs.

Chapter 12 Quiz Answer: A. A research design that involves measurements made over some period may be defined as time series designs.

Chapter 12 Quiz _____ is a form of applied research. A. Evaluation B. Experimentation C. Interviewing D. Field research

Chapter 12 Quiz Answer: A. Evaluation is a form of applied research.

Chapter 12 Quiz _____ determine whether the results of a program can be justified by its expense. A. Longitudinal studies B. Needs assessment studies C. Cost-benefit studies D. Monitoring studies

Chapter 12 Quiz Answer: C. Cost-benefit studies determine whether the results of a program can be justified by its expense.

Chapter 12 Quiz A key variable for evaluation researchers is to measure the _____. A. independent variable B. dependent variable C. response variable D. none of the above

Chapter 12 Quiz Answer: C. A key variable for evaluation researchers is to measure the response variable.

Chapter 12 Quiz _____ are distinguished from “true” experiments primarily by the lack of random assignment of subjects to an experimental and a control group. A. Time series designs B. Quasi experiments C. Multiple time series designs

Chapter 12 Quiz Answer: B. Quasi experiments are distinguished from “true” experiments primarily by the lack of a random assignment of subjects to an experimental and a control group.

Chapter 12 Quiz Professor Yee wants to do an evaluation study of the effects of a patient education program on patient anxiety. He uses one wing in a hospital for the experiment and compares the results with a similar group of patients in a similar wing in another hospital. Which design would be best? A. classical B. nonequivalent control group C. time-series D. posttest-only control group design

Chapter 12 Quiz Answer: B. Professor Yee wants to do an evaluation study of the effects of a patient education program on patient anxiety. He uses one wing in a hospital for the experiment and compares the results with a similar group of patients in a similar wing in another hospital. The nonequivalent control group design would be best.

Chapter 12 Quiz Evaluation researchers encounter more logistical problems than other researchers because evaluation research A. occurs in the context of real life. B. takes longer. C. is more costly. D. has more measurement problems. E. examines more variables.

Chapter 12 Quiz Answer: A. Evaluation researchers encounter more logistical problems than other researchers because evaluation research occurs in the context of real life.