Height Weight I.Q. Income Body mass index Age Testosterone Highest grade #years college Amount of hair Measures of Self-esteem Acceptance of Others Narcissism Traditional sex Roles Satisfaction with life Qualitative measures with words Defining what makes a difference Qualitatively and quantitatively Observe behavior Observe environ Look at documents (diploma etc) Observe appearance Interview spouse Person, family Key informants Survey friends Go thru trash Count Big Mac Wrappers in car Look, listen and Describe, describe describe Write, write, write Analyze, analyze, analyze

QUALITATIVE MEASURING BY USING WORDS (RIGHT HAND COLUMN) REQUIRES DIFFERENT SKILLS AND METHODS THAN USING NUMBERS

Process Evaluations and qualitative research Quantitative answers Is there a difference? A change, effective? And How much? Are there similarites? Relationships? (correlation) And How much? Qualitative describes! Answers What is it? Why is it? How does it work? And What does it mean? Qualitative approaches are typically used with process evaluations.. Quality assurance, CQI, TQM, program monitoring, needs assessments, formative evaluations, implementation evaluation and to some degree consumer satisfaction are all formative. Qualitative research typically does not use measurement, numbers and statistics,. It uses words, descriptions, artifacts and observations! PROCESS EVAL = QUALITATIVE = IMPROVEMENT OF PROGRAM OUTCOME EVAL = QUANTITATIVE = JUDGMENT ABOUT PROGRAM WORTH

Here are the things that a qualitative researcher does: observes events and activities as they occur in the program setting. gets directly involved with the people in the program and sometimes personally experiences the processes used in the program. acquires an insiders’ point of view while maintaining an analytic perspective uses a variety of observational and interviewing techniques and social skills in a flexible –often not pre-determined way-which allows the evaluator to follow the answers to the research questions wherever they might go. produces data in the form of extensive written notes, maps, diagrams and sometimes pictures in order to provide very detailed descriptions. watches, listens and collects data uses artifacts of the program like records, treatment plans, staffing patterns, absentee rates, dropout rates, ashtray use, lounge use, patterns of wear on equipment, but only to reach conclusions not to count! uses observations and/or several forms of interviewing. reviews the collected data (in the form of written notes, videotapes, in-depth written interviews, audiotapes. in reviewing the data, begins to look for repetitive ideas, themes, common conceptions, incongruities, strengths and weaknesses. reviews the data again and begins to make certain hypotheses grounded solely in the collected data! begins to draw meanings and conclusions from the data collected. writes a narrative about those meanings and conclusions.

ALTHOUGH CLEAR OBJECTIVES SHOULD BE A PART OF ANY PROGRAM, THEY ARE NOT AS NECESSARY TO DO A PROCESS EVALUATION. HOWEVER YOU STILL NEED TO HAVE SOME SENSE OF A. WHAT YOU ARE MEASURING AND B. HOW YOU WOULD MEASURE IT IN A PROCESS EVALUATION. REALLY WHAT WE ARE TALKING ABOUT IS HOW TO OPERATIONALIZE THE RESEARCH QUESTION.

EXAMPLE OF A (SOMEWHAT VAGUE) PROCESS EVAL. RESEARCH QUESTION ARE THE CLIENTS ACTIVELY ENGAGED IN THE DAY PROGRAM? SO NOW YOU ASK YOURSEL F THESE QUESTIONS: 1. WHAT DOES ACTIVELY ENGAGED LOOK LIKE? 2. HOW DO I OPERATIONALIZE IT? (WHAT DO PEOPLE WHO ARE ACTIVELY ENGAGED DO?) 3. WHAT DO I LOOK FOR? 4.HOW DO I MEASURE IT USING WORDS?

POSSIBLE OPERATIONAL DEFINITIONS: 1. RATE OF DAILY ATTENDENCE, ABSENTEEISM AND TARDINESS BASED ON AVAILABLE CLIENT RECORDS 2. OBSERVATIONS BY EVALUATOR OF VERBAL INTERACTIONS AMONG CLIENTS AND STAFF DURING ACTIVITIES. 3. OBSERVATIONS BY EVALUATOR OF CLIENTS FOLLOWING DIRECTIONS AND COMPLETING TASKS IN ACTIVITIES. 4. OBSERVATIONS BY EVALUATOR OF CLIENT-GENERATED QUESTIONS AND STATEMENTS THAT ARE RELEVANT TO THE ACTIVITY AT HAND. 5. OBSERVATIONS BY EVALUATOR OF NUMBER OF CLIENTS WHO LEAVE ACTIVITIES BEFORE THEY ARE OVER OR GO OUT FOR UNSCHEDULED CIGARETTE BREAKS. 6. IN-DEPTH INTERVIEWS WITH 3 RANDOMLY SELECTED CLIENTS WHICH ASK THE QUESTIONS a. HOW IMPORTANT IS THE PROGRAM IN YOUR LIFE? b. WHY OR WHY NOT? c. WHAT DO YOU LIKE ABOUT THE PROGRAM? DON’T LIKE? THE EVALUATOR WILL LOOK FOR EVIDENCE OF IMPORTANCE AND RELEVANCE TO CLIENTS LIFE, EVIDENCE OF ENJOYMENT WHEN DISCUSSING THE PROGRAM AND THE NUMBER OF POSITIVE STATEMENTS ABOUT THE STAFF AND PROGRAM. 7. A BRIEF SURVEY OF ALL STAFF ASKING THEM TO RATE 1. OVERALL LELVEL OF CLIENT PARTICIPATION BASED ON A FIVE POINT LIKERT SCALE. 8. NUMBER OF DOCUMENTED CLIENT OUTBURSTS, NON-COMPLIANCE OR NON-PARTICIPATION, BASED ON A RANDOM AUDIT OF CLIENT CHARTS FOR ONE WEEK

NOTICE THAT BY ANSWERING THOSE FOUR QUESTIONS, YOU CREATE YOUR 1.SOURCES OF DATA 2.METHODS OF DATA COLLECTION 3.MEASURES OF “ACTIVELY ENGAGED”. 4.Coding categories (more later)

IN QUALITATIVE RESEARCH, IT IS IMPORTANT TO MAKE SURE YOU UNDERSTAND WHAT ALL OF THE TERMS IN THE RESEARCH QUESTION LOOK LIKE!!! COMMONLY CONFUSED TERMS –PARTICIPATION –IMPROVEMENT –ADEQUATELY PREPARED –HOW CAN WE REACH CLIENTS –DO CLIENTS FEEL HELPED –ARE STAFF CARRYING OUT THE PROGRAM AS DESIGNED –ARE CLIENTS AND/OR STAFF “BUYING INTO” OR “INVESTED” IN THE PROGRAM –WHAT ARE SOME OF YOURS? IN QUALITATITIVE RESEARCH, YOU MUST BEGIN TO THINK ABOUT WHAT THESE MEAN AND WHAT TO LOOK FOR