Process Evaluations and qualitative research Quantitative answers Is there a difference? A change, effective? And How much? Are there similarites? Relationships? (correlation) And How much? Qualitative describes! Answers What is it? Why is it? How does it work? And What does it mean? Qualitative approaches are typically used with process evaluations.. Quality assurance, CQI, TQM, program monitoring, needs assessments, formative evaluations, implementation evaluation and to some degree consumer satisfaction are all formative. Qualitative research typically does not use measurement, numbers and statistics,. It uses words, descriptions, artifacts and observations!

Here are the things that a qualitative researcher does: observes events and activities as they occur in the program setting. gets directly involved with the people in the program and sometimes personally experiences the processes used in the program. acquires an insiders’ point of view while maintaining an analytic perspective uses a variety of observational and interviewing techniques and social skills in a flexible –often not pre-determined way-which allows the evaluator to follow the answers to the research questions wherever they might go. produces data in the form of extensive written notes, maps, diagrams and sometimes pictures in order to provide very detailed descriptions. watches, listens and collects data uses artifacts of the program like records, treatment plans, staffing patterns, absentee rates, dropout rates, ashtray use, lounge use, patterns of wear on equipment, but only to reach conclusions not to count! uses observations and/or several forms of interviewing. reviews the collected data (in the form of written notes, videotapes, in-depth written interviews, audiotapes. in reviewing the data, begins to look for repetitive ideas, themes, common conceptions, incongruities, strengths and weaknesses. reviews the data again and begins to make certain hypotheses grounded solely in the collected data! begins to draw meanings and conclusions from the data collected. writes a narrative about those meanings and conclusions.

Most of the time, qualitative research uses a combination of observation and interviewing! Methods used in qualitative data collection I. Key informants – Can be selected purposively or randomly. A. Key informants are persons who are asked to provide critical information which will contribute to an understanding of the situation. B. An initial step in this approach involves the selection of informants C. Techniques to secure information from informants includes the following: 1. In-depth personal interviews – involves the use of a few open-ended questions. Interviewer may choose to not deviate from or add to pre-determined questioned (Structured interview) OR choose a structure in which the interviewer may add questions based upon the type of info that arises from the initial questions (Semi-structured). Usually possess the following qualities: Respondents free to answer however they choose, is usually in-depth, there is a two way flow of information, interviewer is free to re-direct based upon responses. All data is recorded (Taped or written) This is known as Thick description. It is difficult to do more than 8 to 30 of these. 2. Telephone interviews – Often survey format. Usually limited to mostly closed- ended questions and no more than one or two open-ended ?. 3. Mailed questionnaires -surveys

Here is what don Ratcliff ( – a qualitative researcher - has to say about in-depth interviews: “Good interviews have lots of open-ended questions, most of which are formed prior to the interview. I personally like questions that come out of observations better than those created out of the student's imagination. Sometimes, though, good questions emerge during an interview because of what has been said by the one interviewed. Usually I'd go with the flow and ask the emergent questions, if it's appropriate. Transcribing interviews can be done several ways. Word-for-word transcriptions are probably best, but they are laborious. If you are well funded, this can be hired out. But there is value in the researcher listening to interviews, as the researcher may be able to figure out a muffled word that a transcriptionist cannot. Also, the interviewer may learn how to better interview by listening to his or her mistakes. It is also possible to use word-for-word transcriptions of some sections and summarize others when typing up the interviews. It is also possible to listen to the tape of the interview several times in order to better discover what sections are important enough to transcribe, which sections need to be summarized, and which sections should be ignored. But keep the tape--later in the research you may find that what was not typed was indeed more important than you thought! It is even possible to code directly from the tape--there are computer programs that allow you to connect a cassette player or even a videotape player directly to the computer, so that time markers and even your transcription can be added to the audio or video data.”

II. Direct observations – this is the one that you will use for the ‘feasibility evaluation’ *Site visits – often occurs one or two times. Data recorded, mostly by several independent observers *Participant/observer – researcher attends program regularly in an effort to de- sensitize staff from observer role and also to be able to observe/record on a regular basis. Taken from Anthropology – often referred to as ethnography. *Observing in accordance with Model standards, Policy standards, Best Practice standards, proscribed protocols etc. Here elements of the program are compared with particular standards for operation. *Observing in accordance with with Literature review. Here the program is compared with what is recommended in the literature. (Same as using an expert). *Case study – a particular form of participant observer data collection in which the person seeing the client is often the main data collector, alone or in cooperation with researcher. Involves thick description may or may not use measures. Different from “SSRD” which uses measurement.

When observing, in a feasibility evaluation, you are looking specifically for two things: 1.Is the program actually there in the way it was said to be there or told to be there in section 1. 2.What is missing or problematic enough to get in the way of an actual evaluation if I conducted one?

There is a term in qualitative research “thick description”. It refers to describing not only what you see but the context in which it occurs. So thick description refers to description that is 1.Intensely detailed enough to capture the very small pieces of phenomena (each tree in the forest) 2. But broad enough to capture the larger pieces and how the small pieces all fit together (the forest itself). Thick description requires intensive taking of notes

III.Semi-structured groups – combines interviewing with direct observation and group processes. Focus Groups and/or Nominal groups All of the above usually involves some type of “content analysis” in which the data is re-“viewed” or re-“read” as the observer looks for themes and commonalities in the data. May involve the counting of particular words or “like sentences”. Usually involves creating ‘categories’ for data or ‘coding’ of data. Often use more than one observer and attempt to establish inter-rater reliability.

IV. Unobtrusive measures – Type A - involves studying secondary data sources such things as records (Charts, files etc.), treatment plans, staffing patterns, absentee rates, dropout rates etc. Often used in program monitoring and quality assurance. Unobtrusive measures – Type B - involves more creative things such as garbage, ashtray use, lounge use, patterns of wear on equipment. Referred to as unobtrusive because it is usually not noticed directly by staff. Often referred to as “non-reactive” measurement. I.e. In order to find out who was considered the most influential (informally & formally) member of a office, wear patterns on carpet were analyzed. The heaviest traffic was to the secretary of the office supervisor.