Russell & Jamieson chapter Evaluation Steps 15

Evaluation Steps Step 1: Preparing an Evaluation Proposal Step 2: Designing the Study Step 3: Selecting a Sample Step 4: Collecting Information Step 5: Interpreting the Information Gathered Step 6: Preparing the Final Report Step 7: Putting the Findings Into Action

Preparing the Proposal Conduct a literature review. –Previous literature related to your evaluation study provides a context of how your study will fill the gaps in what is understood. –The findings will add to what we know. Explain the study design and methods. –Detail procedures for gaining evaluation information. –Describe intentions regarding research design, the sample, instrumentation, and data analysis. (continued)

Preparing the Proposal (continued) Develop a monetary budget. –Direct costs of carrying out the study Printing Postage Video recorder –Indirect costs of carrying out the study Use of office space Use of computer (continued)

Preparing the Proposal (continued) Plan the dissemination of findings. –How are the results and recommendations to be communicated and implemented? –How will what is learned be applied? –What changes will be made and when? –Who will make the changes?

Designing the Study Following are six evaluative research designs: Control group pretest−posttest One-group pretest−posttest Survey Ethnographic study Case study Content analysis

Control Group Pretest−Posttest This design uses two samples, one referred to as the experimental group and the other as the control group. The experimental sample participates in the program. A comparison is made between pretest and posttest data to determine if the program was effective.

One-Group Pretest−Posttest One sample receives a pretest before experiencing the program and a posttest after concluding the program. Any changes would be the result of the program.

Survey Respondents are interviewed or questioned to determine their attitudes, preferences, and self-reported behaviors. This design require the evaluator to estimate from the participants’ perspectives. Surveys can be conducted in person, by mail, by , by phone, or face to face. Response rate will affect sample bias.

Ethnographic Study The evaluator attempts to obtain as complete an understanding as possible. Ethnographic study is a discovery and descriptive process (Miles & Huberman, 1994). The idea is to uncover what is typical, not to explain or justify.

Case Study A case study involves focusing on one participant and determining how and why the program is working, or not working, for this person. Through the study of a unique individual, insights are gained that may suggest ways to help others.

Content Analysis A content analysis is a special kind of case study. The contents of a written, visual, or recorded document are analyzed. –Meeting minutes, long-range plans, newspapers, speeches

Selecting a Sample A sample is a subset of everyone of interest to you. The sample must be representative. –The sample selected should be a good “likeness” of the total group.

Sampling Strategies Probability sampling strategies –Simple random sampling All members have an equal chance of being selected as part of the sample. –Stratified random sampling The overall population is divided into several subgroups according to common characteristics, and then random samples are selected from the subgroups. –Systematic random sampling Every nth person is selected. (continued)

Sampling Strategies (continued) Nonprobability sampling strategies –Purposive sampling Evaluators purposely seek out respondents they believe, based on judgment or intuition, will answer questions the best. –Snowball sampling This technique involves identifying the next respondent in the evaluation based on the recommendation of the previous respondent.

Collecting Information Questionnaires and structured interviews –Questionnaires are distributed to individuals by way of mail or preformed groups. –Questionnaires become structured interviews when they are conducted face to face or over the phone. Unstructured interviews –This approach is more flexible and open ended. –Questions may not be in the original plan. (continued)

Collecting Information (continued) Participant observation –Watch what participants do. –The observer acts as naturally as possible, seeking to become a full member of the group. Archival sources –Continuous documents are public records and mass media archives. –Episodic records are more discontinuous and private. Nonreactive measures –These are generally used in tandem with other data collection. –These methods do not alter the natural actions of people.

Interpreting the Information Gathered This step involves ordering and analyzing information collected. Quantitative data is in the form of numbers. –Descriptive statistics numerically describe data. Frequency distributions Measure of central tendency Measures of variability Measures of relationship (continued)

Interpreting the Information Gathered (continued) Qualitative data analysis is necessary for data in the form of words. –Preparing notes –Focusing the data –Displaying the data –Drawing conclusions

Preparing the Final Report The evaluation report communicates what has been done to answer the evaluation question. It summarizes the evaluation purpose, design, procedures, and findings. Based on these findings, conclusions are developed and recommendations for changes are presented. (continued)

Preparing the Final Report (continued) The evaluation report should contain the following: –Orientation to the evaluation issue, question, or problem –Background information –Methods and procedures –Results and analyses –Conclusions –Recommendations for professional practice

Putting the Findings Into Action Put the recommendations in the final report into action. The recommendations instruct both policy and practice decisions for future programs. Evaluation is not finished until findings are acted on.