AES Conference Sydney 2011 Ralph G Straton Murdoch University Evaluation Design for Differential Effects Evaluation

Evaluation Design for Differential Effects Evaluation Overview Nature of Evaluation Design Evaluation Questions Inference Issues Evaluation Design for Differential Effects Qualitative Evaluation and Differential Effects Conclusion

Evaluation Design for Differential Effects Evaluation Nature of Evaluation Design Blueprint for the Study What information to be gathered From which Sources When it is to be gathered How will it be Obtained Limits Nature and Extent of Information Obtained

Evaluation Design for Differential Effects Evaluation Evaluation Questions Does the program have an effect? and What are the effects of the program? Does the program achieve its goals? and To what extent does the program achieve its goals? Is the program better than the present program (or some other specific program)? and Is the program the best program?

Evaluation Design for Differential Effects Evaluation Evaluation Questions To what extent are program outcomes related to status on Characteristic X? Do NESB students perform as well as ESB students on the new ‘Culture Free’ maths program? Is performance among NESB students related to their culture of heritage?

Evaluation Design for Differential Effects Evaluation Inference Issues Idealised View of Evaluation Inference Inference to the Population Sampled From utos > UTOS Reality of Inference in Evaluation Inference to ‘Other’ Populations utos > *UTOS Maximising Utility of Evaluation Inferences Choose Sample to Reflect ‘Other’ Populations Observe Relevant Person, Program and Setting Characteristics

Evaluation Design for Differential Effects Evaluation Evaluation Design for Differential Effects Identify and Observe Relevant Characteristics Variables Build Program Variations into the Design Observe ad hoc Program Variations Determine and Report Diversity on Relevant Variables Determine and Report Program Interactions

Evaluation Design for Differential Effects Evaluation Qualitative Evaluation and Differential Effects Include Qualitative Descriptions of Program Variations Include Qualitative Descriptions of Program Settings