Training Evaluation- Design Issues Basic Designs –Post-test only –Pre-rest/Post-test Complex Designs –Post-test with control group –Pre-test/Post-test with control group –Time Series –Solomon 4 group

Basic Designs Post-test Only –occurs when training is provided and then a post-test is given X T2 X – training T2 – post-test evaluation

Basic Designs - Post-test Only (cont’d) Goals of evaluation: –to determine if change has taken place –to determine if a level of competence has been reached Post-test only design is sufficient if the evaluation is for the second goal –e.g. a training which is developed to equip employees with certain legal requirements

Basic Designs - Post-test Only (cont’d) Post-testing can be turned into a pre-test/post- test design by utilizing data from the following sources: –needs analysis –archival data, e.g. performance appraisal measures of quality An equivalent group could also be chosen and given the same post-test and the evaluation becomes post-test only with control group

Basic Designs - Post-test Only (cont’d) Post-test only design is problematic for assessing change because change could be caused by a number of other competing courses

Basic Designs- Pre-test/Post-test The design can be represented as: T1 X T2 T1 – Pre-test X – Training T2 – Post-test Main criticism is that it is without a control group – difficult to determine the impact of training on the changes that occur

Basic Designs- Pre-test/Post-test (cont’d) A method of dealing with the lack of a control group is known as Internal Referencing Strategy (IRS) IRS includes using relevant and non- relevant test questions in the pre- and post-test –relevant : questions that deal with training contents –non-relevant : questions with aspects not trained (these questions serve as control)

Complex Designs Two factors to consider in developing a sound evaluation design: –control group: a group of similar employees who do not receive the training –random assignment: the assignment of employees to either the control group or training group by chance

Complex Designs – Post-test Only with Control Group This design can be represented by the following Trainee GroupX T2 Control Group T2 Differences in test scores between the groups will provide evidence of the effectiveness of the training Weakness: there is no pre-test score to indicate equivalence of the groups before training Representative sampling: matching of employees on specific variables e.g. age, educational background, experience etc. may reduce the weakness

Complex Designs- Pre-test/Post- test with Control Group It is represented by the following expression: Trainee GroupT1 X T2 Control GroupT1 T2 Provides indication of equivalence between the two groups (through T1) Post-test differences will provide evidence for training effectiveness

Complex Designs- Time Series Design The expression for this design is: Trainee Group T1 T2 T3 T4 X T5 T6 T7 T8 It uses a series of measurements before and after training The design can be made more powerful by using a control group Trainee Group T1 T2 T3 T4 X T5 T6 T7 T8 Control Group T1 T2 T3 T4 T5 T6 T7 T8

Complex Designs- Multiple Baseline Design Expression: Trainee Group A T1 T2 T3 X T4 T5 T6 T7 T8 T9 T10.. Trainee Group B T1 T2 T3 T4 T5 X T6 T7 T8 T9 T10.. Trainee Group C T1 T2 T3 T4 T5 T6 T7 X T8 T9 T10.. Trainee Group D T1 T2 T3 T4 T5 T6 T7 T8 T9 X T10.. Multiple measures are taken (as in time series) but each group is trained at different times If each group improves after receiving training, it can be said that the changes are due to training

Complex Designs- Solomon 4 Group Design Expression: Group 1T1XT2 Group 2T3T4 Group 3XT5 Group 4T6 Members of the groups should be based on representative sampling If T2 > T1 and T4, and if T5 > T6, and if T5 > T3 then inference about the effects of training can be made with confidence If T6 = T1 and T3 (history and maturation are not the cause of improvement) If T2 and T5 are equal (both higher than T1) then the reactive effect of testing is not the cause

Evaluation– The Choice of Design The true impacts of training depends on the validity of evaluation results The more complex the design, the more valid the results When multiple measures are not possible, the following design may be useful: Trainee Group A T1 X T2 Trainee Group B T1 X T2 Trainee Group C T1 X T2 Trainee Group D T1 X T2