Advances in Human Resource Development and Management Course Code: MGT 712 Lecture 29

Recap of Lecture 28 Payoff of Training Why Do HRD Programs Fail to Add Value? HRD Process Model Effectiveness HRD Evaluation Purposes of Evaluation Models and Frameworks of Evaluation Comparing Evaluation Frameworks 2Lecture 29

Learning Objectives: Lecture 29 Data Collection for HRD Evaluation Data Collection Methods Advantages and Limitations of Various Data Collection Methods Choosing Data Collection Methods Type of Data Used/Needed Use of Self-Report Data Research Design Ethical Issues Concerning Evaluation Research 3Lecture 29

4 Data Collection for HRD Evaluation Evaluation efforts require the collection of data to provide decision makers with facts and judgments upon which they can base their decisions. Three important aspects of providing information for HRD evaluation include: – Data Collection Methods – Types of Data – Self-report Data

Data Collection Methods – Interviews – Questionnaires – Direct observation – Written tests – Simulation/Performance tests – Archival performance data Lecture 295

6 Interviews Advantages: Flexible Opportunity for clarification Depth possible Personal contact Limitations: High reactive effects High cost Face-to-face threat potential Labor intensive Trained interviewers needed

Lecture 297 Questionnaires Advantages: Low cost to administer Anonymity possible Honesty increased if anonymous Respondent sets the pace Variety of options Limitations: Possible inaccurate data On-job responding conditions not controlled Uncontrolled return rate

Lecture 298 Direct Observation Advantages: Nonthreatening Excellent way to measure behavior change Limitations: Possibly disruptive Reactive effects are possible May be unreliable Need trained observers

Lecture 299 Written Tests Advantages: Low purchase cost Easily administered Readily scored Quickly processed Wide sampling possible Limitations: May be threatening Possibly no relation to job performance Measures only cognitive learning Reliance on norms may distort individual performance

Lecture 2910 Simulation/Performance Tests Advantages: Reliable Objective Close relation to job performance Includes cognitive, psychomotor and affective domains Limitations: Time consuming Simulations often difficult to create High costs of development and use

Lecture 2911 Archival Performance Data Advantages: Reliable Objective Job-based Easy to review Minimal reactive effects Limitations: Lack of knowledge of criteria for keeping/ discarding records Information system discrepancies Indirect Need conversion to usable from Records prepared for other purposes

Lecture 2912 Choosing Data Collection Methods Reliability Consistency of results, and freedom from error and bias in a data collection method Validity Does the data collection method actually measure what we want it to measure? Practicality Does it make sense in terms of the resources used to get the data?

Lecture 2913 Type of Data Used/Needed Individual performance Systemwide performance Economic performance

Lecture 2914 Individual Performance Data Individual performance data emphasize the individual trainee’s knowledge and behaviors. – Test scores – Performance quantity, quality, and timeliness – Attendance records – Attitudes

Lecture 2915 Systemwide Performance Data System wide performance data concern the team, division, or business unit in which the HRD program was conducted and could include data concerning the entire organization. – Productivity – Scrap/rework rates – Customer satisfaction levels – On-time performance levels – Quality rates and improvement rates

Lecture 2916 Economic Data Economic data report the financial and economic performance of the organization or unit – Profits – Product liability claims – Avoidance of penalties – Market share – Competitive position – Return on investment (ROI) A complete evaluation effort is likely to include all three types of data.

Lecture 2917 Use of Self-Report Data Most commonly used data Collected for personality, attitude, and perception Problems: Mono-method bias If both reports pre-test and post-test come from the same source at the same time (after training) the conclusions may be questionable. Socially desirable responses Respondents may report what they think the researcher or boss wants to hear rather than the truth. Fearful to admit that they leaned nothing Response Shift Bias Respondents’ perspectives of their skills before training change during the training program and affect their after training assessment. Relying on self-report data only may be problematic

Lecture 2918 Research Design Research design is a plan for conducting an evaluation study. Research design is critical to HRD evaluation. It specifies in advance: – The expected results of the study – The methods of data collection to be used – How the data will be analyzed

Research Design If outcomes are measured at all, they are only collected after the training program has been completed. – Such one-shot approach may not capture real changes that have occurred as a result of training – We cannot be certain that outcomes attained were due to the training. Lecture 2919

Experimental Design: Pretest-Posttest with Control Group Pretest and Posttest Including both pretest and posttest allows the trainer to see what has changed after the training. Control group A group of employees similar to those who receive training, yet who don’t receive training. This group receives the same evaluation measures to make a comparison of their scores. Random assignment to treatment and control groups so trainees have similar characteristics This four group design is the minimum acceptable research design for training/HRD evaluation efforts It controls the effects of pretest and prior knowledge. 20Lecture 29

Experimental Design: Pretest-Posttest with Control Group Time series design Allows the trainer to observe patterns in individual performance. Sample size Number of people providing data for a training evaluation is often low than what is recommended for statistical analysis. As a bare minimum, the training and control groups each need at least thirty individuals to have even a moderate chance of obtaining statistically significant results. Lecture 2921

Ethical Issues Concerning Evaluation Research Confidentiality When confidentiality is ensured, employees would be more willing to participate Informed Consent Some evaluations are monitored so that employees know the potential risks and benefits informed consent motivates researchers to treat the participants fairly, it may improve the effectiveness of training by providing complete information. Withholding Training When results of training are used for raises or promotions, it seems unfair to place employees in control groups just for the purpose of evaluation. Use of deceptions When an investigator feels a study would yield better results if an employee did not realize they were on an evaluation study. Pressure to produce positive results When Trainers are under pressure to make sure results of the evaluation demonstrates that the training was effective 22Lecture 29

23 HRD Evaluation Steps 1.Analyze needs. 2.Determine explicit evaluation strategy. 3.Insist on specific and measurable training objectives. 4.Obtain participant reactions. 5.Develop criterion measures/instruments to measure results. 6.Plan and execute evaluation strategy.

Summary of Lecture 29 Data Collection for HRD Evaluation Data Collection Methods Advantages and Limitations of Various Data Collection Methods Choosing Data Collection Methods Type of Data Used/Needed Use of Self-Report Data Research Design Ethical Issues Concerning Evaluation Research 24Lecture 29

Reference books Human Resource Development: Foundation, Framework and Application Jon M. Werner and Randy L. DeSimone: Cengage Learning, New Delhi 25Lecture 29

Thank you! 26Lecture 29