3-1 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e C HAPTER 3 TAEDES505A Evaluate a training program

3-2 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Plan and prepare evaluation For an evaluation undertaken to be of any real value it must be properly planned: you need to be clear about the purpose of the evaluation and what criteria will be used to determine outcomes.

3-3 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Aim and scope Aim Examine the effectiveness of the training Is there a demand for this qualification? Will training ensure an employment outcome for the learner? Return on investment Increase in productivity Scope Examine the whole of the organisation Examine only specific parts of the organisation Examine only specific qualifications Examine operational issues

3-4 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Information sources Analysis of work performance/observation Analysis of workplace productivity Feedback from employers, students and trainers Secondary research from government, ISBs or other industry bodies Organisational documents: policies and procedures Legislative documents

3-5 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Who are the stakeholders? Employers Training graduates Workplace supervisors Trainers and assessors

3-6 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e The project plan Roles and responsibilities of review team members Constraints on the project Project risk assessment Ground rules for the project

3-7 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Collect and collate data With the project plan prepared and task roles and responsibilities allocated, the review process can begin. In the initial stages this means gathering all relevant documents and information and collating the results for evaluation.

3-8 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Evaluation instruments Surveys Questionnaires Direct examination of outcomes Diaries or logs Workplace observation

3-9 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Example of an observation/competency checklist In responding to the scenario, the student: YesNo  Described the value of interpersonal and communication skills in building a rapport with customers  Described how seeking assistance from others can overcome limitations in customer service delivery  Discussed methods by which they can determine and address customer needs and expectations  Described methods by which customer complaints could be dealt with quickly, effectively and courteously  Explained how gathering feedback can assist an organisation in improving its customer service standards  Explained the steps involved in building a positive relationship with customers  Discussed methods by which customer service levels could be monitored to ensure consistency  Described methods of reporting deficiencies in service standards to management  Described the difference between different customer satisfaction levels  Successfully identified opportunities to enhance the service given to their customer  Demonstrated problem solving skills in overcoming customer complaint and negative reactions by: o using positive listing skills o using positive body language o asking effective questions o offering solutions and listening to customer reactions

3-10 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Analysing information Regardless of whether you are reviewing the RTO’s entire operation or only selected segments, the process will be the same: the actual documents and procedures will need to be crosschecked and compared to the legislative or organisational standards to determine if they are compliant and meet requirements. Examples of what to look for follow...

3-11 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Learning and assessment strategy Have these strategies been developed in line with the relevant qualification’s packaging rules? Has the latest version of the qualification summary been used so that all unit codes and benchmarks are correct? Have all the core units of the qualification been included? Have the elective units been chosen with workforce development outcomes in mind? Have all necessary prerequisite units been observed and included? Are these units being delivered in the right order? Is the strategy aimed at a specific target group and if so: –does the strategy detail all the necessary resources needed for these learners to complete the program successfully? –are the delivery methods relevant to this target group? –is the duration of the training program sufficient to allow the target group to complete successfully? –does the program address workforce development needs?

3-12 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Assessment tools The review of these tools should be conducted with the unit/s of competency close to hand. The assessment tool should be compared to and should encompass the unit’s performance criteria as well as any other requirements as outlined in the individual units. Other aspects of assessment review will include the level of moderation and validation the assessment tool has received: –how often is moderation and validation undertaken? –what recommendations were made and what action was taken? Check that the assessment is appropriate to the AQF level at which it is aimed.

3-13 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Program documentation Have student details been entered correctly? Is the record of their progress through the qualification accurate and up to date? Have completed assessment tools been marked and stored appropriately? Have all necessary forms and documents been completed accurately and in a timely manner? Are reporting requirements (such as AVETMISS) being completed accurately and on time?

3-14 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Compliance with ASQA Standards In all cases, regardless of what is being reviewed, the RTO must measure its performance against the ASQA Standards. A condition of registration is that these standards be upheld and met at all times. This can be achieved through adherence to a continuous improvement program and the development of checklists or forms that can be used during an evaluation process.

3-15 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Documenting findings ASQA STANDARD SNR 4: The applicant must have strategies in place to provide quality training and assessment across all of its operations 4.1 The applicant has a defined continuous improvement strategy that allows for the collection and analysis of data. The strategy includes implementation of continuous improvement activities for training and assessment. EvidenceResult  The RTO does not currently have a formalised method of ensuring continuous improvement.  No records are kept of student feedback and there is no evidence that the RTO discussed feedback during staff meetings. Compliant Non-compliant Not audited Table 3.4 Self-audit document with non-compliances shown Source:

3-16 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Factors that affect performance Job redesign Change in supervision arrangements Change in work support/mentoring Communication and information campaigns Job aids Standard operating procedures

3-17 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Reporting and recommendations The final part of the review process is the formal report on the findings of the evaluations undertaken. This report should then, ideally, form part of the organisation’s continuous improvement program. The report should include information on any breaches of compliance identified as well as suggestions for their rectification.

3-18 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Making recommendations Any suggestions made must be discussed with relevant stakeholders prior to formalising the report. There is little point in making a prescribed recommendation if the organisation is unable to meet the suggested pathway or alternative. While compliance with the ASQA standards is not negotiable, there are generally a number of methods that can be used to ensure that these standards are indeed met and discussions with stakeholders at the end of the evaluation process can identify the best ways of rectifying any areas identified for improvement.

3-19 Copyright © 2013 McGraw-Hill Education (Australia) Pty Ltd Hill, Hill, Perlitz, Professional Training & Assessment, 1e Evaluation report ASQA STANDARD SNR 4: The applicant must have strategies in place to provide quality training and assessment across all of its operations 4.1 The applicant has a defined continuous improvement strategy that allows for the collections and analysis of data. The strategy includes implementation of continuous improvement activities for training and assessment. EvidenceResult  The RTO does not currently have a formalised method of ensuring continuous improvement.  No records are kept of student feedback and there is no evidence that the RTO discussed feedback during staff meetings. Compliant Non-compliant Not audited Rectification required:  Produce and file minutes of all meetings to ensure that continuous improvement issues are discussed with staff  Produce action plans, based on feedback received from students, employers and staff  Keep a written continuous improvement log Table 3.6 Rectification report (extract)