Integrating Needs Analysis, Assessment & Evaluation of Training Catherina Opperman & Marius Meyer 2009 www.roionline.co.za 2008

FOCUS D E S I G N NEEDS ANALYSIS ASSESSMENT EVALUATION T R A I N G Skills Audit www.roionline.co.za 2008

STRATEGIC LEARNING ALIGNED TO TRAINING Identify skills gaps: Current gaps that need further development or are lacking Future skills that the organisation will need in the next two to five years www.roionline.co.za 2008

STRATEGIC LEARNING ALIGNED TO TRAINING Skills gaps & training needs: Corporate training needs across entire organisation, example communication skills Job-specific training needs specific to a particular area or position, example product knowledge or marketing skills www.roionline.co.za 2008

STRATEGIC LEARNING ALIGNED TO TRAINING Identify strategic learning objectives from: Business Plan Workplace Skills Plan Strategic HRD Plan Ensure that all job titles are represented on the Plan www.roionline.co.za 2008

STRATEGIC LEARNING ALIGNED TO TRAINING Record and categorise these needs in a Strategic learning and development Plan: Example: BS: Business specific, product knowledge IP: Interpersonal: communication skills IT: computer skills: Excel ML: Management & leadership skills www.roionline.co.za 2008

MAIN ELEMENTS OF A JOB ANALYSIS Purpose of a job: job demands, responsibilities, accountability, activities, procedures, processes Tasks: how often performed, sequence of tasks, equipment needed, how do tasks relate to other jobs continue www.roionline.co.za 2008

DEVELOP THE JOB PROFILE Focus on the job position, not the employee Divide the position into knowledge, skills & behaviours that are essential www.roionline.co.za 2008

METHODS TO COLLECT DATA Job interviews Observing employees in the workplace Interview individuals Interview groups Interview supervisors & technical specialists Questionnaires Self assessment questionnaires www.roionline.co.za 2008

JOB PROFILE ANALYSIS & PERFORMANCE MANAGEMENT The next step is to link the job profile to performance management to assess specific competencies & address gaps that require training Identify gaps, assess & evaluate by linking output requirements to the individual’s Performance management plan www.roionline.co.za 2008

Target population analysis TERMINOLOGY Needs assessment refers to different types of analysis that are referred to as needs analysis: Performance analysis Gap analysis Target population analysis www.roionline.co.za 2008

Needs assessment refers to the three elements of: organisational-, person- and task analysis www.roionline.co.za 2008

DIFFERENT CATEGORIES OF TRAINING NEEDS Macro level: sector skills needs Meso level: a single organisation Micro level: individual needs www.roionline.co.za 2008

Gaps exist as a result of: PERFORMANCE GAPS Gaps exist as a result of: Performance gaps in the way individuals work Management gaps in the way people are managed Organisation gaps in the manner in which organisations are designed www.roionline.co.za 2008

STAKEHOLDERS IN HRD & ASSESSMENT Government External Management Training committee Trade unions ETD providers Critical interest groups Community groups Assessors Moderators Learners Mentors/coaches Customers Past participants Professional bodies SDF Business partners Suppliers www.roionline.co.za 2008

THE NEEDS ANALYSIS PROCESS Obtain data from two sources: The business plan: key strategies & broad competencies Skills of individual employees: assessed against the business plan will identify gaps in performance www.roionline.co.za 2008

REPORT RESULTS AND MAKE RECOMMENDATIONS Feedback will depend on the nature and extent of the issue/problem and could be done verbally or: A comprehensive document with charts & graphs presented to top management Recommendations to take into account: Training strategy, methods & costs Be aware of the influence of culture & management buy-in to training www.roionline.co.za 2008

SELECT OR DESIGN A LEARNING INTERVENTION If recommendations are accepted, use the identified needs for selecting or designing a learning intervention: Learners need to be aware of the objectives of training related to the purpose and expected outcomes of learning activities: What they should be able to know and do Criteria that indicates acceptable level of performance What criteria they will be assessed against Scope of learning www.roionline.co.za 2008

DESIGNING A LEARNING PROGRAMME Take the following into account: Long term vision & mission of organisation Strategic short term goals & strategies Key skills competencies to meet those needs Skills priorities of the organisation Objectives of learning continue www.roionline.co.za 2008

DESIGNING A LEARNING PROGRAMME Take the following into account: Learning outcomes Assessment instruments Assessment procedures Numeracy & literacy levels of learners www.roionline.co.za 2008

OTHER TYPES OF ANALYSIS Organisation or situation analysis Target population analysis Task analysis Skills & competency analysis Skills Audit – Skills Gap Analysis (SGA) www.roionline.co.za 2008

OTHER TYPES OF ANALYSIS: Task analysis Task analysis form: Five levels of function: Job title Specific duty Tasks under each duty Sub tasks Knowledge www.roionline.co.za 2008

SKILLS AUDIT PROCESS Steps in the Skills audit process: 1.Identify current & future priorities from the Business Plan; core work & future strategies 2. Identify essential skills for current & future business 3. Assess current skills level, use a rating scale, example: expert/satisfactory/limited/no experience 4. Identify action to address skills gap continue www.roionline.co.za 2008

2.4 OTHER TYPES OF ANALYSIS Assessment methods: Employee self assessment 360 degree Panel assessments Formal assessment: Diagnostic, formative, summative & recognition of prior learning (RPL) www.roionline.co.za 2008

COMPETENCE Perform whole work roles Against standards of performance Ability to work with your hands (skills), head (knowledge) & heart (attitude, values & behaviour) www.roionline.co.za 2008

ASSESSMENT STRATEGY Analysis of source documents (policy & plan) Assessment context (environment/circumstances) Assessment activities Logistics Scope: range of assessment Instructions to assessors & candidates Evidence requirements Methods to gather evidence www.roionline.co.za 2008

OVERALL ASSESSMENT STRATEGY Plan for the assessment Prepare the learner Conduct assessment Assess evidence & store it Evaluate the process Provide feedback Review at regular intervals www.roionline.co.za 2008

COMPETENCY-BASED ASSESSMENT OBE is a learner-centred process: What learners are to learn is clearly defined Learners’ progress is based on demonstrated achievement Learners’ needs are accommodated through multiple teaching and learning strategies and assessment tools Each learner is provided the time and assistance to realise his/her potential www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS The candidate & assessor meet Draw up an Assessment Plan Conduct assessment Evaluate the evidence www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Record the decision Provide feedback Plan reassessment if necessary Certification Record the results www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Stages of planning the assessment: Initial planning Pre-assessment meeting with learners Draft an Assessment Plan Prepare the candidates Collect the evidence Give feedback continue www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Special education needs examples: Illiterate learners Semi-literate learners Innumerate learners Physical disabled learners Blind learners Deaf learners www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Plan against contingencies that may arise Obtain results of previous assessments Design activities & instruments that are appropriate to the outcomes & resources Assessment documentation is prepared for the recording of information continue www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Potential unfair barriers are identified and plans are made to address these barriers without compromising the validity of the assessment. The required physical and human resources are to be ready and available for use, and logistical arrangements are confirmed continue www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Provision for moderation is made in accordance with relevant assessment policies and ETQA requirements. A variety of assessment methods are described and compared i.t.o. strengths, weaknesses and applications www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS The Appeals procedure The re-assessment procedure Follow up and support The signatures of all the relevant parties & dates www.roionline.co.za 2008

PLAN THE ASSESSMENT PROCESS Prepare the candidates: Use appropriate language level Carry out checks to ensure candidates are ready for assessment Ensure assessments are in line with policies Provide opportunities for input from candidates www.roionline.co.za 2008

ASSESSMENT PRACTICES Validation of assessment: Verification of the process Statistical analysis Examination of the assessment instrument Sampling of evidence of applied competence Observation of processes Site visits Interviews www.roionline.co.za 2008

ASSESSMENT PRACTICES The principles of good assessment practice: Fairness Validity Reliability Practicability www.roionline.co.za 2008

ASSESSMENT PRACTICES Valid Authentic Current Sufficient Quality of evidence refers to ‘VACS’ criteria: Valid Authentic Current Sufficient www.roionline.co.za 2008

ASSESSMENT PRACTICES Types of evidence: Refers to the method that will be used to collect evidence. Will it be collected by directly observing the learning? By making use of secondary sources of evidence? or Historical evidence? www.roionline.co.za 2008

ASSESSMENT PRACTICES Appeals can be brought against: Unfair assessments Invalid assessments Unreliable assessments The assessors’ judgement; if considered biased Inadequate expertise and experience of the assessor if it influenced the assessment decision Unethical assessment practices www.roionline.co.za 2008

ASSESSMENT PRACTICES Guide and support learners: Advise Coach Tutor Counsel Mentor continue www.roionline.co.za 2008

ASSESSMENT INSTRUMENT Design the assessment instrument according to the assessment strategy that includes the following elements: Determine the purpose of learning Analyse the needs of learners Decide on the learning objectives and outcomes Select the content that will support the achievement of outcomes Decide on the activities, methods, and media for learning and development www.roionline.co.za 2008

ASSESSMENT PROCESS Apply integrated assessment strategy Include Policy documents Provide for RPL, Reassessment, Special needs, language requirements Identify the level of learner’s competence Activities to include knowledge, skills & attitudes continue www.roionline.co.za 2008

ASSESSMENT PROCESS Identify methods: Knowledge Observation 3rd form of evidence Design instruments: Case study Discussions Portfolio continue www.roionline.co.za 2008

COLLECT THE EVIDENCE Identify the sources to collect evidence from: Direct evidence Indirect evidence Historical evidence www.roionline.co.za 2008

MARKING MEMORANDUM Marking Memorandum is used to score the outcomes of assessment Include: Marks assigned per question The total marks of the assignment & paper The marking memorandum can be combined with model answers www.roionline.co.za 2008

Establish the value of learning Decide to continue with a programme PURPOSE OF EVALUATION Establish the value of learning Decide to continue with a programme Obtain info on how to improve future learning programmes Improve delivery mechanisms to be more efficient and less costly www.roionline.co.za 2008

WHAT IS EVALUATION? “The collection of, analysis, and interpretation of information about any aspect of a programme of education or training as part of a recognised process of judging its effectiveness, its efficiency, and any other outcomes it may have.” (Mary Thorpe) www.roionline.co.za 2008

ELEMENTS OF AN EVALUATION STRATEGY Background: purpose The scope of evaluation: levels & activities Output: measure for evaluation Methodology: integration with other processes continue www.roionline.co.za 2008

ELEMENTS OF AN EVALUATION STRATEGY Work plan: roles & responsibilities Evaluation instruments: questionnaires, etc. Resources: human, technology, budget Reporting & communication www.roionline.co.za 2008

AREA OF MEASUREMENT EXAMPLES Customer support Systems Product knowledge Technology www.roionline.co.za 2008

What can you increase / decrease? MEASUREMENT IMPACT Quality – How good? Standards? Criteria? Quantity – How many? Cost - What are the cost implications? Savings? Time – How quick / long? When? Faster? What can you increase / decrease? www.roionline.co.za 2008

TRAINING COSTS Gap analysis Design & Development Facilitation fees Assessment Evaluation Fixed costs Other costs www.roionline.co.za 2008

EVALUATION MODELS Kirkpatrick Phillips Nadler Brinkerhoff Holton Coetsee and Van Zyl www.roionline.co.za 2008

EVALUATION PROCESS: DATA COLLECTION METHODS Checklists Questionnaires Surveys Needs Analysis Performance Audit Case studies Focus groups continue www.roionline.co.za 2008

EVALUATION PROCESS: DATA COLLECTION METHODS Meetings Interviews Observation Role plays Simulations Work samples www.roionline.co.za 2008

ROI% = (benefits – costs) x 100 ROI FORMULA R R R R ROI% = (benefits – costs) x 100 ___________ costs R www.roionline.co.za 2008

NON-TRAINING VARIABLES External influences that had an impact on the measurement are referred to as non-training variables. External influences such as new systems and products, trends and business cycles pose threats to validity when measuring ROI. www.roionline.co.za 2008

NON-TRAINING VARIABLES Organisational influences Technology Human influences Other influences www.roionline.co.za 2008

REPORTING Executive summary Background of project Purpose, intent and design of evaluation study Results Discussion Costs and benefits Conclusions, recommendations and options www.roionline.co.za 2008