Consistency of Assessment (Validation) Webinar – Part 2 Renae Guthridge WA Training Institute (WATI)

RQI What three things do you recall about yesterday’s webinar? What two questions do you have about yesterday’s? What was one interesting point about yesterday’s webinar?

Foxes and a Vixen Principles of Assessment Rules of Evidence Four Very Rampant Foxes A Satisfied Cunning Vixen

Ensuring Validity - Unpacking the Unit of Competence Mandatory unit requirements include: Pre-requisites and/or co- requisites Elements Required Skills Required Knowledge Range Statement (‘must’) Critical Aspects of Evidence Other “must” requirements Mandatory unit requirements include: Pre-requisite(s) Elements Foundation Skills Performance Evidence Knowledge Evidence These can be listed in a Unit Requirements List…

Unit Requirements List Unit of competency: SHOP123A Review stock Mandatory unit requirements Elements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Foundation skills 1. Literacy skills (reading and writing) 2. Numeracy skills Performance evidence 1. Accurate list of stock on display 2. Accurate list of stock in store-room 3. Accurate list of stock to order 4. Two categories of stock Knowledge evidence 1. Organisation procedures 2. Range of stock management processes Pre-requisites : SHOP101A Unit of competency: SHOP123A Review stock Mandatory unit requirements Elements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement (musts) 1. Stock on display 2. Stock in storeroom Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list) Pre-requisites : SHOP101A Other (specify): Two contexts Old Unit StructureNew Unit Structure

Ensuring Validity - Identifying Mandatory Unit Requirements Highlight all mandatory requirements in a unit of competency. Prepare a list of all the mandatory requirements. Validate your list with a colleague by comparing lists. Prepare a flexible and fair evidence-gathering plan. Validate your plan with a colleague.

Planning to Gather Evidence The Unit Requirements List can now be extended to show the relationship between the unit requirements and evidence sources.

Unit of competency: SHOP123A Review stock Mandatory unit requirements Sources of evidence WP1WP2SIM1SIM2Q/A Elements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement 1. Stock on display 2. Stock in storeroom Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (see list) Pre-requisites: SHOP101A Other (specify): Two contexts WP1: Context of workplace assessment WP2: Context of workplace assessment SIM1: Context of simulation SIM2: Context of simulation Assessment Planning Matrix Shade the cells that don’t apply. The blank cells in a column provide a space to record evidence later on when it has been gathered.

Developing Evidence-gathering Tools - Assessing Required Knowledge / Knowledge Evidence Assessing ‘know-how’… …tell me… …written or oral short answer tests.

Ensuring Validity, Reliability and Sufficiency - Assessing Required Knowledge / Knowledge Evidence Is the AQF level of the questions appropriate? Is the context and content of the questions relevant to the unit? Do the questions extend beyond the elements and demonstrate transfer? Is ALL required knowledge covered? If other knowledge is covered, has it been distinguished from the required knowledge? Is ALL required knowledge demonstrated before signing off on the unit record?

Developing Evidence-Gathering Tools - Assessing Elements and Required Skills / Performance Evidence Assessing can do… …show me… …observation in the workplace or a simulation of a workplace.

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Mandatory unit requirementsPerformance CriteriaCommentsResult E1. Identify stock requirements 1.1 Contextualised for third party observer. The observer would write a narrative of what was observed for the assessor to review E2. List stock in store RK1 Stock on display Display stock not reviewed N/a RK2 Stock in storeroom CAE1 Accuracy and timeliness Review completed to company standards CAE2 Full review of stock All storeroom stock reviewed RS1 Communicates with staff RS2 Correct forms selected Assessment context: Two different contexts requiredFrozen foods Observer’s comments: Observer’s name: M. Ployer Observer’s position: Supervisor Signature and date : M. Ployer 5/8/10 Student signature and date: Joe Bloggs 5/8/10 Workplace Observation Tool 1 (WP1) (3 rd Party Observer)

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Mandatory unit requirementsPerformance CriteriaCommentsResult E3. List stock shortages E4. Prepare stock order forms RK1 Stock on display RK2 Stock in storeroom Storeroom stock not reviewedN/a CAE1 Accuracy and timeliness Met our company standards CAE2 Full review of stock Full range of display stock reviewed RS1 Communicates with staff Did not ask for advice or report problems to supervisor  RS2 Correct forms selected RS3 Correct calculations Assessment context: Two different contexts requiredDry goods Observer’s comments: Observer’s name: Sam Smith Observer’s position: RTO Assessor Signature and date: Sam Smith 7/9/10 Student signature and date: Joe Bloggs 7/9/10 Workplace Observation Tool 2 (WP2) (RTO Observer)

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Mandatory unit requirementsPerformance CriteriaCommentsResult E1. Identify stock requirements E2. List stock in store E3. List stock shortages3.1 Incorrect stock list used.  3.2 E4. Prepare stock order forms4.1  RK1 Stock on display RK2 Stock in storeroom N/a CAE1 Accuracy and timeliness CAE2 Full review of stock RS2 Correct forms selected RS3 Correct calculations Assessment context: Two different contexts requiredDry goods Observer’s comments: Observer’s name: Sam Smith Observer’s position: RTO Assessor Signature and date: Sam Smith 9/10/10 Student signature and date: Joe Bloggs 9/10/10 Simulation Observation Tool (SIM)

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Required knowledge items QuestionsVerbatim answers or commentsCorrectScoreCriterionResult RK RK  6 Hot water is mixed with soap powder to make a warm nourishing drink  7 8  RK RK Record the incorrect answer here ….  RK RK RK  Overall unit result  Extra knowledge Questions EK1. 19 EK2. 20 Sandals can be worn in the cool room  EK3. 21 EK4. 22 Observer’s name: Sam Smith Observer’s position: RTO Assessor Signature and date: Sam Smith 9/8/15 Student signature and date: Joe Bloggs 9/8/15 Question & Answer Tool (Q/A)

Ensuring Validity, Reliability and Sufficiency - Assessing Elements and Required Skills / Performance Evidence Are clear instructions provided to the observer? Are clear instructions provided to the candidate? Are all four dimensions of competency (task skills, task management skills, contingency management skills, and job/role environment skills) met? If third-party evidence-gatherers are used, have the performance criteria been contextualised by writing them in familiar language (plain English) and with examples, and has space been provided for narrative to explain the or  assigned to each criterion?

Recording evidence Use the Assessment Planning Matrix as a record of individual evidence by adding the name of the student and a ‘Result’ column and recording evidence from the tests and observation checklists into the blank cells.

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Mandatory unit requirements Sources of evidence WP1WP2SIM1SIM2Q/A Elements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages  4. Prepare stock order forms  Range Statement 1. Stock on display N/a 2. Stock in storeroom N/a Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff  RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list)  Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts Frozen foodsDry goods WP1: Frozen foods in storeroom WP2: Dry goods on display SIM1: Dry goods on display SIM2: Supplementary assessment Name of assessor: Sam Smith Student signature and date: Signature and date:Result reported date: Assessment Record

Meeting the Rules of Evidence - Recording evidence of competency Are checklists used to record evidence as it happens? Is poor quality evidence discarded? Is evidence gathered and collated from checklists? Is evidence from different units in a cluster distributed to appropriate unit records?

Making the Assessment Judgement The Assessment Record can now become a Judgement Tool showing which requirements had been met in the ‘Results’ column.

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Mandatory unit requirements Sources of evidence Result WP1WP2SIM1SIM2Q/A Elements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages   4. Prepare stock order forms  Range Statement 1. Stock on display N/a 2. Stock in storeroom N/a Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff   RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list)  Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts Frozen foodsDry goods Result  WP1: Frozen foods in storeroom WP2: Dry goods on display SIM1: Dry goods on display SIM2: Supplementary assessment Name of assessor: Sam Smith Student signature and date: Joe Bloggs 6/11/15 Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15 Judgement Tool

Meeting the Foxes and Vixen - Making the assessment judgement Has sufficient evidence been gathered for each mandatory requirement? Is the evidence authentic and current? Have alternative arrangements been offered for candidates with specific needs? Has each mandatory requirement been demonstrated? Has the record been signed off by person(s) with assessor qualifications and vocational expertise? Have other uses of the record been quarantined from the competency judgement?

Training and Assessment Plan The Assessment Planning Matrix, Assessment Record and Judgement Tool can now morph into a Training Delivery and Assessment Plan by listing the learning resources for each unit (and any extra) requirements.

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs Mandatory unit requirements Learning resourcesSources of evidence Result SessionReferenceWP1WP2SIM1SIM2Q/A Elements of competency 1. Identify stock requirements 1 - 3pp List stock in store 4pp List stock shortages 5 - 7Handout 1   4. Prepare stock order forms 9Handout 3  Range Statement 1. Stock on display 8Handout 2N/a 2. Stock in storeroom 8Handout 2 N/a Critical Aspects of Evidence CAE 1 Accuracy and timeliness 4p. 38 CAE 2 Full review of stock 4p. 39 Required Skills RS 1 Communicates with staff 7Handout 1   RS 2 Correct forms selected 8Handout 2 RS 3 Correct calculations 7Handout 1 Required Knowledge (See list) 1 - 9All  Pre-requisites: SHOP101A Student records Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts 1 - 9All Frozen foodsDry goods Result  WP1: Frozen foods in storeroom WP2: Dry goods on display SIM1: Dry goods on display SIM2: Supplementary assessment Workplace or regulatory requirements: OH&S: Use of steps & cool room access Name of assessor (Assessment): Sam Smith Name of Assessor (Vocational): Sam Smith Student signature & date: Joe Bloggs 6/11/15 Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15 Training and Assessment Record

Thoughts… What do you think? Would you use this approach? What do you like / dislike about this approach?

Strategies for Systematically Validating Assessment Procedures Review assessment strategies Compare assessment evidence Evaluate assessment outcomes

How Can We Review Assessment Strategies? Reviewing involves the inspection of assessment processes and products to determine the ‘face validity’ of the assessment strategy…‘does it look right’?

What criteria could be used to guide the review of your assessment processes? Review Criteria…

How Can We Compare Assessment Evidence? Comparing assessment processes involves comparing assessment documentation and comparing assessment outcomes to determine content validity, reliability, consistency, and reproducibility of the assessment strategy.

What criteria could be used to guide the comparison of your assessment processes with those of another assessor? Comparison Criteria…

How Can We Evaluate Assessment Outcomes? Evaluating assessment processes involves gathering stakeholder feedback to determine the predictive validity, impact, effectiveness and credibility of the assessment strategy.

What criteria could be used to guide the evaluation of your assessment processes. Evaluation Criteria…

How Can We Improve Assessment Processes? The quality cycle How can you follow this cycle to improve your assessment processes?

What Records of Validation Should We Keep and Why?

Records of Validation… Create a separate file for each unit of competency. Each file to contain: UoC; Assessment plan; Current assessment instruments and instructions; Validation form; A schedule for the administration of validation forms showing past and future activities; A summary of evidence gathered to date from each validation source; A summary of problems identified; and A summary of actions taken to remedy them.

Before You Go… Answer the following with a yes, no or maybe… 1.I have learned something about validation during these two sessions. 2.I will change some of the assessment processes I currently use. 3.I am more confident about validating with others and sharing my assessment processes. Is there anything else you need to know about validation?

Thank you and watch out for Foxes and Vixens! All pictures in these slides were taken from