An integrated approach: Holistic assessment of NZ dairy farming trainees Graeme Couper

Background Research project for a thesis towards a Masters degree in Education at Massey University At the time, I was Education Manager at Primary ITO Left Primary ITO in December, 2016 and am currently Education programmes manager at the Dept of Corrections This research project was done with the permission of Primary ITO during my time there, however I, and this presentation do not represent Primary ITO at this forum

Context of the research Changes to qualifications due to TRoQ Previous assessment approach had been unit standard focussed and practical and knowledge units had been assessed separately Looking for a change in assessment approach: More holistic and better integration of theory and practical Better aligned to Graduate Outcomes of new qualifications Less over-assessing Looking for ways to assess knowledge in a more practical context Primary changes due to TRoQ were the introduction of graduate outcomes that assessed learners skills, knowledge and behaviour – the psychomotor, cognitive and affective domains. Describe current state at the time. Theory Taught by contracted providers Classroom based Unit standard focussed Not necessarily done at a time when practical was being done on farm – e.g. animal health unit standard all done at one time Assessment mostly written, short answer, open book assessments Practical Assessment based around a practical “work diary” of evidence of various tasks Unit standard based Not necessarily aligned to theory Practical assessment done by Primary ITO training advisers, with workplace employer or supervisor as a “verifier”.

Purpose of the research Evidence base to determine the effectiveness of the new assessment approach Add to the current (very minimal) amount of research on workplace-based assessment in the Vocational Education and Training sector.

Research design Qualitative, case study – based approach Dairy farm trainees Process/Timeline Literature review and research design (Feb – Nov 2015) Identify suitable trainees from database (Nov/Dec 2015) Trainees begin programme (Jan/Feb 2016) Initial visit to seek involvement (April 2016) Final interview visit (June/July 2016) Reason for choosing dairy farm trainees was the pastoral quals were the first programmes that had been rolled out. Much greater number of dairy trainees compared to sheep/beef trainees and generally more accessible, and in tighter clusters.

Research questions How does an integrated theory and practical assessment approach compare with traditional, separate written and practical assessment methods, from a learner’s, an employer’s and an assessor’s perspective? To what extent has this integrated assessment approach enabled learners to apply theoretical knowledge they have learned to practical workplace situations?

Data collection realities! Some drop-off of participant numbers Had to modify interview technique Range of other issues cropped up Consent given Final data collection Trainees 14 10 Verifiers 12 9 Assessors 4 Total 30 23 Trainee numbers dropped off at various stages. One North Island trainee and his employer gave consent but subsequently the employer decided to leave the country in May (at the end of the season) and their trainee found a new job for next season in the South Island so they were removed from the study. When the final visits were done in June/July, one trainee and his employer had not really engaged in the training and had not undertaken any assessment conversations and a further trainee had pulled out of the programme, which meant that no usable data could be obtained from those employees or their employers. A further trainee was not available on the day of the final visits. Two attempts were made to do the interview by phone at a later date but the trainee did not engage either of these times. Modification of interview techniques Initially structured around pre-determined interview questions that asked participants to firstly discuss the old method, then the new on, and then make a comparison This did not work well, as all participants, when asked questions about the old assessment method, made direct comparisons with the new one Had to refine interview technique to allow for this. Other issues included: Unavailability on the day – had to do a couple by phone at a later date A couple were busy and only had a short time to interview

Research findings All groups of research participants found that using the new assessment method, compared to the previous method resulted in: Theoretical knowledge learning being better integrated with practical on-farm skills More flexible, active and engaged learning More “real world” evidence available to make assessment decisions More active assessment interaction between those involved in the assessment process Greater satisfaction and enjoyment with new assessment method Some practical challenges of the change Practical challenges were: Difficulties adjusting to the new method Timing of the marking of the knowledge aspects of the portfolio Timing of the classroom and on-farm activities

Participant observations Below are some examples of the participants observations about the new integrated assessment portfolio “It makes it easier for them to learn it. They are practical people and they are relating it (to their farm)every day” – Assessor “It was more practical. It was more about what’s going on on our farm” – Trainee. “There seems to be a lot more cross-referencing between farm and class than in the past” – Verifier (farmer) “Theory and practical work more hand in hand because it is right there. They are completing them at the same time in the workbook.” – Assessor “I learn it much better than just doing a written assessment” – Trainee “They have a better understanding of why” - Assessor

Implications of the research Application of knowledge to the workplace Resulted in: Deeper understanding of theoretical concepts More ability to apply theory to workplace settings Knowledge being constructed in a relevant, workplace-based setting Practical performance as an indication of theoretical knowledge The key points here are: The contextualisation of the theoretical knowledge learning, to a familiar workplace setting Ability to immediately apply new learning in a practical context Use of authentic evidence, from learner’s own workplace in the assessments Example of Practical performance as an indication of theoretical knowledge (and not over-assessing) practical performance for dairy cow mating instead of theory-based assessment Ability to diagnose and treat diseases Allocation of grazing area and feed amounts

Implications of the research More Authentic and Robust Assessment Due to: Inclusion of “real world” on-farm evidence in knowledge assessment More interaction and discussion between Trainees, Verifiers and Assessors

Implications of the research Opportunity to revisit “competence” Holistic, rounded view of learners’ skills, knowledge and behaviour Suggested model could incorporate: Technical – Skills and knowledge related to workplace tasks Application – Ability to apply skills and/or knowledge to different contexts using analytical, critical thinking or problem-solving skills Personal – Behavioural aspects such as attitudes, motivation, self-reflection, self-correction, self-organisation, ability to learn Social – Communication and social interaction Current model tends to separate skills and knowledge, and doesn’t really consider behaviour. It certainly has not traditionally considered these three aspects in a holistic model.

Questions? Acknowledgements Supervisors -Dr Jenny Poskitt and Dr Alison Sewell Primary ITO Management for their support Assessors and field staff who participated Farmers and trainees My very patient, tolerant and supportive wife Questions?