1. An integrated approach: Holistic assessment of NZ dairy farming trainees
Graeme Couper

2. Background
Research project for a thesis towards a Masters degree in Education at Massey University
At the time, I was Education Manager at Primary ITO
Left Primary ITO in December, 2016 and am currently Education programmes manager at the Dept of Corrections
This research project was done with the permission of Primary ITO during my time there, however I, and this presentation do not represent Primary ITO at this forum

3. Context of the research
Changes to qualifications due to TRoQ
Previous assessment approach had been unit standard focussed and practical and knowledge units had been assessed separately
Looking for a change in assessment approach:
More holistic and better integration of theory and practical
Better aligned to Graduate Outcomes of new qualifications
Less over-assessing
Looking for ways to assess knowledge in a more practical context
Primary changes due to TRoQ were the introduction of graduate outcomes that assessed learners skills, knowledge and behaviour – the psychomotor, cognitive and affective domains.
Describe current state at the time.
Theory
Taught by contracted providers
Classroom based
Unit standard focussed
Not necessarily done at a time when practical was being done on farm – e.g. animal health unit standard all done at one time
Assessment mostly written, short answer, open book assessments
Practical
Assessment based around a practical “work diary” of evidence of various tasks
Unit standard based
Not necessarily aligned to theory
Practical assessment done by Primary ITO training advisers, with workplace employer or supervisor as a “verifier”.

4. Purpose of the research
Evidence base to determine the effectiveness of the new assessment approach
Add to the current (very minimal) amount of research on workplace-based assessment in the Vocational Education and Training sector.

5. Research design Qualitative, case study – based approach
Dairy farm trainees
Process/Timeline
Literature review and research design (Feb – Nov 2015)
Identify suitable trainees from database (Nov/Dec 2015)
Trainees begin programme (Jan/Feb 2016)
Initial visit to seek involvement (April 2016)
Final interview visit (June/July 2016)
Reason for choosing dairy farm trainees was the pastoral quals were the first programmes that had been rolled out. Much greater number of dairy trainees compared to sheep/beef trainees and generally more accessible, and in tighter clusters.

6. Research questions
How does an integrated theory and practical assessment approach compare with traditional, separate written and practical assessment methods, from a learner’s, an employer’s and an assessor’s perspective?
To what extent has this integrated assessment approach enabled learners to apply theoretical knowledge they have learned to practical workplace situations?

7. Data collection realities!
Some drop-off of participant numbers
Had to modify interview technique
Range of other issues cropped up
Consent given
Final data collection
Trainees 14 10
Verifiers 12 9
Assessors 4
Total 30 23
Trainee numbers dropped off at various stages.
One North Island trainee and his employer gave consent but subsequently the employer decided to leave the country in May (at the end of the season) and their trainee found a new job for next season in the South Island so they were removed from the study.
When the final visits were done in June/July, one trainee and his employer had not really engaged in the training and had not undertaken any assessment conversations and a further trainee had pulled out of the programme, which meant that no usable data could be obtained from those employees or their employers. A further trainee was not available on the day of the final visits. Two attempts were made to do the interview by phone at a later date but the trainee did not engage either of these times.
Modification of interview techniques
Initially structured around pre-determined interview questions that asked participants to firstly discuss the old method, then the new on, and then make a comparison
This did not work well, as all participants, when asked questions about the old assessment method, made direct comparisons with the new one
Had to refine interview technique to allow for this.
Other issues included:
Unavailability on the day – had to do a couple by phone at a later date
A couple were busy and only had a short time to interview

8. Research findings
All groups of research participants found that using the new assessment method, compared to the previous method resulted in:
Theoretical knowledge learning being better integrated with practical on-farm skills
More flexible, active and engaged learning
More “real world” evidence available to make assessment decisions
More active assessment interaction between those involved in the assessment process
Greater satisfaction and enjoyment with new assessment method
Some practical challenges of the change
Practical challenges were:
Difficulties adjusting to the new method
Timing of the marking of the knowledge aspects of the portfolio
Timing of the classroom and on-farm activities

9. Participant observations
Below are some examples of the participants observations about the new integrated assessment portfolio
“It makes it easier for them to learn it. They are practical people and they are relating it (to their farm)every day” – Assessor
“It was more practical. It was more about what’s going on on our farm” – Trainee.
“There seems to be a lot more cross-referencing between farm and class than in the past” – Verifier (farmer)
“Theory and practical work more hand in hand because it is right there. They are completing them at the same time in the workbook.” – Assessor
“I learn it much better than just doing a written assessment” – Trainee
“They have a better understanding of why” - Assessor

10. Implications of the research
Application of knowledge to the workplace
Resulted in:
Deeper understanding of theoretical concepts
More ability to apply theory to workplace settings
Knowledge being constructed in a relevant, workplace-based setting
Practical performance as an indication of theoretical knowledge
The key points here are:
The contextualisation of the theoretical knowledge learning, to a familiar workplace setting
Ability to immediately apply new learning in a practical context
Use of authentic evidence, from learner’s own workplace in the assessments
Example of Practical performance as an indication of theoretical knowledge (and not over-assessing)
practical performance for dairy cow mating instead of theory-based assessment
Ability to diagnose and treat diseases
Allocation of grazing area and feed amounts

11. Implications of the research
More Authentic and Robust Assessment
Due to:
Inclusion of “real world” on-farm evidence in knowledge assessment
More interaction and discussion between Trainees, Verifiers and Assessors

12. Implications of the research
Opportunity to revisit “competence”
Holistic, rounded view of learners’ skills, knowledge and behaviour
Suggested model could incorporate:
Technical – Skills and knowledge related to workplace tasks
Application – Ability to apply skills and/or knowledge to different contexts using analytical, critical thinking or problem-solving skills
Personal – Behavioural aspects such as attitudes, motivation, self-reflection, self-correction, self-organisation, ability to learn
Social – Communication and social interaction
Current model tends to separate skills and knowledge, and doesn’t really consider behaviour. It certainly has not traditionally considered these three aspects in a holistic model.

13. Questions? Acknowledgements
Supervisors -Dr Jenny Poskitt and Dr Alison Sewell
Primary ITO
Management for their support
Assessors and field staff who participated
Farmers and trainees
My very patient, tolerant and supportive wife
Questions?