1. Seminar on the Evaluation of AUT STEM Programme
13 February 2019
Seminar on the Evaluation of AUT STEM Programme
Pauline Hoyle
Director
Cinnabar Consultancy

2. AUT STEM-TEC seminar Aims of this session:
To consider the elements needed for effective evaluation of the AUT STEM-TEC programme
To consider the elements of the Theory of Change for the AUT STEM -TEC programme
Outcomes of this session:
To start to define the key elements of the Theory of Change for AUT STEM -TEC programme
To start to define the strategy for evaluating the AUT STEM-TEC programme

3. AUT STEM-TEC programme
Mission statement:
Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy
We aim to achieve this by:
Conducting high quality research in STEM education
Implementing best pedagogical strategies in the teaching and learning of STEM subjects
Ensuring equity and creating opportunity via social and community engagement

4. Measuring our impact
“Those grants supported projects on enhancing teaching and learning of STEM subjects, in particular contributing to increasing the pass and retention rates. Innovative learning and teaching approaches and resources developed in those projects lead to growth in successful graduates and to number of highly skilled employees in STEM related industry”.
“Due to the nature of the publications in STEM education as a teaching-research nexus most of them have a direct impact on and implementation in the teaching practice. In particular, more productive and successful teachers and students retained in the system”.
“Those collaborative activities inform and influence STEM education policy and practice in the participating institutions and countries and increase the national and international reputation of AUT”.
STEM-TEC Centre’s Highlights over

5. Theory of Change for STEM -TEC
Our mission
Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy
Who we work with
What we do
What we achieve ( outcomes)

6. University of London 50:50 Engineering Engagement programme
Theory of Change and evaluation strategy

7. Theory of Change of University of London 50:50 Engineering Engagement programme
Vision:
“Engage, inform and inspire a new generation of engineers from a diverse range of backgrounds”
Six priorities of the programme
Promoting inclusiveness and equality
Learning through experimental engineering
Inspiring relatable engineering role models
Discovering STEM Career pathways
Prioritising primary education
Supporting teachers’ professional development and learning

8. Theory of change UCL – Who we work with
The 50:50 UCL Enginnering Engagement programme works with
Children and young people - pre and post 16 in schools and colleges
Teachers – in schools, colleges and informal settings
UCL staff and students who are trained to deliver engagement activities
Employers and funders who provide funding, resources and staff to help deliver the engagements
Partners – in STEM learned bodies, STEM organisations and educations, museums, industry who support the work of the programmes

9. Theory of Change UCL- what we do
The programme provides sustained, meaningful engagements through activities that are about
Designing solutions for contextualised engineering challenges
Problem solving in real contexts relevant to young people
Engaging with engineering skills and design processes in practical hands-on experience in laboratories
Working with expert role models: engineers, scientists and post graduate students from diverse backgrounds
Engagement with cutting edge STEM research

10. Theory of Change UCL – Outcomes
Outcomes for young people:
Inspiring more young people from a range of diverse backgrounds into a future in engineering and technology
Increasing young peoples’ knowledge, skills and understanding in engineering and technology
Increasing the likelihood of young peoples’ knowledge of pathways into engineering and technology
Increasing young peoples’ knowledge of the work of real-life engineers and engineering processes
Increasing young peoples’ knowledge about pathways into UCL Faculty of Engineering

11. Theory of Change UCL – outcomes
Outcomes for Adults
Increasing teachers’ knowledge, skills and understanding of, and their confidence in, teaching concepts, engineering design skills and process
Increasing teachers, employers, UCL students and staff knowledge, skills and understanding of how to inspire young people into engineering and technology
Increasing teachers’, employers’, UCL staff and students’ understanding of an inclusive approach in engineering and technology

12. What makes evaluation effective?
To be effective evaluation needs to
measure achievement of intended outcomes and additional impacts
be embedded in the learning process and post-activity actions
evaluation and learning processes are merged so that learning instruments simultaneously function as evaluation tools
evaluation is integrated into planning and implementation of post-activity intervention
not to be heavy and not difficult to implement
be useful for participants and providers
Additional benefits:
Evaluation of interventions and its outcomes speeds up integration of successful strategies (embedding)
If you have tangible evidence it is easier to convince colleagues, leaders – more dissemination
Scaffolding professional learning

13. Research behind self –reported evaluation
Guskey ( 2000) model of impact levels
Level 1 participant’s reactions (immediately)
Level 2 participants learning from engagement (immediately and after 6-8 weeks)
Level 3 organisational support and change (after 6-8 weeks and 6-9 months)
Level 4: participants’ use of new knowledge and skills (after 6-9 months)
Level 5: Student outcomes (after 6-9 months)
For CPD/engagement to be effective and impactful:
CPD/engagement outcomes are clearly stated and their achievement evaluated
post-CPD/engagement intervention and its outcomes are clearly planned and reflected in the Action Plan
evaluation process is embedded in the planning and implementation of post-CPD/engagement actions

14. UCL Evaluation Strategy
Mixed methodology:
analysis of young peoples’ responses pre and post engagement
the impact observed by the teachers of young people
impact noted by tutors (staff or students)
evaluation tasks embedded into engagement strands (where young people engage in activities of 30+ hours)
external evaluator observations.
Evidence of impact from:
initial reaction to events
changes in participants’ knowledge and/or practice over 3-6 months for medium impact activities or 6-9 months for high impact activities,
changes in participants’ institution (i.e. teachers practice or school provision).

15. Evaluating outcomes for different strands
Evaluation tools
Science and engineering summer schools
SECONDARY
OC1: Inspiring more young people from a range of diverse backgrounds into a future in engineering and technology
OC2: Increasing young people’s knowledge, skills and understanding in engineering and technology;
OC3: Increasing the likelihood of young peoples’ knowledge of pathways into engineering and technology
Young people
Pre-activity questionnaire (outcomes1,2,3,)
During-session card activity ( outcome 2)
Post activity questionnaire ( outcomes 1,2,3)
Teachers/Leadership
post activity questionnaire (outcomes 1,2,3)
UCL Tutors
post activity questionnaire outcomes 1,2,3)
Supporting the Computing Curriculum
PRIMARY
OC2: Increasing young people’s knowledge, skills and understanding in engineering and technology;
OC3: Increasing the likelihood of young peoples’ knowledge of pathways into engineering and technology
OC6: Increasing teachers’ knowledge, skills and understanding of, and their confidence in, teaching concepts, engineering design skills and process
Young People
Pre-activity questionnaire (outcomes 2,3)
During- session card activity (outcome 2)
Post activity questionnaire (outcomes 2,3)
Teachers/ Leadership
post activity questionnaire (outcomes 2,3,6)
post activity questionnaire (outcomes 2,3,6)

16. Discussion of the Theory of Change for STEM –TEC
Our mission
Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy
Card activity to define who work with and what we do.
Report back after 15 mins
Card activity to define our outcomes

17. Outcomes of the session
Outcomes of this session:
To start to define the key elements of the Theory of Change for AUT STEM -TEC programme
To start to define the strategy for evaluating the AUT STEM-TEC programme

18. Thanks for your time and input today