Seminar on the Evaluation of AUT STEM Programme 13 February 2019 Seminar on the Evaluation of AUT STEM Programme Pauline Hoyle Director Cinnabar Consultancy

AUT STEM-TEC seminar Aims of this session: To consider the elements needed for effective evaluation of the AUT STEM-TEC programme To consider the elements of the Theory of Change for the AUT STEM -TEC programme Outcomes of this session: To start to define the key elements of the Theory of Change for AUT STEM -TEC programme To start to define the strategy for evaluating the AUT STEM-TEC programme

AUT STEM-TEC programme Mission statement: Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy We aim to achieve this by: Conducting high quality research in STEM education Implementing best pedagogical strategies in the teaching and learning of STEM subjects Ensuring equity and creating opportunity via social and community engagement

Measuring our impact “Those grants supported projects on enhancing teaching and learning of STEM subjects, in particular contributing to increasing the pass and retention rates. Innovative learning and teaching approaches and resources developed in those projects lead to growth in successful graduates and to number of highly skilled employees in STEM related industry”. “Due to the nature of the publications in STEM education as a teaching-research nexus most of them have a direct impact on and implementation in the teaching practice. In particular, more productive and successful teachers and students retained in the system”. “Those collaborative activities inform and influence STEM education policy and practice in the participating institutions and countries and increase the national and international reputation of AUT”. STEM-TEC Centre’s Highlights over 2014-2018

Theory of Change for STEM -TEC Our mission Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy Who we work with What we do What we achieve ( outcomes)

University of London 50:50 Engineering Engagement programme Theory of Change and evaluation strategy

Theory of Change of University of London 50:50 Engineering Engagement programme Vision: “Engage, inform and inspire a new generation of engineers from a diverse range of backgrounds” Six priorities of the programme Promoting inclusiveness and equality Learning through experimental engineering Inspiring relatable engineering role models Discovering STEM Career pathways Prioritising primary education Supporting teachers’ professional development and learning

Theory of change UCL – Who we work with The 50:50 UCL Enginnering Engagement programme works with Children and young people - pre and post 16 in schools and colleges Teachers – in schools, colleges and informal settings UCL staff and students who are trained to deliver engagement activities Employers and funders who provide funding, resources and staff to help deliver the engagements Partners – in STEM learned bodies, STEM organisations and educations, museums, industry who support the work of the programmes

Theory of Change UCL- what we do The programme provides sustained, meaningful engagements through activities that are about Designing solutions for contextualised engineering challenges Problem solving in real contexts relevant to young people Engaging with engineering skills and design processes in practical hands-on experience in laboratories Working with expert role models: engineers, scientists and post graduate students from diverse backgrounds Engagement with cutting edge STEM research

Theory of Change UCL – Outcomes Outcomes for young people: Inspiring more young people from a range of diverse backgrounds into a future in engineering and technology Increasing young peoples’ knowledge, skills and understanding in engineering and technology Increasing the likelihood of young peoples’ knowledge of pathways into engineering and technology Increasing young peoples’ knowledge of the work of real-life engineers and engineering processes Increasing young peoples’ knowledge about pathways into UCL Faculty of Engineering

Theory of Change UCL – outcomes Outcomes for Adults Increasing teachers’ knowledge, skills and understanding of, and their confidence in, teaching concepts, engineering design skills and process Increasing teachers, employers, UCL students and staff knowledge, skills and understanding of how to inspire young people into engineering and technology Increasing teachers’, employers’, UCL staff and students’ understanding of an inclusive approach in engineering and technology

What makes evaluation effective? To be effective evaluation needs to measure achievement of intended outcomes and additional impacts be embedded in the learning process and post-activity actions evaluation and learning processes are merged so that learning instruments simultaneously function as evaluation tools evaluation is integrated into planning and implementation of post-activity intervention not to be heavy and not difficult to implement be useful for participants and providers Additional benefits: Evaluation of interventions and its outcomes speeds up integration of successful strategies (embedding) If you have tangible evidence it is easier to convince colleagues, leaders – more dissemination Scaffolding professional learning

Research behind self –reported evaluation Guskey ( 2000) model of impact levels Level 1 participant’s reactions (immediately) Level 2 participants learning from engagement (immediately and after 6-8 weeks) Level 3 organisational support and change (after 6-8 weeks and 6-9 months) Level 4: participants’ use of new knowledge and skills (after 6-9 months) Level 5: Student outcomes (after 6-9 months) For CPD/engagement to be effective and impactful: CPD/engagement outcomes are clearly stated and their achievement evaluated post-CPD/engagement intervention and its outcomes are clearly planned and reflected in the Action Plan evaluation process is embedded in the planning and implementation of post-CPD/engagement actions

UCL Evaluation Strategy Mixed methodology: analysis of young peoples’ responses pre and post engagement the impact observed by the teachers of young people impact noted by tutors (staff or students) evaluation tasks embedded into engagement strands (where young people engage in activities of 30+ hours) external evaluator observations. Evidence of impact from: initial reaction to events changes in participants’ knowledge and/or practice over 3-6 months for medium impact activities or 6-9 months for high impact activities, changes in participants’ institution (i.e. teachers practice or school provision).

Evaluating outcomes for different strands Evaluation tools Science and engineering summer schools   SECONDARY OC1: Inspiring more young people from a range of diverse backgrounds into a future in engineering and technology OC2: Increasing young people’s knowledge, skills and understanding in engineering and technology; OC3: Increasing the likelihood of young peoples’ knowledge of pathways into engineering and technology Young people Pre-activity questionnaire (outcomes1,2,3,) During-session card activity ( outcome 2) Post activity questionnaire ( outcomes 1,2,3) Teachers/Leadership post activity questionnaire (outcomes 1,2,3) UCL Tutors post activity questionnaire outcomes 1,2,3) Supporting the Computing Curriculum PRIMARY OC2: Increasing young people’s knowledge, skills and understanding in engineering and technology; OC3: Increasing the likelihood of young peoples’ knowledge of pathways into engineering and technology OC6: Increasing teachers’ knowledge, skills and understanding of, and their confidence in, teaching concepts, engineering design skills and process Young People Pre-activity questionnaire (outcomes 2,3) During- session card activity (outcome 2) Post activity questionnaire (outcomes 2,3) Teachers/ Leadership post activity questionnaire (outcomes 2,3,6) post activity questionnaire (outcomes 2,3,6)

Discussion of the Theory of Change for STEM –TEC Our mission Increasing the number and improving the quality of STEM graduates for the New Zealand knowledge economy Card activity to define who work with and what we do. Report back after 15 mins Card activity to define our outcomes

Outcomes of the session Outcomes of this session: To start to define the key elements of the Theory of Change for AUT STEM -TEC programme To start to define the strategy for evaluating the AUT STEM-TEC programme

Thanks for your time and input today