DR LOU COMERFORD BOYES STEM EVALUATION MANAGER AND RESEARCH FELLOW UNIVERSITY OF BRADFORD Developing Evaluation as Institutional Research: a STEM case study Evaluation Workshop University of Birmingham Sept 2011

Plan for workshop Brief overview of the University of Bradford NE Spoke Legacy project ‘Greening STEM’; Evaluation activities attached to strands; Relationship between project evaluation activity and institutional research at the UoB; Practical activities, discussion and feedback.

Plan for workshop cont. Practical activity 1: - Experiences to date of evaluating your STEM project? What supports you in this? - What STEM project evaluation would you like to do? What’s stopping you? Practical activity 2: - Finding the links between your STEM project evaluation and your organisation’s ways and means of knowing itself (i.e. institutional research): - Where are there existing institutional mechanisms and practices that can help you? - How will your project level good evaluative practice help inform what institutional research should be?

N.E Spoke Legacy Project: Greening STEM Institutional rationale: the project integrates two major themes relevant to the UK economy: STEM and sustainable development Connections between STEM and SD are often weak and fragmented project aim is to pilot, trial and evaluate a range of curriculum learning materials across a range of STEM subjects in order to build support, capacity and leadership for STEM and sustainable development.

Levels of resolution Institutional demonstrator for the ‘greening’ of STEM which comprises as 3-aspected test bed and involves three academic schools: A curriculum stream that will yield self-generated learning materials that combine STEM and SD; A skills stream that enhances existing employer-led initiatives at the University: the creation of a Greening STEM Skills Forum to help identify the scale and nature of employer requirements with regard to STEM and SD - ‘the green skills gap’ - and to create resources; An operations stream to highlight and stimulate opportunities to minimise the environmental footprint and waste associated with STEM activities in order to start addressing the obvious disconnect.

Project strands in brief: Green chemistry: environmental monitoring in a laboratory context – the relationship between the different ways of communicating energy usage and subsequent student behaviour: a 7 fume cupboards experiment; Automotive engineering: multidisciplinary, practical project located in a high impact, substantial core module – a ‘green vehicle’ build to launch a national racing competition; Computing: the creation of web-based resources to encourage students to think about and address the relationship between computing and sustainable development; comparison of an directed and intensively delivered versus light but continuous exposure to materials.

Evaluation map Project strand evaluations Captured as standard e.g. Environmental monitoring can evidence behaviour change in the lab Build opportunities for qualitative data capture with all stakeholders including less usual suspects Tried and tested student voice capture practices in the classroom - sharing of practice between colleagues Reflective practice and student diaries as used in group projects Project wash-ups with students – who do these involve, who should they? Evidence of student progression in assessed work Standard end of module questionnaires - module leaders own question Measurable change in staff behaviour, how & what do they plan in the next iteration, module handbooks.

Mixing up what’s tailor made and existing……….. Snapshot capture: - Ecoversity Short Questionnaires - Post-its and flipcharts at conferences - Talking Wall at events - Mapping the Campus project - Module and Stage evaluations In-depth qualitative : - Ambassadors' reflections - Baseline interviews - Student-led interviews with students Existing secondary data sources: - Acceptors and Decliners data - Estates and Facilities surveys - FYE Experience survey - National Student Survey - Develop Me! (SaPRA) Naturalistic ‘experiment’ of different groups: - Curricular - modules with ESD content - Ambassador & internships - Volunteering (sustainable schools) - Wider student body How can we gauge the impact on students' experiences?

What do these approaches have to do with institutional research? The need to show what’s different and what’s changed in terms of intervention; You can’t reinvent the wheel – the evaluation toolkit is a finite collection of instruments; What are the best tools to use? Look first at the tools and practices which already exist in your institution and how they can be adopted, adapted and tweaked to fit; Those practices that don’t exist – but should because they worked for you: how do you get them into the system?

Activity 1 Practical activity 1: - Experiences to date of evaluating your STEM project? What supports you in this? - What STEM project evaluation would you like to do? What’s stopping you?

Practical activity 2: - Finding the links between your STEM project evaluation and your organisation’s ways and means of knowing itself (i.e. institutional research): - Where are there existing institutional mechanisms and practices that can help you? - How will your project level good evaluative practice help inform what institutional research should be?

Recommendations for project evaluations: To align oneself with established institutional data capture mechanisms that work, to aim to improve others; To use existing data banks (be they quantitative or qualitative) to good effect – to consciously know what we ‘know’ and make it the starting point; To be creative and imaginative in the way that we create new data capture events – its less onerous! To re-consider the relationship of researcher and researched: to rethink who the stakeholders are; To revisit the research approaches that are used – from monitoring KPI to appreciative inquiry.....