The excellent practitioner who once failed to be excellent (and other short stories) Dr Liz Austen Senior Lecturer in Research, Evaluation and Student Engagement (STEER) l.austen@shu.ac.uk @lizaustenbooth SHU LTA Conference June 2017

Dawson & Dawson (2017) Reporting bias in L&T research the impact on an 'excellence identity' the skew of L&T funding bodies to request examples of good/excellent practice institutional funding for interventions varied methodological practices power dynamics (students as partners literature)

Activity In small groups, discuss the vignettes and debate your response to the question.

What impact have these publication habits had on the scholarship of learning and teaching? Academic A teaches human anatomy at a university. As an innovative, scholarly teacher she often trials new teaching approaches, and writes them up for journal publication if they are successful. When an innovation fails, Academic A chalks it up to experience and reflects with close colleagues. But she doesn’t publish when an approach doesn’t work – sharing would only bring the faculty into disrepute, and she’s busy enough with her scientific research.

At what point is Academic B engaged in an act of research? Academic B teaches sociology at a university. Over the past five semesters he has redesigned his assessment several times to incorporate what he has been reading in the literature: authentic assessment; frequent low-stakes assessment; feedback loops; and other new and exciting approaches. Last semester’s assessment results were excellent, which prompted Academic B to apply for retrospective research ethics approval to study the work previously produced by his students.

How are unintended outcomes measured and reported? Academic C is an aspiring learning and teaching researcher who is in the middle of leading a small government-funded learning and teaching grant to improve critical thinking. Students love the pedagogical approach he developed, but to his surprise show no significant difference in critical thinking. Embarrassed the funder may view this as failure, Academic C focuses his publication and reporting efforts on sharing how this program improves the student experience – without mentioning critical thinking.

What impact is the reward and recognition of 'excellence' having on reporting bias? Academic D teaches a final-year linguistics subject with 1000 students. She plans to apply for an award for her excellent teaching, and knows her case will be strengthened if she publishes about her excellent teaching. Academic D is a self-confessed data geek who has kept all of the logs from her Learning Management System site for 2010–2015. She runs a dozen different statistical tests, and finds if she restricts her analyses to just the data from 2012 there are significant associations. She focuses in on that data, eventually publishing a paper on that year’s results.

What are the risks of honest accounts of failure? STEER is a central directorate working to address SHU strategic priorities concerning the student experience. Working alongside a national, HEFCE Catalyst funded project team (REACT), the research team designed co-design and peer learning interventions to make positive differences to the confidence levels of BME students and enhance longer-term belonging. The team, including student researchers, approached a range of stakeholders from across the institution, but failed to secure the necessary support for developing these interventions. The project was therefore unsuccessful in meeting the initial project objectives. The research team decided to openly discuss this outcome, and the unintended benefits of the process, at internal and external events. The team also published articles on the notion of 'institutional readiness' in an honest and transparent way.

Methodological reasons for 'failure' small sample sizes (statistical significance) limited methodological innovation lack of ethical approval limits methodology and dissemination difficulties in measuring unintended outcomes Reframing failure as misjudgement Storify evidence trail of support questionnaires measure inferences and reconstructions of learning - new and innovative measures needed (

Reflections Can we constructing a series of institutional principles for designing, producing and sharing methodologically sound L&T research and evaluation? What support and development needs are apparent? (Requests will be collated by the facilitators in order to design bespoke development sessions across the institution)

Extra vignette (if you are really keen....) D is a professional staff member who operates a university-wide co-curricular Peer Assisted Learning program. The program – including Richard’s salary – is funded by strategic initiative funding. At the end of each semester Richard generates reports on the effect of his program on student marks and retention. Last year’s results were so poor he was worried the program might lose funding, but this year’s results are so positive that his manager has encouraged him to write them up for journal publication.