Quality and the learning environment: what can our evidence tell us? Jessica Lichtenstein Head of Quality Assurance

Current QA framework Our QA Framework shows how we monitor and check to make sure the standards are being met. However, one area of concern is the GOOD PRACTICE, and I’ll come on to that in a moment.

How does evidence inform our QA? Set standards for educational environments and implementation of curricula & assessment Use intelligence to screen medical education providers and check standards are met Identify risks and issues and prioritise them Make interventions and take targeted action locally (GMC or other regulators) Use intelligence to monitor impact of interventions and actions Identify risks and issues and prioritise them

NTS results and learning environments There is a report like this for every training post in the UK That’s over 5,000 reports So how do we decide where to focus our attention? and 225,000 indicator outcomes

Learning environments – what’s currently bad? To find out where the worst places are right now, we look for the training posts with… the most red flags This is converted into a proportion to allow fair comparison with different posts We look at low means as well as flags to double check the worst performing places are picked up the lowest mean scores

Learning environments – what’s been bad for a while? To find the places with ongoing problems we look at training posts with… the most triple reds

Learning environments – what might be bad in the future? To find the places where things might get worse in the future we look at training posts with… the most big drops in mean score This allows us to detect movement in within the ‘white’ flags which can otherwise hide negative changes

Triage - rankings These measures are then all converted to rankings and given a percentile score: Current red ranking: 89 Low score ranking: 90 Triple red ranking: 97 Change in score ranking: 00 We currently refer any training post with a score of 100 in any of these measures to Visits and Monitoring for further scrutiny

What does this look like?

What do we do with this? We discussed organisations on this list with HEE/deaneries and compared evidence We agreed status of each organisation e.g. : local quality team to investigate issues confirm evidence so escalate to enhanced monitoring circumstances mean that results don’t warrant further investigation Monitor appropriately

Future development Currently using only NTS post specialty by site reports- we think this tells us the most about a singular learning environment Future plans: Add trainer survey and progression report outputs Develop views of programme-level results Positive results & significant improvements

Questions??