How Data and Analysis techniques inform effective Teaching and Learning ■ Powerful things that schools can now do with their data Mike Bostock, New Media Learning

Should a school ever be surprised by its Ofsted report? Question for discussion Should a school ever be surprised by its Ofsted report? Consensus view of group: A school that knows itself well would not be surprised by the findings of a school inspection

“Satisfactory was no longer good enough” – HMCI Gilbert Are schools doing better or worse than they used to? 11% outstanding, 48% good, 34% satisfactory, 8% inadequate - Ofsted Annual Report Nearly 60% of schools are good or better “Satisfactory was no longer good enough” – HMCI Gilbert Only 50% of schools are average! “Half of schools are failing” - Guardian October 06 Headline figures about ‘standards’ are not always helpful

The data used to judge schools RAISEonline (ex-PANDA) data will form the basis of the main judgments about a school’s standards because: School inspections are too brief to add much to these judgements A school’s own self-evaluation evidence provided in the SEF tends to be descriptive rather than quantitative e.g. “We do lots of lesson observations”

Issues to do with using National Data to judge schools It can come too late in the year to be useful rather than simply judgemental Ofsted global figures are good for Government stats but less useful for school improvement Contextual Value Added adjustments are not a precise way to compare one school with another

CVA Contextual Value Added adjustment discounts one or more variables (ethnicity, gender etc) so that comparisons can be made on a fairer basis but… It is a statistical technique that really only works with large sample sizes. To make CVA accurate enough to make judgements between one school and another it would need to be refined by discounting every factor known to affect a pupil’s performance It has no meaning at class level. A good teacher will aim to do the best for every pupil - and not expect some individual pupils to do less well because of their ethnicity or gender etc.

CVA Where inspectors say that CVA data shows the school should be doing better, what independent, pupil-level source of measures can a school use to put this judgement into context? A growing number of schools are able to provide secure alternative evidence arising from their own pupil-level analysis and research

How would school self-analysis complement RAISEonline data? School level analysis available Spring Term the following year available the day the exam results arrive inspection could be based on information that is a year out of date inspection would be based on the most up-to-date performance information ownership and control in the hands of Government ownership and control in the hands of schools offers an external evaluation of a school’s performance offers teachers an opportunity to make their own evaluation evaluates pupils’ performance with national contextual data evaluates pupils’ performance with school contextual data forms the basis of Ofsted judgments on the school provides measures to support the school's own view of its performance judges leadership on basis of CVA comparisons with other schools allows a school to show that good leadership equates with successfully tackling negative variation

Within School Variation (WSV) There is over 4 times the variation in provision within schools than between schools - source OECD Reducing negative variation would raise standards by 10% - source DfES "We have always know that there is a difference in performance between schools. But what can make a bigger difference is the experience that children have within one school. So a child can do really well in one subject and not do well in another subject. And that can make an even bigger difference to children's life chances than differences between schools." - Jane Creasy, Assistant Director of Research, NCSL

High Achievement and Low Variation What is the better basis on which school leadership should be judged? High Standards High Achievement and Low Variation 5 A*-Cs How well every pupil achieves Some Children Matter Every Child Matters Headline figures can hide significant pockets of underachievement Headline figures should show how well every child achieves Good Leadership = high attainment Good Leadership = evidence of doing the best for every pupil League table position Measures of Within School Variation Performance analysis is done by the few and passed to the many All teachers are involved with analysing their pupils’ performance Judged by official data Secure school-level evidence of effectiveness School Improvement = more 5A-Cs School Improvement = less negative variation Ofsted Global measures are useful to know where to target the Education budget but not much use to schools

“Schools that are proactive in showing inspectors the evidence of their own pupil-level analysis and research tend to do better in their inspection.” - Dr. Mike Treadaway, Fischer Family Trust, Naace ‘Making Information Work’ Conference 27.04.07

Jack is a C level pupil with an F in this subject – should Jack or his teacher try harder?

An analysis of variation can shine a light on the achievement of different groups of pupils http://www.4matrix.org/example

The Data Confidence Toolkit The use of pupil-level data analysis and research is relatively underdeveloped at this point in time Good schools will be those that can show that they know themselves well, are targeting negative variation, and can show evidence of improvement “The Data Confidence Toolkit is a good starting point for schools” - Dr Mike Treadaway http://www.4matrix.org/toolkit/

Question: Should every school be expected to prove that for them ‘Every Child Matters’ - rather than just those pupils who will get 5 A*-C grades?