Introduction to CEM and Computer Adaptive Assessments Belfast, March 6th 2013 Neil Defty Business & Development Manager CEM Neil@cem.dur.ac.uk

Who / What is CEM ?

Centre for Evaluation & Monitoring Not for Profit Organisation Part of Durham University Close links to School of Education Established 1983 Nursery / Reception  Post-16 Monitoring Systems, Research and Evaluation Projects UK & International Over 1/3 of UK Secondary & Post-16 establishments Informed by Evidence from Research

CEM Provides Information Systems from Nursery to Post-16 Student Guidance & Chances Performance Monitoring Evaluation & Research Projects Entrance Testing Support – Phone, email, documentation Training INSET Roadshows Conferences (including exhibitions)

The Systems

The Systems Alis Yellis MidYIS A / AS / Btec / IB etc Year 13 Year 12 GCSE Computer Adaptive Baseline Test Year 13 Year 12 GCSE Year 11 Yellis Computer Adaptive Baseline Test Year 10 KS3 Combines curriculum tests with developed ability INSIGHT Year 9 MidYIS Computer Adaptive Baseline Test Year 8 Year 7

Early Testing & Predictions Typical Timeline Testing is available from June at end of previous academic year e.g. Y6 / Y9 / Y11 Early Testing & Predictions The baseline score is measured as early as possible and the information sent to CEM The institution then receives feedback from CEM: baseline test results and predictions The public examination results are sent to CEM when received by the institution The value added data is available from September via the Institution’s secure website at CEM

Basic Principles: Measure Student’s ability Intake Profiles IPR - Strengths and Weaknesses Project future performance from national trends Predictions Record Final Grade Compare actual performance to Projected Performance Value-Added Reports

Baseline Assessment and the Computer Adaptive Baseline Test (CABT)

Baseline Assessment Year 7 (8) Year 9 Year 10 (11) Year 12 (13+) Computer Adaptive Baseline Test Vocabulary Maths Non Verbal

What is an Adaptive test ? All questions allocated to groups of different difficulty All students in a year group get common difficulty starting question If answer correct, next question harder If answer wrong, next question easier Test looks at all answers in a section and homes in on questions of a suitable difficulty for student Try it yourself at www.intuproject.org/demos

Adaptive approach Low Average High

X          13

Standardisation Test scores are standardised; Mean = 100, SD = 15 Standardised Score National Percentage Comment >130 Top 2.5% Traditional classification of ‘mentally gifted’ >120 Top 10% >100 Top 50% <80 Bottom 10% <70 Bottom 2.5% Potential special educational needs?

The Computer Adaptive Baseline Test: Adaptive components common to MidYIS, Yellis, and Alis are: Vocabulary Maths Non- adaptive sections: Non-verbal (MidYIS, Yellis, Alis) Skills (MidYIS only) Assessments are ‘curriculum free’: i.e. students are not taught prior to the test Measure of acquired ability There is no preparation for the test The vocabulary and maths sections are the most important to the prediction of all subjects. Maths most difficult section to be curriculum free Non verbal is suitable for pupils for English is a second or additional language – allows ability to be displayed without the confounds of language No preparation for the test – this is important to ensure that every pupil starts from the same fair point. No lesson of preparation – each pupil hears the same examples, instructions and receives the same amount of help. This standardised administration ensures that you get a measure of typical performance for each pupil an indication of how much work you will need to put into take each pupil through to the next stage of education – through KS3 and onto GCSE. Developed ability is a set of acquired skills For example you are born with an IQ but your environment is so important. Yes you are born with a particular set of genes but you can’t read – you have to learn how to, you’re born with NV ability, but you can’t you can’t solve non verbal puzzles – you have to learn how to answer them. 15

Difficult vocabulary Vocab - Closely related to reading and English – a proxy – pupils need to be able communicate ideas across the curriculum areas Vocab is caught not taught – absorbed from the world around us. On its own is an excellent predictor for later academic achievement. Contributes well to the prediction of English History Foreign Languages Choose word or phrase with most similar meaning from those shown on right. How is vocab relevant to my subject? 16

Difficult maths x/4 + 2 And a hard question – could you do this? Seems tricky – especially for 11 year olds! However 1 in 15 pupils are getting this right! Good maths skills will mean that any areas of the curriculum that require numeracy - the pupil to think logically, perform calculations, reading data Maths – here’s an item of medium difficulty. Good predictor of Maths# Stats ICT D&T Economics 17

Non-Verbal: Cross-sections Non verbal – 3D visualisation, spatial awareness, pattern recognition. NV consists of 3 sections – crss sections, block counting, pictures. A low score here will mean that students will struggle will any elements of the curriculum that needs them to be able to think in 3 dimensions – for example geography, Art Drama Maths Science D & T Solid been cut with plane to form a cross section. 18

Non-Verbal: Blocks 3 Piles of blocks number of small and large blocks 19

Non-Verbal: Pictures Pictures 20 Pictures – addition, subtraction, sequences of pictures. Low scores in Non verbal – anecdotally number of times PH has been told on INSET that low NV corresponds to observed problems in D&T. 20

Skills: Proof Reading 21 Skills = PR & PSA – skills for modern world Proof reading - Essay writing 21

Skills: Perceptual Speed and Accuracy PSA – skimming & scanning skills. Decoding skills – can pupils read and understand the question? Low skills relate to under performance – not checking work, not able to work quickly (related to speed of work in class), not spotting errors In PH’s school he found that pupils with a D band in vocab and a D band in Skills on many occasions turned to be pupils who were dyslexic – either already diagnosed or diagnosed following further testing. PSA - related to speed of work in class High ability who is a slow worker, - ma doubt thaier ability because they don’t get wok finished Interesting to look at pupils who have low skills scores overall but the PSA element is low – here you will get pupils who do not take care intheir work – ie don’t check it or spot mistakes but who will rush. You may blame their incorrect work on their rushing but their low skills score means they probably would have got it wrong anyway! ie low Skills – rushing to get get work finished, getting things wrong 22

Alis - Patterns PSA – skimming & scanning skills. Decoding skills – can pupils read and understand the question? Low skills relate to under performance – not checking work, not able to work quickly (related to speed of work in class), not spotting errors In PH’s school he found that pupils with a D band in vocab and a D band in Skills on many occasions turned to be pupils who were dyslexic – either already diagnosed or diagnosed following further testing. PSA - related to speed of work in class High ability who is a slow worker, - ma doubt thaier ability because they don’t get wok finished Interesting to look at pupils who have low skills scores overall but the PSA element is low – here you will get pupils who do not take care intheir work – ie don’t check it or spot mistakes but who will rush. You may blame their incorrect work on their rushing but their low skills score means they probably would have got it wrong anyway! ie low Skills – rushing to get get work finished, getting things wrong 23

Alis – Logical Reasoning PSA – skimming & scanning skills. Decoding skills – can pupils read and understand the question? Low skills relate to under performance – not checking work, not able to work quickly (related to speed of work in class), not spotting errors In PH’s school he found that pupils with a D band in vocab and a D band in Skills on many occasions turned to be pupils who were dyslexic – either already diagnosed or diagnosed following further testing. PSA - related to speed of work in class High ability who is a slow worker, - ma doubt thaier ability because they don’t get wok finished Interesting to look at pupils who have low skills scores overall but the PSA element is low – here you will get pupils who do not take care intheir work – ie don’t check it or spot mistakes but who will rush. You may blame their incorrect work on their rushing but their low skills score means they probably would have got it wrong anyway! ie low Skills – rushing to get get work finished, getting things wrong 24

INSIGHT Curriculum-based assessments covering: Maths - Number & Algebra - Handling Data - Space, Shape & Measurement Science - Biology, Chemistry, Physics, - Attitudes to Science Reading - Speed Reading - Text Comprehension - Passage Comprehension Additionally for full INSIGHT: Developed Ability Vocabulary Non verbal, Skills Attitudinal measures

INSIGHT - Maths 7

INSIGHT - Maths

INSIGHT - Science Plus Attitude to Science: Perception of Science, Relevance to Self, Environmental Actions, Enjoyment of Science

INSIGHT – Reading (Text Comprehension)

INSIGHT – Reading (Passage Comprehension)

INSIGHT – Reading (Speed Reading)

INSIGHT – Developed Ability