evidence-based web-portfolios project e-scape evidence-based web-portfolios - to empower learning - for reliable assessment Richard Kimbell TERU: Goldsmiths University of London Karim Derrick TAG developments in association with …. 1

e-scape system enables us to support and capture live performance • performance assessment • on task • in real time science to display the performance in web-portfolios • easily distributed • URL / log in / password geography to assess performance • for GCSE / Diploma • linked to Awarding Body systems • with very high reliability (0.95) design

capturing performance evidence data and notes mind-maps sketching (and collaborating) audio reflections photos of the state of play video presentation

client/user can choose the degree of control continuum of control • tight control of tasks • strict timings • controlled response modes • standardised portfolio • learner autonomy of tasks • flexible timings • learner choice of tools • customised portfolio formal examinations extended coursework projects client/user can choose the degree of control

the portfolio as a ‘big-picture’ of the activity … that can be dipped into for the detail 5

the assessment challenge

teachers / examiners review portfolios with ‘RED PEN’ different markers have separate ‘layers’ teacher speaks directly to the student teachers notes on student portfolio criteria displayed with the work and evidence Is noted 7

but there is another (quite different) possibility L.Thurstone (1927) • law of comparative judgement • judges’ personal standards cancel out A.Pollitt (2004) • comparative pairs methodology • for reliability studies between exam boards e-scape (2007 & 2009)‏ • the 1st and 2nd complete data sets • the ‘pairs engine’ • for front line assessment 8

eye testing

teachers and assessment teachers are good at • relative judgements • ranking because their judging standard cancels out teachers are not good at • absolute judgements • scoring because ‘criteria’ become personalised

…but then they make a relative holistic judgement • teachers discuss criteria • teachers use them as ‘signposts’ to look into the portfolios • teachers identify strengths and weaknesses of portfolios …but then they make a relative holistic judgement 11

portfolio A portfolio B the ‘pairs engine’ manages the assessment process portfolio A portfolio B

the whole process is run on-line in the ‘pairs engine’ • judges in Ireland, Israel and across the UK • shared viewing of exemplar work • on-line marker training

the pairs engine dynamically generates the rank what emerges is the collective professional consensus of an expert group of judges

} 2009 sample 350 pupils awarding grades is a separate process multiple judges (28)‏ multiple comparisons (17+)‏ reliability coefficient = 0.95

“ Verbal and Mathematical Reasoning tests “ Verbal and Mathematical Reasoning tests .. were developed for 11+ and similar purposes from about 1926 to 1976. These were designed to achieve very high reliability, by minimising inter-marker variability and maximising the homogeneity of the items that made them up. KR20 statistics for those were always between 0.94 and 0.96. With e-scape, a level of internal consistency comparable to those reasoning tests has been achieved without reducing the test to a series of objective items. “ Alastair Pollitt: Dec 2008

identifying problems the engine identifies portfolios where judges disagree and individual judge profiles show their consensuality or ‘misfit’ 17

reducing standard error • the first 10 judgements do most of the work • the second 10 fix the scale • thereafter the gain in accuracy per extra judgement is small 18

how long do judgements take? median: • for all 28 judges • 130 judgements each = 4 mins 6 secs = total of 8 hrs 45 mins

The variety of media used within the portfolio seems to give everyone a fair chance to display their design thinking regardless of their ability. It was fairly easy to see that there was a range of performance evident in the portfolios. Judges’ reflections ….. the learner voice emerges much, much faster .. and less scary Portfolios displayed in this way have a huge advantage in that the big picture can be seen immediately. It’s very easy to get a feel for the project ... The ability to dip in and out of the different sections enabled me to reinforce my holistic mental picture of the project. Surprisingly pupils’ attitudes were also often evident. • would encourage learners’ understanding of capability • would encourage learning through critique • could support e.g. PLTs evidence It’s brilliant not having to carry around boxes full of portfolios/folders and practical work.

current implementations

new applications for e-scape & ‘pairs’ Edexcel Examination Board: UK 14-19 Diploma in ‘Creative & Media’ West Australia Curriculum Council school leaving certificate examinations (engineering / applied IT / phy ed / LOTE ) Limerick University: Ireland engineering design project (mobiles) + assessment UK government agency writing assessments at age 11 Scotland SQA / LTS formative assessment in transition years (ege 11-13) Bath Spa University & AstraZeneca Science Teaching Trust science/technology for age 11-13

e-scape references and citations by authors outside the e-scape team

project e-scape Contact me at: r.kimbell@gold.ac.uk e-solutions for creative assessment in portfolio environments Contact me at: r.kimbell@gold.ac.uk Goldsmiths University of London www.gold.ac.uk/teru 24