CAA Thinking Skills Tests for University Admission Assisting Admissions Decisions Robert Harding Director TSAT Project and ITAL Unit Interactive Technologies in Assessment and Learning

CAA Thinking Skills Assessment : Recent history in Cambridge  Need for additional information to select applicants  BMAT: BioMedical Admissions Test: now used also by Oxford and UCL, Royal Vet College  About 4000 candidates in 2003, expecting 6000 in 2004  TSA numbers (used in some Cambridge Colleges):  289 in 2001  478 in 2002  Over 1500 in 2003  There is evidence that in the Cambridge context the test is a better predictor of performance than either A-Levels or interview scores.

CAA Some admissions tests in ROW  The United States Medical Licensing Examination (USMLE)  The Law School Admission Test (LSAT)  Graduate Medical School Admissions Test (GAMSAT)  “The SAT is a three-hour test that measures verbal and mathematical reasoning skills students have developed over time and skills they need to be successful academically.”

CAA BMAT : what is tested?  Section 1 Thinking Skills: 40 multiple choice questions, 1 hour  Section 2: Science and Maths core knowledge: 30 questions, 30 mins  Section 3 essay: capacity to develop ideas and to communicate them effectively in writing 30 mins.

CAA Thinking Skills Assessment : what is tested?  Critical Thinking and Problem Solving  50 multiple choice questions, online or on paper  One ‘paper’: test lasts 90 minutes  Origins in academic research around mid 1980’s  MENO 1993/4 tested six ‘thinking skills’  Today:  OCR’s AS Level in Critical Thinking  CIE’s AS Level in Thinking Skills  Evidence that these skills are highly relevant to successful study.

CAA Thinking Skills Assessment : Is it useful? “The TSA Project has been running on a pilot basis since 2001 in Cambridge. Its results are used carefully and conditionally by Colleges as additional information and never replace any traditional criteria on which admission decisions are based. More research and evaluation is being done in this pilot project to establish its value for admissions purposes. Although this research is not yet complete, the use of the TSA test within Cambridge has grown each year as confidence in its usefulness grows.”

CAA Thinking Skills Assessment : what does this do for ‘access’?  Provides a uniform measure of readiness for Higher Education  Can be self-tested; encourage applicants  Low cost support materials widely available, eg via www  Worth learning these skills  Facilitates access research.

Data for 48 candidates who took the TSA in December 2001 and their first University examination (the Tripos) in the summer of 2003, all Computer Science.

2001/2003 data for 48 candidates, showing the top 12 TSA scorers and the bottom 12 TSA scorers.

2001/2003 data for 48 candidates, but Tripos marks replaced by randomised values.

Scatter plot of 2002 TSA score vs 2004 Tripos score: 478 candidates took the TSA, 83 went on to 1st year exams in Natural Sciences, Computer Science, and Mathematics courses.

TSA 2003 total scores - distribution

TSA 2003 scores distribution Critical ThinkingProblem solving

CAA What’s the CAA perspective?  Whole process is computer-assisted:  Item banking & test construction  Administration and registration  Answer sheet scanning (ICR)  Results return and reporting  System developed by Cambridge’s CARET (Centre for Applied Research in Educational Technology)  To be made Open Source, QTI compliant  MCQ’s, so can be taken on paper or on-line: Can compare on-line & paper performance.

CAA What’s the CAA perspective?  Whole process is computer-assisted:  Item banking & test construction  Administration and registration  Answer sheet scanning (ICR)  Results return and reporting  System developed by Cambridge’s CARET (Centre for Applied Research in Educational Technology)  To be made Open Source, QTI compliant  MCQ’s, so can be taken on paper or on-line: Can compare on-line & paper performance.

Delivery Mode Score typeNMeanStd Dev PaperTotal Prob. Solv Crit. Think OnlineTotal Prob. Solv Crit. Think

CAA TSAT in 2004 and Beyond? yUCLES is planning to continue pilot testing the TSA in Cambridge, on a similar basis to 2003 yThe BMAT continues in 2004 with additional universities yWe will continue to consult interested parties and stakeholders yWe will continue to develop the technology yWe may look for interest from other Universities, and will consider overseas testing yWe will look at issues of timing and location yWe will continue to research admissions issues.

CAA  Some URL’s: Thinking Skills Admissions Testing Acknowledgements and thanks … To many colleagues in UCLES and to our partners in CARET who played very full parts in this work.