Eng I Fall 2010: Diagnostics & Results David Minugh English Department, SU May, 2011

Diagnostic Tests: Key aspects Intake hurdle or not: a basis for admission? If students are already accepted, is the test mandatory or not? Costs (no “not” alternative here!) Information to students or a placement test? Computerized or not? Reusable or not?

Intake Hurdle: A basis for admissions? If we limit intake to those with scores better than X, – We need a new test every time (or a large pool of items) – It needs to be legally acceptable (including student IDs) – What about our intake profile (broader recruitment?) – Teacher candidates will have to be treated separately – We need to have good validity – What about motivation and test-taking skills? – Time: a summer test?! – Retakes for those who were sick, abroad, etc.? – Is it relevant if we can’t afford to turn away students? – Current status: offered to those who actually register

Mandatory or not? Our paper-based test (up to 2006) was “mandatory,” – no sanctions for those who didn’t take it – identical from year to year (over a 30-year span) – make-ups were offered on an ad-hoc basis – dropped to save marking and room costs Our most recent (voluntary) net-based diagnostic test was taken by 139 of about 300 students – No way of checking which students took it – The process took several weeks The new, scheduled diagnostic test (Sept. 3) reached 300 of about 330 students – On their schedule (three 100-student waves), Fri 9-11 am, 11-1, 1-3 pm

Information or Placement Test? Any diagnostic test contains potential information for students – however, they may not know what it means – we may not know what it means, either – motivation and commitment are not easily and cheaply measured It could be used for group placements – SU groups are not differentiated: 10 regular + 2 future- teacher groups – Students choose groups by schedule or teacher or books; 11 different combinations of teachers Group placement implies non-recyclable tests

Costs Test-making: – One-time development cost for our test at SU – Higher for Q pools, much higher for new tests each semester – Oxford’s test (A1-C2 rating) runs about 20 cr/student – DiaLang (an EU project) is free, but now unsupported Room costs: regular exam halls v ones with computers – At SU the cost is the same (!) – TKR, c cr/hr for a 140-person hall + 5 computer- literate proctors, 3 waves – IT support costs and level of support? Marking: – Free for computerized tests – Student/admin markers for paper-based version – Admin time is at a premium when the semester starts

Computerized or Not? Volume: some 300 new students per semester Advantages: – no mountain of papers – instant feedback – easy administration of results and student contacts – students get into the VLE system quickly – error analysis (practical or research) Disadvantages: – test limited to electronic marking and relatively simplistic questions; very limited productive component – student unfamiliarity with the system – costs to install and run

Starting Points for SU 285 new English I students took our net-based diagnostic test on Sept. 3, 2010, at the start of the Fall Semester 2010 – A few others from other courses were excluded from this discussion The test was strictly proficiency-oriented: Grammar (60), vocabulary (60), collocations (20), phonemes (10), word stress (10) The maximum score was 160; actual scores ranged from 39 to 152, a bell curve peaking in the range (see next slide) Similar results were obtained in the Spring 2011 diagnostic test Those who had 80 or less received an individual about this, but were not called to mandatory counseling For the results shown here, all tests have been offered twice, and all papers graded, i.e. a full semester had been completed when the results were calculated (March 2011)

Eng I Diagnostic Test HT10

Credits Obtained (max 30) v. Diagnostic Results

Credits Achieved (max 30) and Diagnostic Results

Percentage Course Dropouts per Diagnostic Results (n = 25)

Course Drop-outs/Finishers per Diagnostic Results

Course Completers/Partials/ Drop-outs per Diagnostic Results

Percentage Course Completers/Partials/ Drop-outs per Diagnostic Results

Literature (12 cr): Completers/Partials/ Drop-outs per Diagnostic Results

% Literature (12 cr): Completers/Partials/ Drop-outs per Diagnostic Results

Linguistic Analysis (6cr): Completers/Partials/ Drop-outs per Diagnostic Results

% Linguistic Analysis (6 cr): Completers/Partials/ Drop- outs per Diagnostic Results

Linguistics Test (6 cr): Completers/Partials/ Drop-outs per Diagnostic Results

% Linguistics Test (6 cr): Completers/Partials/ Drop-outs per Diagnostic Results

Eng I Final Grade v Diagnostic Score

Tentative Conclusions Proficiency alone will not guarantee success Drop-outs (25  10%): – Half had scored < 80 on the diagnostic test – Almost 30% had scored above 100 – Another 18 took 0 credits (40% of them scored < 80) Results: – No student scoring under 72 finished within one semester – The first linguistic test (lectures based on an introductory linguistics textbook (Yule, The Study of Language) is the biggest stumbling block – Quite a few lack only one or two components – Very few A grades were granted, and no Es (this may change as further make-up papers and spring retakes come in) – Both the literary and the linguistic course seem accessible even to quite weak students (60% in the range passed the literary and/or linguistic course) – The diagnostic test is not a strong predictor for most students, but is relatively reliable at the lower and upper ends of the scale – The tipping point (50% chance of succeeding within a semester) appears to be around 100 on the diagnostic test