Programming Aptitude Testing Materials, Data analysis Saeed Dehnadi Middlesex University

I talk about: Test materials Results Collaborators Summary Questions Mental Models Answer Sheet Mark Sheet Interpretation Algorithms Results Middlesex University (initial test) - 2005 University of Newcastle (Australia) - 2006 York - 2006 Strathclyde - 2006 Middlesex University (second test) - 2006 Collaborators Summary

Questions 1 (single assignment)

(M1) Mental model “move value” a = b; The value of b is given to a and b changes its value to zero. a  b // b  0 Ans ( a = 20 , b = 0 ) 8th Answer

(M2) Mental model “copy value” a = b; The value of b is given to a and b keeps its original value. a  b // b unchanged Correct conception of LHS  RHS Ans ( a = 20 , b = 20 ) 4th Answer

(M3) Mental model “move value” a = b; The value of a is given to b and a changes its value to zero. b  a // a  0 Ans ( a = 0 , b = 10 ) 3rd Answer

(M4) Mental model “copy value” a = b; The value of a is given to b and a keeps its original value. b  a // a unchanged Ans ( a = 10 , b = 10 ) 1st Answer

Mental models (M5) (copy + add) value a = b; The sum of a and b is given to a, and b keeps its original value. a  (a + b) // b unchanged Ans ( a = 30 , b = 20 ) 2nd Answer

Mental models (M6) (move + add) value a = b; The sum of a and b is given to a, and b changes its value to zero. a  (a + b) // b  0 Ans ( a = 30 , b = 0 ) 10th Answer

Mental models (M7) (copy + add) value a = b; The sum of a and b is given to b, and a keeps its original value. b  (a + b) // a unchanged Ans ( a = 10 , b = 30 ) 9th Answer

Mental models (M8) (move + add) value a = b; The sum of a and b is given to b, and a changes its value to zero. b  (a + b) // a  0 Ans ( a = 0 , b = 30 ) 5th Answer

Mental models (M9) “no changes” a = b; a and b keep their original values. a unchanged // b unchanged Ans ( a = 10 , b = 20 ) 6th Answer

Mental models (M10) Equality a = b; Assignment is a simple equation, and then all equal values of a and b are acceptable. Ans ( a = 10 , b = 10 ) and Ans ( a = 20 , b = 20 ) both 1st and 4th answers

Mental models (M11) “Swap values” a = b; a and b swap their values simultaneously. a  b a gets b’s value b  a b gets a’s value Ans ( a = 20 , b = 10 ) 7th Answer

Models for a single assignment int a = 10; int b = 20; a = b;

Answer Sheet (Question1)

Questions 2 (single assignment)

Questions 3 (single assignment)

Question 4 (two assignments)

Mental model “M1 + Sequence” a = b; b = a; Model is M1 that applies sequentially through both statements: L1) a  b and b  0 then a = 20 and b = 0 L2) b  a and a  0 then b = 20 and a = 0 Single answer: a = 0 , b = 20

Mental model “M1 + Independent (M1 I)” a = b; b = a; Model is M1 that applies independently for each individual line. L1) a  b and b  0 L2) b  a and a  0 Multiple answers: a = 20 and b = 0 a = 0 and b = 10

Mental model “M1 + Simultaneous-single (M1 Ss)” a = b; b = a; M1 Ss Same as (M1 I), subjects only interested on Left-hand-side values of statements and ignores the right-hand-side values. L1) a  b and b  ignores L2) b  a and a  ignores Single answer: a = 20 , b = 10

Answer Sheet (question4)

Question 5 (two assignments)

Question 6 (two assignments)

Answer Sheet (question6)

Question 7 (three assignments)

Answer Sheet (question7)

Mark Sheet

Mark Sheet Interpretation (Tree Structure)

Mark Sheet Interpretation (Mode Structure)

Data sample

Data provided by: Saeed Dehnadi Middlesex University Barnet College First Experiment 2005 Data provided by: Saeed Dehnadi Middlesex University

Experiment Subjects: Time: Questions 30 students from Middlesex University Introductory to programming (Java) 31 students from Barnet College Further Education Programming Course Time: Week 0 of the course Questions 12 questions

Categories Week 0 Consistent (45%) Inconsistent (39%) Blank (16%)

Categories Week 3 Consistent (67%) Inconsistent (33%) Blank (0%)

Stable group membership (Week3) Data provided by: Saeed Dehnadi Middlesex University

Pass/Fail statistics (Initial Result) Data provided by: Saeed Dehnadi Middlesex University

Two populations (Initial Result)

University of Newcastle Australian Data University of Newcastle 2006 Data provided by: Simon School of DCIT (Design, Communication, and Information Technology)

Pass/Fail statistics (First glance) Data provided by: Simon The University of Newcastle

Pass/Fail statistics Exclude m2 Data provided by: Simon The University of Newcastle

Pass/Fail statistics Exclude m2 Data provided by: Simon The University of Newcastle

Data provided by: Linxiao Mu Strathclyde Data The University of Strathclyde 2006 Data provided by: Linxiao Mu

Data provided by: Linxiao Mu The University of Strathclyde Mental models in week 20 Value assignment Reference assignment Data provided by: Linxiao Mu The University of Strathclyde

Value assignment mental models in final exam P < = 0.05 Data provided by: Linxiao Mu The University of Strathclyde

Value assignment mental models in 4 in-course tests P < = 0.01 Data provided by: Linxiao Mu The University of Strathclyde

Reference assignment mental models in final exam P < = 0.025 Data provided by: Linxiao Mu The University of Strathclyde

Reference assignment mental models in 4 in-course tests P < = 0.05 Data provided by: Linxiao Mu The University of Strathclyde

Data provided by: Dimitar Kazakov York The University of York 2006 Data provided by: Dimitar Kazakov

Data provided by: Dimitar Kazakov The University of York Pass/Fail statistics P < = 0.001 Data provided by: Dimitar Kazakov The University of York

Data provided by: Saeed Dehnadi Middlesex University Second Experiment 2006 Data provided by: Saeed Dehnadi Middlesex University

Data provided by: Saeed Dehnadi Middlesex University Pass/Fail (Quiz 1) P < = 0.20 Data provided by: Saeed Dehnadi Middlesex University

Data provided by: Saeed Dehnadi Middlesex University Pass/Fail (Quiz 2) P < = 0.01 Data provided by: Saeed Dehnadi Middlesex University

Collaborators Ireland - School of Computing in DCU Australia - Charles Stuart University Mozambique - Instituto Superior de Transportes e Comunicações, Maputo UK- Bournemouth University (Royal School of Signals in Blandford) UK – University of Birmingham UK – University of Sheffield UK – King’s College, London Canada - University of Toronto Denmark – University of Aarhus USA – A-level in Computing, High school UK - A-level in Computing, North West College Germany – Computing study, teacher training Institution

Summary Originality We are the first to have an aptitude test that works. Objectivity We have objective test materials that can be used by others. Speculation Students who can handle meaningless rules learn programming more easily.

To be continue …..