1. Investigation of student engagement with programming in TU100 The impact of using a graphical programming environment?
Helen Jefferis, Soraya Kouadri & Elaine Thomas
Computing & Communication Department

2. Agenda Motivation Project's aims Methodology Our findings
Summary & future work

3. Motivation
There is a strong need for the teaching of introductory programming at level 1 in the Computing and IT degree programme.
The majority of new OU students will not have experienced the new National Curriculum.
Previous teaching of programming at level 1 (M150) involved a text-based programming, JavaScript.
Over half of students  avoided answering the question on programming in the EMA.  
TU100 ‘My digital life’ uses a graphical programming environment Sense based on Scratch.
NB there seems to be a perception that students that aren’t good at programming will automatically be good at eg Networking

4. Project's aim
The aim of this eSTEeM project is to investigate the impact of using a graphical programming environment on student engagement with programming.
It will seek to address the fundamental question as to whether the visual programming environment actually engages novice programmers or not in ‘TU100’.

5. Methodology
Identification of the Sense programming questions in each TMA and in the EMA.
Identification  and collection of data related to the numbers or students who completed these questions and their overall performance.
Analysis of textual comments in a selection of SEaM surveys  of TU100 relating to students’ experience of programming.

6. Comparison of OES Scores
Presentation
No. of Students
OES
Sense
Rest
Mean
Median
13J
1340
70.66
74.40
71.32
80.33
70.52
74.52
14B
801
69.62
74.19
69.64
83.33
69.63
14J
1343
73.63
78.80
85.00
94.00
70.79
75.50
15B
767
73.19
78.00
84.10
96.00
70.46
74.50
15J
1234
70.03
74.80
75.14
86.00
68.75
73.00
16B
674
70.28
73.26
85.62
69.54
73.50
NB students that scored 0 on OES were excluded, some students were excluded where it proved impossible to ‘recalculate’ their OES score based on the OSCAR individual scores.
Not a great deal of difference, but the next slide shows this in graphical form, so perhaps more clearly.

7. Comparison of OES Scores
Red line shows number of students (right hand axis) – less on B presentations
Green is the OES Mean, Yellow the Sense Mean, Blue the ‘rest’ mean.
In 13J and 14B there were less marks allocated for Sense but the Average scores for the programming and non-programming elements were about the same. (The marks have been normalised so that these figures are showing the percentage of the available marks for each element)
In more recent presentations, students seem to have gained more marks, on average, in the Sense / programming elements of the EMA. NB the J presentation and the following B presentation have very similar questions – which tends to suggest that perhaps Sense was ‘easier’ on 14J / 15B?

8. Correlation between Sense and Non-programming scores.
Presentation
Correlation (In all cases p =0.00)
13J
.569
14B
.602
14J
.522
15B
.512
15J
.551
16B
.555
Looking that the scatter graph it is quite hard to see what is going on,
Sense scores increasing along the x – axis, the ‘rest’ along the y. There is a clear cluster of students in the top right – those that do well in both elements – a few in the bottom right who do well on Sense and poorly on the rest, and some in the top left who do badly on Sense and well on the rest – plus several who score zero on Sense (left hand vertical line) and 100% on Sense – (right hand vertical line)
NB presentations in different colours – with J circles and B triangles
However overall there is a strong (r=-0.543, p<0.01) correlation between the scores that students achieved in the programming and non-programming elements of the EMA. (and not a lot of difference in this by Presentation – see the table)

9. Are Students passing without passing Sense?
<7.5%
(460 out of 6,159 students) failed Sense and passed OES
This chart shows the students that scored less than 40%
The red column is the students that also failed OES
Out of 6,159 students in total only 460 ( ) passed OES – this is less that 7.5% of students.
So the simple answer is no!

10. Summary & Future Work Summary Future Work
More students have engaged with the programming element than on previous modules.
There is a strong correlation between the scores that students achieved in the programming and non-programming elements of the EMA.
There is little or no difference in the performance of students in the programming elements and the non-programming elements. 
Future Work
Analysis of SEaM surveys’ textual comments relating to students’ experience of programming.
Investigation of whether there is improved students engagement with programming comparing to M150