Title: Validating a theoretical framework for describing computer programming processes 29 November 2017

Research problem How do we describe the process by which people write computer programs?  Important in order to understand how teachers might better support the development of computer programming capabilities? Activity Theory is extensively used framework for analysing how activities are undertaken within social constructivist contexts. Subject pairs of students working together, Object is the computer program, the tools (instruments) that can be used by subjects to develop the program are computers, integrated development environments, etc. To better understand teachers might change things to improve the decision process, and give us the way to impact what teachers will do to affect what students will do Limited studies on how people write computer programs/investigate the computer programming process. Figure 1. The elements of an activity system and their interrelationships (Engeström, 1987)

Computational thinking & notional machine Computational Thinking: Solving problems, designing systems, and understanding human behaviour, by drawing on the concepts fundamental to computer science (Wing, 2006). Notional Machine: The notional machine is an abstract version of the computer, “an idealised, conceptual computer whose properties are implied by the constructs in the programming language employed” (du Boulay, et al., 1989). A model of the computer as it relates to executing programs These concepts can be used to theoretically ground how people go about performing computer programming process. Notional machine OFFICE | FACULTY | DEPARTMENT

Methodology Data was collected from 10 pairs of students completing a scratch programming activity. Scratch is a visual programming language Subjects 5 pairs of pre-service teachers with little experience 5 pairs of third year computing students Program a story game where a hero has to overcome a challenge in order to defeat the villain(s). Each pair spent approximately 40 minutes to undertake the task. Both finished the IQ tests; Did the Computational thinking tests (similar CT scores) ; they did not have any prior programming knowledge; decent scratch products

Research model Thematic analysis using an informed grounded theory approach used, starting with an initial framework. Computational thinking 17 categories and we had the definitions of the categories and we made changes via the entire process Notional machine

Data analysis – Thematic analysis Dual coding by two researchers Nvivo 11 to conduct the thematic analysis Two raters coded the first two transcripts independently to identify areas of coding discrepancy. After each of the two videos was coded the pair met to analyse the reasons for the differences and agree upon a category. The inter-rater reliability improved from the first to the second video, leaving the team with a sense of conceptual agreement about the meaning and boundaries of the categories. Following this, the final X videos were coded by one team member, who flagged potential utterances where coding could potentially have been ambiguous. The second coder was then consulted on each of these, and the pair formed a consensus about the most appropriate category.

Overall Results (1)- Refinement of coding scheme – Final framework Three categories were merged into others because they were not observed in majority pairs (Reconstruction Requirements checking; Decoding  Evaluation; Technology  Help seeking) Percentages of each category observed shown below. Computational thinking Task specification (0.2%) Requirements checking/Reconstruction (0.3%) Deconstruction (1.4%) Design (11.1%) Top 5 categories highlighted in red Evaluating/Decoding (3.5%)   Encoding (15.3%) Help seeking/Technology (0.8%) Organising (1.0%) Off topic (1.2%) Unclassifiable (3.3%) Reflections on learning (1.4%) Notional machine Implementation (47.0%) Testing (9.2%) Debugging (4.5%)

Over all results (2) – Some significant differences between computing and education students Chi-square test: = 154.47, df = 13, p-value < 2.2e-16 The categorisations are dependent on different disciplines (e.g. education students or computing students) Significant difference between the expected and observed frequencies

Overall results (3) Computing students made relatively greater focus on deconstructing problem. Computing students made relatively greater focus on evaluating program. Education students made relatively greater focus on help seeking. Computing students made relatively greater focus on implementation. Education students made relatively greater focus on reflections on learning. Organising was not significant (p=0.0067) but very low value may indicate that with larger sample education pairs tended to be more collaborative. 1. Think more about the relationship between tasks and design. 2. Computing students are more skilful to evaluate the program. 3. Education students are less knowledgeable on how to use Scratch hence they have to seek help, like watching the tutorials or asking others. 4. Education students did more self-reflections exercises in their discipline. OFFICE | FACULTY | DEPARTMENT

Summary and future work Project has empirically validated a theoretical framework for describing computer programming processes and shows, using a single contrast, how it can be used to perform educational analysis. Further research could investigate how programming process differs for: Different tasks (e.g. more complex specifications) Different languages (e.g. C++, Python, Blockly) Different programming environment (e.g. visual vs. text interfaces) Different Cohorts (e.g. children, experts, gender) Different teacher interventions (e.g. forms of scaffolding and modelling)

Thank you! Q&A