Validating a theoretical framework for analysing computer programming learning processes Presentation to the ACARA Digital Technologies National Working Group 6 December 2017 Matt Bower, Jens Siemon, Garry Falloon, Jennifer Lai

How do we know… Playing with robots and online programming environments can be fun, but how do we know if it is developing students’ computational thinking capabilities? What pedagogical strategies actually help students to become better computer programmers and problem solvers using technology?

Research problem How do we describe and analyse the process by which people learn to write computer programs?  Important in order to understand how teachers might better support the development of computer programming capabilities? Activity Theory has been used as framework for analysing how activities are undertaken within social constructivist contexts Subject pairs of students working together, Object is the computer program, the tools (instruments) that can be used by subjects to develop the program are computers, integrated development environments, etc. Activity theory does not encompass any of the specific computer programming processes that people undertake To better understand teachers might change things to improve the decision process, and give us the way to impact what teachers will do to affect what students will do Limited studies on how people write computer programs/investigate the computer programming process. Figure 1. The elements of an activity system and their interrelationships (Engeström, 1987)

Computational thinking & notional machine Computational Thinking: Solving problems, designing systems, and understanding human behaviour, by drawing on the concepts fundamental to computer science (Wing, 2006) Notional Machine: The notional machine is an abstract version of the computer, “an idealised, conceptual computer whose properties are implied by the constructs in the programming language employed” (du Boulay, et al., 1989) These concepts can be used to theoretically ground how people go about performing computer programming process Theory should represent reality - model requires empirical validation Notional machine OFFICE | FACULTY | DEPARTMENT

Research model Thematic analysis using an informed grounded theory approach, starting with an initial framework. Task specification Computational thinking Requirements checking Reconstruction Deconstruction Help seeking Technology Organising Off topic Unclassifiable Reflections on learning Design 17 categories and we had the definitions of the categories and we made changes via the entire process Evaluating Decoding Encoding Notional machine Implementation Testing Debugging

Methodology Data was collected from 10 pairs of students completing a scratch programming activity: 5 pairs of pre-service teachers with little experience 5 pairs of third year computing students Program a story game where a hero has to overcome a challenge in order to defeat the villain(s). Each pair spent approximately 40 minutes to undertake the task. Both finished the IQ tests; Did the Computational thinking tests (similar CT scores) ; they did not have any prior programming knowledge; decent scratch products

Data analysis – Thematic analysis DUAL CODING BY TWO RESEARCHERS Nvivo 11 to conduct the thematic analysis Two raters coded the first two transcripts independently to identify areas of coding discrepancy. After each of the two videos was coded the pair met to analyse the reasons for the differences and agree upon a category. The inter-rater reliability improved from the first to the second video, leaving the team with a sense of conceptual agreement about the meaning and boundaries of the categories. Following this, the final X videos were coded by one team member, who flagged potential utterances where coding could potentially have been ambiguous. The second coder was then consulted on each of these, and the pair formed a consensus about the most appropriate category.

Refined coding scheme and final results Three categories were merged into others because they were not observed in majority pairs (Reconstruction Requirements checking; Decoding  Evaluation; Technology  Help seeking) Percentages of each category observed shown below. Task specification (0.2%) Computational thinking Requirements checking/Reconstruction (0.3%) Deconstruction (1.4%) Help seeking/Technology (0.8%) Organising (1.0%) Off topic (1.2%) Unclassifiable (3.3%) Reflections on learning (1.4%) Design (11.1%) Top 5 categories highlighted in red Evaluating/Decoding (3.5%)   Encoding (15.3%) Notional machine Implementation (47.0%) Testing (9.2%) Debugging (4.5%)

Some significant differences between computing and education students Chi-square test: = 154.47, df = 13, p-value < 2.2e-16 The categorisations are dependent on different disciplines (e.g. education students or computing students) Significant difference between the expected and observed frequencies

Framework was able to show that Computing students had relatively greater focus on deconstructing the problem Computing students made relatively greater focus on evaluating the program Education students made relatively greater focus on help seeking Computing students made relatively greater focus on implementation Education students made relatively greater focus on reflections on learning Organising was not significant (p=0.0067) but very low value may indicate that with larger sample education pairs tended to dedicate more attention to groupwork 1. Think more about the relationship between tasks and design. 2. Computing students are more skilful to evaluate the program. 3. Education students are less knowledgeable on how to use Scratch hence they have to seek help, like watching the tutorials or asking others. 4. Education students did more self-reflections exercises in their discipline. OFFICE | FACULTY | DEPARTMENT

Summary and future work Study has empirically validated a theoretical framework for describing computer programming processes and shows, using a single contrast, how it can be used to perform educational analysis Further research could investigate how programming process differs for: Different tasks (e.g. more complex specifications) Different languages (e.g. C++, Python, Blockly) Different programming environment (e.g. visual vs. text interfaces) Different Cohorts (e.g. children, experts, gender) Different teacher interventions (e.g. forms of scaffolding and modelling)

Thank you! Q&A Computational thinking Notional machine Task specification Computational thinking Requirements checking / Reconstruction Deconstruction Help seeking / Technology Organising Off topic Unclassifiable Reflections on learning Design Evaluating / Decoding Encoding Notional machine Implementation Testing Debugging