Katrina Le, Hamid Tarmazdi, Rebecca Vivian, Katrina Falkner, Claudia Szabo and Nick Falkner Computer Science Education Research (CSER) Group Directing Teacher Focus in Computer Science Online Learning Environments

Increasing pressures of enrolments Figure: Commencing Domestic IT enrolments in Australia (2001-2015). http://blogs.adelaide.edu.au/cser/2017/02/15/a-look-at-it-and-engineering-enrolments-in-australia-updated/

Increasing pressures of enrolments We are seeing a resurgence in Computer Science enrolments. Increasing pressure on academic staff, and on teaching resources. Academic staff are also under increasing workload pressures, both in expected research output and the breadth of expected curriculum. CRA, Computing Degree and Enrollment Trends, 2012-2013 Taulbee Survey, http://archive2.cra.org/uploads/documents/resources/taulbee/CRA_Taulbee_CS_Degrees_and_Enrollment_2012-13.pdf

Online discussion forums Online discussion forums provide a valuable method for students to engage with their colleagues and academic staff. Asynchronous, and flexible. But can easily get lost amongst the hundreds of messages and requests.

Text Classification “Thanks guys, it helped a lot” “I did an insertion sort but did not use PTList” “These results are also on the website” Non-Critical “Can there be a few more days given for assignment 3?” “Still not working for me” “I cannot access the submission system too” “I am disappointed that no-one from the group has yet responded”. Critical

Research Questions To what extent can a Naive Bayes classifier effectively classify forum posts in Computer Science courses to determine if a post requires intervention? Simple technique, cheap in terms of storage, computationally efficient. But, typically applied for larger discussion items. Assumes conditional independence. Can this still be effective in the context of discussion forums?

Research Questions To what extent can a Naive Bayes classifier effectively classify forum posts in Computer Science courses to determine if a post requires intervention? Can we improve the accuracy by applying negation, parts of speech (POS) tagging, word stemming or removal of stop words ?

Text Classifier Applied tag stripping to remove HTML tags Preprocessing modules able to be applied dependent on experimental conditions, including: Removal of stop words Part of Speech tagging Negation identification Word stemming

Experimental Data Discussion forum data across multiple undergraduate courses. Removed user/timeline information. 4,666 sentences. 1,978 manually identified as critical. 2,688 manually identified as non-critical. Total word count of 83,176.

Results

Analysis Correctly identified as Critical: “I’m quite confused.”, p=0.965 “So what exactly are we supposed to do in the Test file?”, p=0.999 Incorrectly identified as Critical: “Now my code passes example 8.”, p=0.7318

Analysis More importantly, though, is to consider those sentences that are Critical but identified as Non-critical: “Please help us to solve this problem, thank you very much.” , p=0.885

Conclusions Identified as an efficient technique, NB has also been shown to be effective in the classification of critical discussion forum posts. Shows promise to be combined with other indicators of student engagement, such as patterns in resource access and overall participation to direct academic attention.

Computer Science Education Research Group blogs.adelaide.edu.au/cser FIN