Using an Intelligent Tutoring System to increase parental engagement in student learning with automated messages By Zach Broderick, Kevin DeNolf, Jen Dufault, Cristina Heffernan and Neil Heffernan Experiment  The literature suggests that parental engagement in a student’s education is beneficial; however, this engagement requires access to information.  An Intelligent Tutoring System (ITS) replicates the services of a human tutor in software and collects an enormous amount of fine-grained learning data on students. ASSISTments is a free web- based ITS developed at WPI.  We implemented a parental notification component for the ASSISTments system that gives parents access to the valuable data collected by ASSISTments.  We ran an exploratory study to pilot the feature and found that it increased engagement, but that we needed a stronger intervention and experiment. Background Results and Analysis CollaboratorsSponsors About  We developed an extension that sends automated s to parents containing student data, thereby increasing the intervention. We hypothesized this would increase parental engagement and thus improve student performance.  We conducted a randomized controlled experiment at a local middle school with two 7 th grade math teachers that already used ASSISTments in their classroom.  Parents were given a pre and post survey measuring engagement. During the second unit of the year, half of the students’ parents received notification. Students completed 15 assignments and took a test each unit to measure performance. Zach Broderick is a graduate student in computer science at WPI and works as a developer in the ASSISTment lab under his advisor Neil Heffernan. He is also a GK12/PIMSE fellow funded by the NSF to work with Kevin and Jen and their students and help them effectively use the ASSISTment system. Kevin DeNolf and Jen Dufault are are both math teachers at Oak Middle School in Shrewsbury, MA and are partner teachers in the GK12/PIMSE program. They and their students both participated in this study. Cristina Heffernan is the project manager for the GK12/PIMSE grant. Neil Heffernan is a professor at WPI and creator of ASSISTments. He is the PI on the GK12/PIMSE grant and this study. Contact: Neil Heffernan Zach Broderick  Survey results indicated parents felt more engaged in student learning, especially those that opted to receive nightly (vs. weekly) s.  Students completed significantly more of their homework when their parents received notification, especially when controlling for the ceiling effect.  Qualitative feedback from parents and teachers was overwhelmingly positive.  Test scores did not improve at all as a result of the intervention.  Several statistically reliable results were obtained, and all results trended strongly in the right direction. Experiment Local middle school Two ASSISTment teachers 7 th grade math 4 classes/teacher, 20 students/class Students assigned to condition by going through roster alphabetically Two units, 15 assignments and 1 test each Intervention during 2 nd unit for exp. group ConditionUnit 1Unit 2 Experiment Students complete 15 homework assignments on ASSISTments. Students are given unit test Parents are not yet involved Parents are given pre-survey to measure engagement Parents are invited to sign up for accounts on ASSISTments Parents receive automated messages from the system throughout the unit Students complete 15 homework assignments on ASSISTments Students are given unit test Parents are given post-survey Control Students complete 15 homework assignments on ASSISTments. Students are given unit test Parents are not involved Parents are given pre-survey to measure engagement Parents are NOT given accounts and do NOT receive automated messages Students complete 15 homework assignments on ASSISTments Students are given unit test Parents are given post-survey Table 5.2 Parent responses to survey questions on engagement (who received nightly s) Scaled 1-5, 1=Strongly Disagree, 5=Strongly Agree Survey Question ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain I feel I have a good understanding of what is going on in my student’s math class (0.94) 3.94 (0.83) 0.47 (0.87) 3.14 (1.07) 4.00 (0.82) 0.86 (1.07)0.39 I feel I have a good understanding of how my student is performing in math class (0.99) 4.29 (0.92) 0.41 (0.94) 3.43 (1.13) 4.14 (0.38) 0.71 (0.95)0.30 I feel I am being provided enough information about my student's performance (1.06) 4.18 (0.81) 0.18 (1.01) 4.00 (0.58) 4.14 (0.69) 0.14 (0.38)-0.03 I check to make sure my student has completed their homework at night (1.00) 3.24 (1.30) (1.73) 3.29 (1.70) 4.43 (0.79) 1.14 (1.21)1.50 a I closely monitor my student's performance (as in, check grades on assignments and tests) 4.29 (0.92) 4.06 (1.03) (1.35) 4.71 (0.49) 4.71 (0.49) 0.00 (0.00)0.24 Standard deviations are noted in parentheses. N = 24; N(Control) = 17, N(Exp) = 7 a p < 0.05, Effect Size=0.95 Table 5.5 Student performance data (Teacher J only) Performance Metric ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain Avg percent of assignments completed (13.87) (13.82) 1.43 (12.84) (12.75) (6.88) 7.30 (12.22) 5.87 a Avg percent of assignments completed on time (13.95) (17.16) (15.18) (14.65) (15.76) 5.44 (15.80) 8.04 b Avg unit test score 3.29 (0.78) 3.26 (0.69) (0.63) 3.34 (0.54) 3.22 (0.67) (0.64)-0.09 Standard deviations are noted in parentheses. N = 85 a p < 0.05, Effect Size=0.46 b p < 0.05, Effect Size=0.53 Background Experiment Results and Analysis