Increasing parent engagement in student learning using an ITS with automated messages. A thesis presentation for the degree of Master of Science in Computer Science by Zachary Broderick Advisor: Neil Heffernan Reader: Carolina Ruiz March 1 st,

Overview and Outline 3

Introduction > Background 4

5

6

7

Component Development 8

9

10

Exploratory Study 11

Exploratory Study > Methods 12

Exploratory Study > Methods 13

Exploratory Study > Methods Exploratory Study Local middle school Two ASSISTment teachers 8 th grade math, honors and non-honors 4 classes/teacher, 20 students/class Students assigned to condition by class, 1 honors and one non-honors per teacher All parents invited in Fall, exp group sent messages in Spring ConditionFallSpring Experiment Parents are invited to sign up for accounts on ASSISTments and pilot parent notification. Parent activity is recorded in server logs Parents and students are given pre-survey to measure engagement Parents receive messages from teachers throughout the semester Parent activity is recorded in server logs Parents and students are given post-survey Control* *(the control group is largely ignored in this study) Parents are invited to sign up for accounts on ASSISTments and pilot parent notification. Parent activity is recorded in server logs Parents can still use ASSISTments but do NOT receive messages from the teacher 14

Exploratory Study > Results 15

Exploratory Study > Results Table 3.1 Parent responses to pre-survey questions on engagement Survey Question Survey Answer Averages PrePostPost - Pre % of parents whose score: Increased Decrea sedNeither I feel I have a good understanding of what is going on in my student’s math class (0.90) 3.9 (0.88)0.32 a 42%21%37% I feel I have a good understanding of HOW my student is doing in math class (0.80) 4.22 (0.65)0.1733%22%45% My child thinks I know how well he or she is performing in math class (0.69) 4.33 (0.59)0.0017% 66% I feel my school is *not* giving me enough information to adequately monitor my student’s progress (0.78) 1.78 (0.88)0.1728% 44% In the past week, how frequently did you check up on your student’s homework? 2.58 (1.22) 2.16 (0.9) %32%52% How often do you give consequences (rewards/punishments) for grades and homework completion? 2.42 (1.26) 2.16 (1.21) %47%27% How often have you interacted with ASSISTments in the last 30 days? 2.16 (1.26) 2.63 (0.9)0.4758%26%16% How often did you initiate contact with your student’s teacher in the last 30 days? 0.11 (0.32) 0.05 (0.23)-0.055%11%84% Standard deviations are noted in parentheses. a p ≤ 0.05 (approx.), effect size =

Exploratory Study > Results Table 3.2 Student responses to pre-survey questions on engagement Survey Question Survey Answer Averages PrePostPost - Pre % of students whose score: Increased Decre asedNeither I think my parents know what I’m doing in math class (1.06) 3.31 (1.03)0.0834%29%37% I think my parents know HOW I’m doing in math class (0.84) 4.34 (0.78)0.0928%21%51% My parents say down and helped me with my ASSISTment work this year. 36%34%-2%7%9%84% Standard deviations are noted in parentheses. 17

Exploratory Study > Results Table 3.3 Changes in Homework Completion Rates Based on Message Reception Homework Completion Rate Gain Received MessagesDidn’t Receive Messages Homework completed on time % (14%) 2.13% (20%) Homework completed by end of semester. 5.09% (22%) 3.77% (13%) Average days late (9.49) (5.96) Standard deviation in parentheses. 18

Exploratory Study > Results Table 3.4 Changes in Homework Completion Rates Based on Message Reception (Non-honors Students) Homework Completion Rate Gain Received MessagesDidn’t Receive Messages Homework completed on time. 4.38% (16%) -0.55% (24%) Homework completed by end of semester. 9.33% (30%) 5.93% (13%) Average days late (11.27) (6.62) Standard deviation in parentheses. 19

Exploratory Study > Results Table 3.5 Correlation of gains in homework completion rates with frequency of requests by parents for non-inbox pages of ASSISTments Homework Completion Rate Gain AverageCorrelation Homework completed on time %0.78** Homework completed by end of semester %0.54 Average days late **p ≤

Exploratory Study > Discussion 21

Experiment > Methods Experiment Local middle school Two ASSISTment teachers 7 th grade math 4 classes/teacher, 20 students/class Students assigned to condition by going through roster alphabetically Two units, 15 assignments and 1 test each Intervention during 2 nd unit for exp. group ConditionUnit 1Unit 2 Experiment Students complete 15 homework assignments on ASSISTments. Students are given unit test Parents are not yet involved Parents are given pre-survey to measure engagement Parents are invited to sign up for accounts on ASSISTments Parents receive automated messages from the system throughout the unit Students complete 15 homework assignments on ASSISTments Students are given unit test Parents are given post-survey Control Students complete 15 homework assignments on ASSISTments. Students are given unit test Parents are not involved Parents are given pre-survey to measure engagement Parents are NOT given accounts and do NOT receive automated messages Students complete 15 homework assignments on ASSISTments Students are given unit test Parents are given post-survey 22

Experiment > Results Table 5.1 Parent responses to survey questions on engagement Scaled 1-5, 1=Strongly Disagree, 5=Strongly Agree Survey Question ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain I feel I have a good understanding of what is going on in my student’s math class (0.94) 3.94 (0.83) 0.47 (0.87) 3.32 (1.08) 3.77 (0.88) 0.45 (1.03)-0.02 I feel I have a good understanding of how my student is performing in math class (0.99) 4.29 (0.92) 0.41 (0.94) 3.74 (0.93) 4.10 (0.79) 0.35 (0.80)-0.06 I feel I am being provided enough information about my student's performance (1.06) 4.18 (0.81) 0.18 (1.01) 4.03 (0.87) 4.23 (0.72) 0.19 (0.91)0.02 I check to make sure my student has completed their homework at night (1.00) 3.24 (1.30) (1.73) 3.74 (1.34) 3.97 (1.14) 0.23 (1.12)0.58 I closely monitor my student's performance (as in, check grades on assignments and tests) 4.29 (0.92) 4.06 (1.03) (1.35) 4.19 (0.79) 4.39 (0.76) 0.19 (0.65)0.43 Standard deviations are noted in parentheses. N = 48 23

Experiment > Results Table 5.2 Parent responses to survey questions on engagement (who received nightly s) Scaled 1-5, 1=Strongly Disagree, 5=Strongly Agree Survey Question ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain I feel I have a good understanding of what is going on in my student’s math class (0.94) 3.94 (0.83) 0.47 (0.87) 3.14 (1.07) 4.00 (0.82) 0.86 (1.07)0.39 I feel I have a good understanding of how my student is performing in math class (0.99) 4.29 (0.92) 0.41 (0.94) 3.43 (1.13) 4.14 (0.38) 0.71 (0.95)0.30 I feel I am being provided enough information about my student's performance (1.06) 4.18 (0.81) 0.18 (1.01) 4.00 (0.58) 4.14 (0.69) 0.14 (0.38)-0.03 I check to make sure my student has completed their homework at night (1.00) 3.24 (1.30) (1.73) 3.29 (1.70) 4.43 (0.79) 1.14 (1.21)1.50 a I closely monitor my student's performance (as in, check grades on assignments and tests) 4.29 (0.92) 4.06 (1.03) (1.35) 4.71 (0.49) 4.71 (0.49) 0.00 (0.00)0.24 Standard deviations are noted in parentheses. N = 24; N(Control) = 17, N(Exp) = 7 a p < 0.05, Effect Size=

Experiment > Results Table 5.3 Parent responses to survey questions on engagement (low students) Scaled 1-5, 1=Strongly Disagree, 5=Strongly Agree Survey Question ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain I feel I have a good understanding of what is going on in my student’s math class (0.98) 3.92 (1.00) 0.25 (0.87) 3.29 (1.06) 3.81 (0.81) 0.52 (1.08)0.27 I feel I have a good understanding of how my student is performing in math class (0.79) 4.17 (1.03) 0.08 (0.67) 3.48 (0.93) 3.90 (0.77) 0.43 (0.81)0.35 I feel I am being provided enough information about my student's performance (0.87) 4.08 (0.90) (0.58) 3.86 (0.91) 4.24 (0.70) 0.38 (0.86)0.55 a I check to make sure my student has completed their homework at night (0.89) 3.17 (1.40) (1.73) 3.67 (1.20) 3.95 (1.17) 0.29 (0.78)0.79 I closely monitor my student's performance (as in, check grades on assignments and tests) 4.33 (0.78) 4.00 (1.13) (1.30) 4.00 (0.77) 4.29 (0.78) 0.29 (0.72)0.62 Standard deviations are noted in parentheses. N = 33 a p < 0.05, Effect Size=

Experiment > Results 26

Experiment > Results 27

Experiment > Results 28

Experiment > Results 29

Experiment > Results Table 5.4 Student performance data Performance Metric ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain Avg percent of assignments completed (10.55) (11.02) 1.52 (10.09) (10.69) (5.58) 4.43 (10.01)2.91 Avg percent of assignments completed on time (14.56) (16.66) 0.41 (15.89) (13.44) (14.49) 2.35 (14.83)1.94 Avg unit test score 3.23 (0.63) 3.20 (0.68) (0.63) 3.32 (0.51) 3.22 (0.62) (0.59)-0.06 Standard deviations are noted in parentheses. N =

Experiment > Results Table 5.5 Student performance data (Teacher J only) Performance Metric ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain Avg percent of assignments completed (13.87) (13.82) 1.43 (12.84) (12.75) (6.88) 7.30 (12.22) 5.87 a Avg percent of assignments completed on time (13.95) (17.16) (15.18) (14.65) (15.76) 5.44 (15.80) 8.04 b Avg unit test score 3.29 (0.78) 3.26 (0.69) (0.63) 3.34 (0.54) 3.22 (0.67) (0.64)-0.09 Standard deviations are noted in parentheses. N = 85 a p < 0.05, Effect Size=0.46 b p < 0.05, Effect Size=

Experiment > Results Table 5.6 Student performance data (nightly s) Performance Metric ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain Avg percent of assignments completed (10.55) (11.02) 1.52 (10.09) (12.94) (5.90) 6.87 (13.45)5.35 Avg percent of assignments completed on time (14.56) (16.66) 0.41 (15.89) (14.79) (21.38) 5.53 (16.86)5.12 Avg unit test score 3.23 (0.63) 3.20 (0.68) (0.63) 3.23 (0.62) 3.13 (0.64) (0.54)-0.06 Standard deviations are noted in parentheses. N = 90; N(Control) = 75; N(Exp) = 15; 32

Experiment > Results Table 5.7 Student performance data (low performing) Performance Metric ControlExperiment Unit 1Unit 2GainUnit 1Unit 2Gain∆Gain Avg percent of assignments completed (15.24) (17.68) 4.57 (15.53) (13.24) (7.79) (13.62)7.89 Avg percent of assignments completed on time (11.68) (21.63) 7.43 (20.32) (7.56) (17.84) (18.15)3.28 Avg unit test score 2.88 (0.79) 2.86 (0.74) (0.90) 3.09 (0.58) 3.00 (0.69) (0.60)-0.06 Standard deviations are noted in parentheses. N = 45 33

Experiment > Discussion 34

Experiment > Discussion 35

Experiment > Future Work 36

Acknowledgements 37