Tracing behaviors associated with motivational states and learning outcomes when students learn with the Cognitive Tutor Team: Matthew Bernacki & Pranav Garg Mentors: Erik Zawadzki & Ryan Baker 2011 Summer School

Overview We investigated relationships between motivation, learning behaviors and learning outcomes amongst high school students learning geometry using the Cognitive Tutor. We identified a series of 3 sequential behaviors (triplets) and plotted their frequency across the logs of 38 learners in one geometry unit. We conducted a factor analysis to reduce 147 triplets into 28 factors and examined their correlation with self-reports of affective state, self-efficacy for the unit and their achievement goals for mathematics.

METHOD: In the Classroom Participants 38 high school geometry students completing Unit 13 in the Cognitive Tutor, which was a standard component of the their rural high school’s geometry curriculum. Instruments Cognitive Tutor for Geometry –Unit 13: Circumference and Area of Circles Achievement Goal Questionnaire-Revised –Elliot & Murayama, 2008 (9 items, 3 per Mastery Approach, Performance Approach, Performance Avoidance subscale) Academic Self Efficacy Survey –Midgely, et al., 2000; Patterns of Adaptive Learning Survey Affect (single items constructed for this project) –Boredom, Confusion, Frustration Engaged Concentration, Positive Experience

METHOD: Data Mining Procedure 1. Exported transaction level log file from Cognitive Tutor 2. Selected only those students who completed the Unit of interest; cleaned data to remove any students who were missing self-report data or a complete log file 3. Calculated the duration (seconds) to complete each learner action in the OUTCOME column –OK – answered problem step correctly –BUG – incorrectly answered the problem step (common error) –ERROR – incorrectly answered the problem step –HINT [1,2,3]– requested a hint –SWITCH – switched their window to consult a worked example 4. Recoded Duration by Quartile (1, middle 2&3, 4) 1. Q1 = Short durations, typically 1-2 seconds; coded as “…_1” 2. Q2&3 = Medium Durations, typically 2-10 seconds; coded as “…_2” 3. Q4 = Longest durations, typically upward of 10 seconds; coded as “…_3” 5. Concatenated Outcome with Q(uartile version of) Duration.

METHOD: Data Mining Cont’d. 6. Ran a script in Python to move a sliding window over the OutcomeQDuration column and populated a column with a triplet: [FIRST TRANSACTION_DURATION_SECOND_D_THIRD_D]. 7. Calculated the total number of unique triplets (n = 7,885) and, with a Pivot Table, determined the frequency each occurred per student. 8. Eliminated those that occurred less than 5 times and those that occurred in less than 2 students (n = 147) 9. Imported into SPSS, merged with a file of their self-reported motivational states and official record of learning outcomes 10. Ran a Principle Components Factor Analysis (unrotated) to determine a factor structure. 11. Correlated Factor Scores with motivation and performance data

RESULTS THOSE WHO…TEND TO CONDUCT BEHAVIORS THAT LOAD ON FACTOR …EXPERIENCE … Boredom15, 28 Confusion27 Frustration12, 13, 27 Self-Efficacy6,22 …PURSUE… Mastery or Performance Approach Goals27 Performance Avoidance Goals17 … PERFORM WELL ACCORDING TO… 4 th Quarter Grades3

Academic Self Efficacy BoredomConfusionFrustrationPositive Experience Engaged Concentration Mastery Approach Goals Performance Approach Goals Performance Avoidance Goals 3rd Qtr Grade 4th Qtr Grade FACTOR * * ** *.398 * * * * ** * ** ** **

THE FACTORS facto r Behavior Triplet with highest factor loading 2nd MAX3rd MAX 1 ['ERROR_3_ERROR_3_OK_3{.858}, ', 'ERROR_2_OK_3_BUG_2{.858}, ', 'OK_2_BUG_1_OK_2{.858}, '] ['OK_3_ERROR_2_ERROR_2{.845}, ']['ERROR_2_ERROR_1_ERROR_1{.835}, ', 'ERROR_1_ERROR_1_ERROR_1{.835}, ', 'ERROR_2_ERROR_2_ERROR_1{.835}, '] 2 ['OK_2_OK_3_OK_3{.825}, ']['OK_3_OK_3_OK_2{.758}, ']['OK_1_OK_3_OK_2{.746}, '] 3 ['OK_3_ERROR_3_OK_3{.623}, ']['hint_2_HINT2_1_hint_3{.542}, ', 'OK_3_BUG_2_OK_2{.542}, '] ['OK_3_hint_3_HINT2_2{.520}, '] 4 ['OK_3_OK_3_BUG_3{.738}, ']['OK_1_OK_2_OK_2{.616}, ']['OK_1_OK_1_OK_2{.584}, '] 5 ['ERROR_2_HINT_2{.647}, ', 'ERROR_3_OK_3_BUG_3{.647}, '] ['ERROR_3_OK_2_OK_2{.600}, ']['OK_2_OK_3_BUG_2{.591}, '] 6 ['OK_3_ERROR_3_ERROR_3{.540}, ']['OK_3_OK_1_OK_3{.470}, ']['OK_3_OK_3_ERROR_3{.467}, '] 7 ['OK_1_OK_1_BUG_3{.599}, ']['HINT3_2_ERROR_3_ERROR_2{.582}, '] ['OK_2_OK_3_BUG_1{.551}, '] 8 ['BUG_2_OK_2_OK_2{.612}, ']['OK_2_OK_2_BUG_2{.480}, ']['OK_1_OK_2_BUG_1{.453}, '] 9 ['OK_1_OK_1_BUG_1{.603}, ']['OK_2_BUG_2_OK_2{.500}, ']['OK_3_OK_3_OK_1{.483}, '] 10 ['BUG_3_OK_3_OK_1{.465}, ']['OK_2_BUG_1_OK_3{.457}, ']['HINT3_3_OK_3_OK_2{.448}, '] 11 ['HINT_2 _HINT_1_HINT2_1{.560}, ']['OK_3_ERROR_3_ERROR_3{.437}, ']['ERROR_2_OK_3_ERROR_2{.433}, '] 12 ['BUG_1_OK_3_OK_2{.490}, ']['BUG_3_OK_2_OK_3{.477}, ']['ERROR_3_OK_3_OK_3{.382}, '] 13 ['OK_3_BUG_1_OK_2{.518}, ']['HINT3_2_ERROR_3_ERROR_2{.476}, '] ['OK_1_OK_3_BUG_1{.430}, '] 14 ['OK_2_OK_3_ERROR_3{.437}, ']['OK_3_ERROR_2_OK_2{.387}, ']['BUG_3_OK_2_OK_2{.349}, ']

THE FACTORS factor Behavior Triplet with highest factor loading 2nd MAX3rd MAX 15 ['HINT2_1_hint_3_HINT2_2{.575}, ']['hint_3_HINT2_1_HINT3_3{.451} ']['BUG_3_OK_3_OK_3{.401}, '] 16 ['ERROR_3_OK_3_OK_2{.463}, ']['OK_3_ERROR_2_OK_2{.424}, ']['OK_3_ERROR_2_ERROR_3{.372}, '] 17 ['OK_2_OK_3_hint_3{.501}, ']['OK_2_ERROR_2_OK_3{.459}, ']['OK_2_OK_2_BUG_2{.374}, '] 18 ['BUG_3_OK_3_OK_1{.396}, ']['OK_1_ERROR_3_OK_3{.356}, ']['HINT3_2_ERROR_3_ERROR_2{.327}, '] 19 ['OK_3_OK_1_OK_1{.377}, ']['BUG_3_OK_2_OK_3{.350}, ']['OK_2_OK_3_OK_2{.337}, '] 20 ['BUG_1_OK_3_OK_2{.373}, ']['ERROR_3_OK_2_OK_3{.372}, ']['OK_3_OK_2_OK_1{.319}, '] 21 ['OK_3_BUG_1_OK_3{.382}, ']['BUG_3_OK_2_OK_3{.312}, ']['BUG_1_OK_3_OK_2{-.395}, '] 22 ['ERROR_2_ERROR_1_OK_3{.337}, ']['OK_3_BUG_1_OK_2{.330}, ']['OK_1_OK_1_ERROR_2{.328}, '] 23 ['OK_1_BUG_3_OK_2{.488}, ']['ERROR_3_OK_3_BUG_2{.430}, ']['OK_2_OK_3_ERROR_3{.331}, '] 24 ['OK_2_OK_1_OK_3{.420}, ']['OK_2_OK_3_BUG_3{.306}, ']['HINT2_1_HINT3_3_OK_3{-.343}, '] 25 NULL*NULL 26 NULL*NULL 27 ['OK_1_OK_3_OK_1{.306}, ']NULL 28 NULL*NULL

THEORETICAL CONCLUSIONS Some behaviors associated with a factor can be interpretted somewhat easily. Factor 12: BUG_1_OK_3_OK_2{.490} BUG_3_OK_2_OK_3{.477} ERROR_3_OK_3_OK_3{.382} –Student made errors, often after some perserveration, then correctly answered items after a medium to long period. –Factor is associated with low frustration.

THEORETICAL CONCLUSIONS However, scores on one factor (#27) were significantly associated with self-reports of confusion and frustration and negatively associated with mastery and performance approach goals. –Only one triplet [OK_1_OK_3_OK_1] loaded higher than.30 on the factor –Low factor loading and meaningfulness of a short period prior to a correct, followed by a long and a short actually run counter to some conclusions based on self-reports.

METHODOLOGICAL CONCLUSIONS Triplets composed of behaviors and their durations can be meaningful measures of behavior They can also be insufficiently descriptive of a students’ behavior re: specificity –number of behaviors captured –Precision of duration when collapsed to quartile –Meaningfulness of cuts between quartiles

NEXT STEPS Test factor structure across additional units –If not, may make sense to abandon factors and examine relations one behavioral trace at a time Generate 4-lets and 5-lets to see if these behaviors provide more intuitive glimpses of student behaviors Once a set of behaviors has been found that associate with motivation –develop a flag for a behavior… and an intervention? Test structural models with paths from motivational state to behavior to learning outcomes