Jose Carlo A. Soriano

Break down Effective Help-seeking Behavior Among Students Using an Intelligent Tutoring System for Math: A Cross-Cultural Comparison

Intelligent Tutoring Systems Seeks to simulate the effectiveness of a good personal human tutor Individual tutoring is more effective than classroom instruction by 2 standard deviations (Bloom, 1984) Knows what specific skills the student is having trouble with Able to offer appropriate help at an appropriate time Objective: help the student learn ITS are better by 1 standard deviation (Koedinger et al, 1998; Corbett et al., 2001)

Intelligent Tutoring Systems

Help-Seeking in ITS The kind of help, and how the help is offered, affects learning (Aleven et al, 2003) Students who use High-level help most frequently have the least learning(Matthews and Mitrovic, 2008) Help-seeking is a Meta-cognitive skill Meta-cognition is “cognition about cognition” One’s knowledge of the processes in play One’s active control of it during learning

Help-seeking Behavior 1. Both the EU and UNESCO declared: developing metacognitive skills, or ‘teaching students how to learn’ should be among the highest educational priorities (Louizidu and Kotselini, 2007) 2. Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991) 3. Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)

Effective Help-Seeking Student is more likely to learn when: Student seeks for help when encountering personal difficulties Student knows what kind of help is needed such that student can work effectively on his/her own Student knows how to ask for help Student is not dependent on help Student spends time understanding help given

Help-seeking in ITS However, students generally do not know when they need help (Aleven and Koedinger, 2000) Students “game the system” (Baker et al) Meta-cognitive tutors have been developed by Aleven et al “Scooter the tutor” developed by Baker et al To teach students “how to learn”

The Problem Is “effective help-seeking” the same across cultures? Very few cross-cultural comparisons Comparing ITS use between USA and Latin American students(Ogan et al, in press) Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010) Implications on Meta-cognitive tutors

Method Find out if effective help-seeking behavior is the transferrable across cultures, or are significantly different Might encourage more cross-cultural comparisons Implications on future efforts on meta-cognitive tutoring

Scatterplot tutor

Data Costa Rica Mexico USA Philippines

Feature Engineering 1. Helpavoidance 2. Nothelpavoidance 3. Helpnonuse 4. Unneededhelp 5. BugmsgLongpause 6. BugmsgShortpause 7. HintmsgLongpause 8. HintmsgShortpause

Feature Engineering 9. HintmsgLongpauseCorrect 10. HintmsgShortpauseCorrect 11. NothelpavoidanceShortpause 12. NothelpavoidanceLongpause 13. UnneededhelpShortpause 14. UnneededhelpLongpause 15. ShortpauseHintmsg 16. LongpauseHintmsg 17. FirstattemptHintmsg

Feature Optimization Most features require a threshold, either p-know or a time threshold Brute-force grid search: For p-know thresholds, grid-size is 0.05 For pause thresholds, grid-size is 0.5 seconds Single-parameter linear regression for each threshold for each feature in grid

Feature Selection Cross-validated r was used as the goodness criterion Correlation between the predicted learning values and the actual learning value The threshold with the best cross-validated r becomes the threshold for each feature As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creation

Model Creation and Evaluation Models were created using Forward Selection Models were evaluated by applying each country’s model to each country’s data set A model were created after combining the four data sets

Brute-Force Grid Search Featurecut-offrFeaturecut-offR CRPH Helpavoidance Nothelpavoidance Helpnonuse Helpnonuse Unneededhelp10.006NothelpavoidanceShortpause BugmsgLongpause NothelpavoidanceLongpause HintmsgLongpause UnneededhelpShortpause HintmsgShortpause US HintmsgLongpauseCorrect Helpavoidance NothelpavoidanceLongpause Helpnonuse UnneededhelpShortpause Unneededhelp UnneededhelpLongpause00.008BugmsgLongpause MXBugmsgShortpause Helpnonuse10.201HintmsgLongpause BugmsgShortpause HintmsgShortpause HintmsgShortpauseCorrect HintmsgLongpauseCorrect NothelpavoidanceLongpause HintmsgShortpauseCorrect ALLNothelpavoidanceShortpause Helpavoidance NothelpavoidanceLongpause Nothelpavoidance UnneededhelpShortpause Helpnonuse00.094UnneededhelpLongpause BugmsgShortpause ShortpauseHintmsg HintmsgLongpauseCorrect10.022LongpauseHintmsg HintmsgShortpauseCorrect NothelpavoidanceShortpause NothelpavoidanceLongpause UnneededhelpShortpause UnneededhelpLongpause10.062

Analysis FirstattemptHintmsg – only feature that was not able to pass in any country Different nature of skills Some can be hard at the very start, some can be easily understood from the start Most skills may require a large number of actions NothelpavoidanceLongpause – had positive cross-validated correlation in all five data sets In contrast to theory, this feature has negative directionality for most countries This might be because students who exhibit the behavior simply do not understand the skill

Forward Selection CountryLearning =Cross-validated r CR * Helpavoidance(0.15) * HintmsgLongpause(47.5) * HintmsgLongpauseCorrect(41.5) * NothelpavoidanceLongpause(0.25, 45.5) MX * Helpnonuse(1) * BugmsgShortpause(2.5) * HintmsgShortpauseCorrect(0.5) PH * Helpnonuse (0.95) * NothelpavoidanceShortpause(0.4, 1) US * Nothelpavoidance(0.25) * BugmsgLongpause(57) * NothelpavoidanceShortpause(0.25, 19.5) *LongpauseHintmsg(37.5) ALL * Helpavoidance(0.05) * Helpnonuse(0) * BugmsgShortpause(2.5)

Analysis Helpavoidance – negative directionality for CR and PH reinforcces Aleven et al’s findings that avoiding help is negatively correlated to learning (Aleven et al., 2006) ButmsgShortpause and NothelpavoidanceShortpause also has negative directionality for MX and US, and PH and US. Possibly effect of not spending enough time to understand bug or hint message provided

Cross-Cultural Evaluation CountryCRMXPHUSALL CR MX PH US All Rows are models, columns are data sets applied to

Analysis Eighteen out of 25 model applications produced positive correlation between model’s predicted learning and the actual learning Negative correlation means model did worse than chance at predicting learning LOOCV r values are very high compared to r when applied to other countries Reinforces hypothesis that help-seeking might not transfer across countries

Analysis MX and US – performance on each other’s data sets are low ( and ) This means that our model of effective help-seeking is not effective when applied to the other country Reinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico students Collaborative nature of students from Mexico may be the reason why help-seeking is different The help they ask from the tutor will only be help that they do not get from other students

Analysis CR and MX – did not perform well on each other’s data sets (-0.285, 0.04) Though collaborative tendencies might be common between Costa Rica and Mexico, help-seeking behavior with the ITS may differ US and PH – performed very well on each other’s data set (0.146 and 0.228) Reinforces findings in (San Pedro, 2011) wherein a carelessness detector is generalizable between the two countries In contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countries But Help-seeking and Disengaged behavior are two different sets of behaviors

Conclusion Results did not expose that effective help-seeking as a whole is very culture-specific (18 out of 25 applications returned positive r) However, it is not apparent that effective help-seeking is transferrable across countries Big difference between LOOCV r and cross-cultural evaluations

Conclusion and Recommendations There are pairs of countries wherein effective help-seeking fail to generalize to each other’s data set. Meaning effective help-seeking from one country may not necessarily be effective in another Single effective help-seeking models used by meta- cognitive tutors may be effective in one culture but not in others Future meta-cognitive tutors might have to use a more generalizable model May have to switch models, depending on the culture where the ITS is used