1. Towards building a better cognitive model
Ph.D. progress talk
Mingyu Feng
Dec. 4th, 2008

2. What is a cognitive model? (drop)
ACT: a set of production rules that embodies the knowledge the student is trying to acquire (Corbett, Anderson & O’Brien, 1995)
Educational measurement: simplified description of human problem solving on educational tasks, “which helps to characterize the knowledge and skills students at different levels of learning have acquired and to facilitate the explanation and prediction of students’ performance” (Leighton & Gierl, 2007a, p. 6).
In this talk, a SKILL model refers to a group of skills that are associated with the problems to be presented to students

3. Importance of cognitive model
It is the “intelligence” in the tutors (e.g. cognitive tutors) as it allows the tutor to “solve” a problem and match students’ action to a production rule
The cognitive model provides guidance for all aspects of tutor development by keeping track of each rule the students have learned
e.g. when/how to deliver tutorial assistance, and what to say
adaptive problem selection
curriculum sequencing
It is also helpful for real teachers by informing the difficulties of their students

4. Previous work
A fine grained model

5. Skills in the current model
WPI-78
WPI-39
WPI-5
WPI-1
Inequality-solving
Setting-up-and-
solving-equations
Patterns,
Relations,
and Algebra
Math
(Unidimensional assessment)
Equation-solving
Equation-concept …
Plot-graph
Modeling-covariation
X-Y-graph
Understand-line-slope-concept
Congruence
Understand-and-applying-congruence-
and-similarity
Geometry
Similar-triangles
Perimeter
Using-measurement-
formulas-and-techniques
Measurement
Area

6. Previous work
The fine grained model did a better job tracking student knowledge than coarser grained models (Feng, Heffernan, Mani & Heffernan, 2006; Pardos, Heffernan, Anderson, & Heffernan, 2006)

7. Thinking of refining existing model
Modeling student knowledge is hard due to various sources of uncertainty (Katz, Lesgold, Eggan & Gordin, 1992 )
multiple sources of student errors; careless slip and lucky guesses; learning and forgetting
The first model comes from “brain-storm” of domain experts and subject to the risk of “expert blind spots”
Our first model fits relative well but not flawless
Residuals for some skills/problems are relative large
Finer grained model generally did better than coarser grained ones but a model of 39 skills beats up the one with 106 skills on some dataset.

8. Effort by the domain expert
Break up coarse skills
e.g. Ordering-numbers
-> ordering integer
-> ordering fractions
-> ordering decimals
Rephrase confusing skills
e.g. Symbolization articulation (everybody asks: what does it mean?)
Adding in some new skills

9. How modeling results can be helpful
Model misfit information
internal residual
external prediction accuracy
Learning result
What skills are hard for students.
What skills are easy vs. hard to learn.
Very low (even negative) learning rate on a skill is a bad sign

10. Power law of learning on a skill
(Anderson, 1993)

11. Finding out misplaced problems
The mixed-effects model we fit does not tell which problems cause more learning
An problem did not produce much learning possibly because
The instruction is not helpful
Or the problem is associated with a “wrong” skill
These problems are potentially the “red lights” that should be examined first, esp. when the amount of problems is too big for a human expert to go through every problem

12. Finding out misplaced problems
The idea of learning decomposition (Beck, 2006)
Suppose we want to tell whether problem X or problem Y that are both related to one skill is more effective at fostering learning, the total number of practice can be broken up into two parts tX and tY; the value of B indicates how effective problem X is comparing to problem Y.

13. Learning factor analysis
A semi-automatic approach for model refinement (Cen, Koedinger, & Junker, 2006)
A base model
Difficulty factors
Operators that mutate the base model based on difficulty factors
A statistical model that evaluates how a cognitive model fits the data
A searching process guided by a heuristic
Now we have talked about improve a model by human, that’s too much work. Why not asking the computer to do it for us?

14. Difficulty factors Maybe figure type matters?
Maybe color of the figure matters?
Maybe whether the figure is presented along or inside another figure.
Maybe the orientation of the figure matters. Maybe this matter only for certain figure types.
DFA refers a property of a problem that causes student difficulties. Human involved procedures, that’s why LFA is a semi-automated method.
S=?
S=?

15. Operators Merge() Add() Split() Ordering-decimals
Ordering-fractions Ordering numbers
Ordering-integers
Add()
Adding in two new skills: embedded-figure; alone-figure
Split()
Circle-area circle-area-embedded
circle-area-alone

16. Statistical model & Heuristics
A statistical model (Cen, Koedinger & Junker, 2006)
Heuristics for selecting models
Goal 1: Best Fitting to the Data
Goal 2: Parsimony- few parameters
AIC = -2*log-likelihood + 2*number of params
BIC = -2*log-likelihood + number of params * number of observations
Bayesian Information Criterion used to weight off the better fit gained from adding more parameters

17. The search space of cognitive models
The space is defined/expanded by applying searching operators on existing models
(Cen, Koedinger & Junker, 2005)

18. Application of LFA
Do students from different districts employ different cognitive models when they solve problems?
Say, Worcester vs. Shrewsbury
Related work
Rafferty & Yudelson (2007), Koedinger & Mathan (2004), Leszczenski & Beck (2007)
Particularly, a compact model fit the students with low initial knowledge and high learning gain best. This group of students shows a steeper learning than others and “are learning a cognitive model in which each skill attempt is equivalent to multiple attempts on different skills for other groups”.

19. Extending LFA to allow hierarchy
Introducing a new operator
make_A_prerequisite_of_B(A, B)
Skill A is on the critical path of the developmental process of B
Every problem that is tagged with B is also tagged with all its prerequisite skills
Skill X is a prerequisite of skill Y implies that X is on the critical path of the developmental process of Y

20. Representing hierarchical relationshipB C D E F 1 R A B C D E F 1
A matrix: adjacency matrix
R matrix: reachability matrix
Q matrix: problem-skill mapping matrix Q1 A B C D E F P1 1 P2 P3 Q2 A B C D E F P1 1 P2 P3

21. Remaining concerns ? Evaluate a hierarchical model
? Which operators are good under certain circumstances
? Impact of the new operator on existing operators
? The hierarchy structure is highly related to problem development and delivery.
Potentially a prerequisite hierarchy structure of skills will have impact on the mutating procedure of other operators, especially the two operators of Split and Merge. There are some questions that wait to be answered. Let’s take the split() function as an example. If skill A and B are prerequisites of skill C, when we want to perform a Split operation on skill C, how the relationship between A, C and B, C should be updated accordingly? Similarly, the merge() should be guided by the prerequisite hierarchy. Suppose skill A is prerequisite of skill C, and skill B is prerequisite of skill D. In this case, can skill A be merged with skill B? Will the new merged skill be prerequisite of C or D too? Or maybe the prerequisite relationship should just be disregarded after split() or merge().
The hierarchy structure of skills provides guidance of test development. (Leighton, Gierl, & Hunka, 2004). Presumably, prerequisite skills should be tested before their children skills. For instance, suppose the skills equation-solving and square-root are prerequisites of applying Pythagorean Theorem. We probably should not consider giving students a problem of applying Pythagorean Theorem before they have mastered equation-solving and square root. The hierarchy structure also represents cognitive specification for test development.

22. Conclusion We are working on building a better cognitive model.
But the approaches mentioned above need to be verified by the modeling results
coming … I don’t have that yet. 