Toward the extraction of production rules for solving logic proofs Tiffany Barnes, John Stamper Computer Science

Toward prod rules… Barnes & Stamper2 Vision Inexpensive, scalable, individualized instruction/learning Lifelong learning Create a shift in teacher role: –Focus on human interactions –Focus on engaging & inspiring students

Toward prod rules… Barnes & Stamper3 Use behavior to offer help to new students Data mining for feedback student Contents Unknown Behavior Known

Toward prod rules… Barnes & Stamper4 Goals Generate feedback & hints for –Students –Teachers Automatically –Based on prior student work –Scale to new problems, new topics –Optimize for student performance

Toward prod rules… Barnes & Stamper5 Overview Context & background Proposed solution Visualization of MDPs on student data Learning from these graphs Future work

Toward prod rules… Barnes & Stamper6 Context Discrete Math & Logic courses –NCSU Discrete Math course –UNCC Logic & Algorithms course –UNCC Philosophy course All include logic proofs Students have difficulty developing strategies to solve proofs

Toward prod rules… Barnes & Stamper7 Logic Proof Tutor Online tool for writing logic proofs Add intelligent feedback & help Assist with most difficult part of proofs! Do this automatically using student data

Toward prod rules… Barnes & Stamper8 Approach Use student data to construct Markov Decision Processes that represent all student solutions Trace student behavior in the MDP Devise MDP reward functions that point toward useful past approaches of students for feedback & hints

Toward prod rules… Barnes & Stamper9 Related work Intelligent Tutoring Systems –Tradeoffs in ITS development: Time to build expert model (Murray 99) –Most on production rules to model student work Constraint-based tutors –Less time to construct –Comparable to cognitive tutors –(Mitrovic, Koedinger, Martin, 2003)

Toward prod rules… Barnes & Stamper10 Related work ASSERT –Authoring tool –Uses theory refinement to learn student models from behavior –(Baffes & Mooney 1996) CTAT: learning pseudo-rules by example –(Koedinger, Aleven, Heffernan, McLaren, Hockenberry, 2004)

Toward prod rules… Barnes & Stamper11 Related work Bootstrapping Novice Data (BND) –Student data to build initial models in CTAT –(McLaren, Koedinger, Scneider, Harrer, Bollen, 2004)

Toward prod rules… Barnes & Stamper12 Logic tutors Logic-ITA –Intelligent tutor for solving proofs –Verifies answers, debriefing feedback –Merceron & Yacef 2005 Deep Thought –Online Java applet –Graphical tool to write proofs –Croy, 2000 Neither offer help on what to do next

Toward prod rules… Barnes & Stamper13 Deep Thought

Toward prod rules… Barnes & Stamper14 Logic Proof Sample Statemt LineReason 1. a → b Given 2. c → dGiven 3. ¬ (a → d)Given ¬ a v d 3IM (error) 4. a ^ ¬ d 3IMplication 5. a 4Simplification b 4MP (error) b 1MP (error) 6. b 1,5Modus Ponens 7. ¬ d 4Simplification 8.¬c 2,7Modus Tollens 9. b ^ ¬c 6,8Conjunction

Toward prod rules… Barnes & Stamper15 State Transition Problem Solving Newell & Simon (Human Problem Solving, 1972) ‘Problem’ Defined in Terms of: –Starting State –Goal State –Transition Rules Related to ACT-R –A cognitive architecture with declarative & procedural knowledge Procedural knowledge in ACT-R –Current problem state (working memory) –Production rules –A rule interpreter: Performs model tracing to match a sequence of production rules to student actions

Toward prod rules… Barnes & Stamper16 Markov Decision Process State set S –Problem state (steps & conclusion) Action set A –Actions a student can take Transition probabilities P –Probability of transitioning between states using a particular action A reward function R –Assigns a value to each state –Negative rewards = penalties

Toward prod rules… Barnes & Stamper17 MDP of solutions Each student solution is represented by states with action transitions All combined into a single graph Identical solutions mapped to same states This is a graph of all student solutions Next step: compute reward function that optimizes path to the goal We propose alternate paths for generating feedback

Toward prod rules… Barnes & Stamper18 Proposed MDPs 1.Find expert-like paths High goal state reward Penalties for actions and errors 2.Find a typical path to the goal state. High goal state reward High rewards for actions of many students 3.Find paths with low probabilities of errors. High error penalties

Toward prod rules… Barnes & Stamper19 Hypothesis Zone of Proximal Development (Vygotsky, 1986) –Students are able to learn new things that are closest to what they already know Therefore, –Feedback based on frequent student actions may be closer to what the typical student is “ready to understand”

Toward prod rules… Barnes & Stamper20 MDPs for feedback We use MDPs to do model tracing as in ACT-R –But the paths are problem-specific If student path found in MDP –When a student presses “help” button –Choose an MDP function as proposed to generate feedback If not, no help generated

Toward prod rules… Barnes & Stamper21 Method Four semesters of data 429 student solutions to one problem –70% complete (13 lines) –30% partial (10 lines) Performed data cleaning Built MDP –Goal: 100, Error: -10, Action: 1 –Used value iteration for one step

Toward prod rules… Barnes & Stamper22 Results Aggregate MDP contains 547 states 90% of student errors related to explaining rule applications Plan to build a tool for teachers to explore student behavior, as shown in following graphs

Toward prod rules… Barnes & Stamper23 Key to visualizations Arrow format: frequency of student path Node format: MDP score

Toward prod rules… Barnes & Stamper24 Frequent student paths Only one reaches the goal state, also optimal

Toward prod rules… Barnes & Stamper25 Frequent paths, error states

Toward prod rules… Barnes & Stamper26 Another view The previous graphs show all different paths for students Perhaps if some were consolidated more patterns would emerge

Unique premises

Toward prod rules… Barnes & Stamper28 Frequent solutions Derived from unique premise graph Errors indicate trouble explaining rule applications

Toward prod rules… Barnes & Stamper29 Secondary approaches Indicate preference for students to us DS rule Longer, non-optimal solution Indicates discomfort with more direct solutions

Toward prod rules… Barnes & Stamper30 Conclusion We are able to extract MDPs for problems Optimal solutions exist in student work MDP visualizations help understand student behavior Can be used to generate hints Providing hints based on the most frequent approach may be useful for many students

Toward prod rules… Barnes & Stamper31 Logic proof implications Need a new interface for explaining rule applications Students need help getting started on problems Students need more practice with rules in presence of negation These surprised the professors!

Toward prod rules… Barnes & Stamper32 Future Work Apply machine learning to MDPs to learn more general production rule sets Add MDPs to Deep Thought and Proofs Tutorial, and a help button Experiment with results based on different types of help

Toward prod rules… Barnes & Stamper33 Tiffany Barnes John Stamper unc Charlotte Thank you! This work was partially supported by NSF #