D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana, Slovenia 2 Faculty of Computer and Information Science, University of Ljubljana, Slovenia Matej Zapušek 1, Martin Možina 2, Ivan Bratko 2, Jože Rugelj 2, Matej Guid 2 12 th International Conference on Intelligent Tutoring Systems ITS 2014: Honolulu, Hawaii 2014

S OME C ONCEPTS ARE D IFFICULT TO E XPLAIN... How to distinguish edible from toxic mushrooms?

I NTRODUCING D OMAIN E XPERTS Even for domain experts it is hard to articulate their knowledge!

M ACHINE L EARNING : H OW TO I NVOLVE A D OMAIN E XPERT ? The expert can state constraints and the domain knowledge in advance... … verify, evaluate, and correct results of machine learning… … or the expert and the computer iteratively improve the model. ABML argument-based machine learning

A RGUMENT -B ASED M ACHINE L EARNING given set of labeled learning examples e i described with attribute values D i where C i is classification of learning example e i goal learn prediction model (hypoteshis) H IF... THEN H CiCi  e i : D i ai ai example e i may have argument a i

I T IS M UCH E ASIER TO E XPLAIN I NDIVIDUAL C ASES ! Is this mushroom toxic? Why?

ABML K NOWLEDGE R EFINEMENT L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Expert explains the example Return to step 1 critical example learn data set Argument ABML

ABML K NOWLEDGE R EFINEMENT L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Expert explains the example Return to step 1 Step 3a: Explaining a critical example (in a natural language) Step 3b: Adding arguments to the example Step 3c: Discovering counter examples Step 3d: Improving arguments Return to step 3c if counter example found

I LLUSTRATIVE E XAMPLE : L EARNING TO D IAGNOSE F LU PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... The current model: IF Temperature < very high THEN Flu = no cannot explain well Pacient 2. The question to the expert: „What is the reason for Pacient 2 having the flu?“

PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... E XPERT ‘ S E XPLANATION Expert‘s explanation: „Pacient #2 has the flue because of a high temperature.“ Expert‘s argument is attached to learning example #2. New model is built. Flu = yes BECAUSE Temperature > Normal

PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... W HAT IF THE E XPERT ‘ S A RGUMENT IS NOT GOOD ENOUGH ? ML method now induced a rule consistent with argument: IF Temperature > Normal THEN Flu = yes The rule is inconsistent with data! Expert is presented with counter example: „Compare pacients #2 and #4. Why Pacient #4 doesn‘t have the flu?“

PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... T HE E XPERT MAY I MPROVE THE A RGUMENT Expert finds the crucial difference between Pacients #2 and #4: „Pacient 2 didn‘t get vaccinated against the flu.“

PacientTemperatureVaccinationCoughingHeadache...Flu 1normalyesno...no 2highnoyesno...yes 3very highno yes...yes 4highyes no...no... I MPROVED R ULES MAY E XPLAIN U NSEEN E XAMPLES AS W ELL ML method induces a new rule: IF Temperature > Normal AND Vaccination = no THEN Flue = yes The new rule also explains diagnosis for Pacient #3: „Has flu because of a high temperature and didn‘t get vaccinated against it.“

ABML K NOWLEDGE R EFINEMENT L OOP : T HE I NNER L OOP Step 3a: Explaining a critical example (in a natural language) „Pacient #2 has the flue because of a high temperature.“ Step 3b: Adding arguments to the example Step 3c: Discovering counter examples Step 3d: Improving arguments with counter examples IF T EMPERATURE > N ORMAL AND V ACCINATION = NO Temperature > Normal

ABML R EFINEMENT L OOP & K NOWLEDGE E LICITATION IF... THEN ABML argument-based machine learning explain single example easier for experts to articulate knowledge “critical” examples expert provides only relevant knowledge “counter” examples detect deficiencies in explanations arguments critical examples counter examples

E XPERT CAN I NTRODUCE NEW C ONCEPTS (A TTRIBUTES ) Pacient...HeadacheFatigueSoreThroatAppetiteFlu 1...no yesnormalno 2...noyes lowyes 3...yes nolowyes 4...no normalno... FluSymptoms no yes no Possible rule with the new attribute: IF Temperature > Normal AND FluSymptoms = yes THEN Flu = yes...

E XPERT CAN C ORRECT C LASSIFICATION OF L EARNING E XAMPLE...HeadacheFatigueSoreThroatAppetiteFluSymptomsFlu...no normalnoyes PacientTemperatureVaccinationCoughing... 37normalno... The question to the expert: „What is the reason for Pacient 37 having the flu?“ Expert corrects the classification of Pacient 37: „Pacient 37 doesn‘t have the flu.“ no

K NOWLEDGE E LICITATION WITH ABML IF... THEN ABML argument-based machine learning inconsistencies in labels are detected automatically misclassificated examples are easily recognized and corrected arguments critical examples counter examples experts introduce new attributes human-understandable models suitable for teaching

IF... THEN ABML argument-based machine learning arguments critical examples counter examples How to use ABML in educational setting? I NTRODUCING S TUDENTS

T HE O UTER L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Student explains the example Return to step 1

T HE O UTER L OOP & THE I NNER L OOP Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Student explains the example Return to step 1 Step 3a: Explaining a critical example (in a natural language) Step 3b: Adding arguments to the example Step 3c: Discovering counter examples Step 3d: Improving arguments with counter examples Return to step 3c if counter example found USING TEACHER‘S ATTRIBUTES!

A RGUING TO L EARN Argumentation involves elaboration, reasoning, and reflection. These activities have been shown to contribute to deeper conceptual learning (Bransford, Brown, & Cocking, 1999) Participating in argumentation helps students learn about argumentative structures (Kuhn, 2001)

A N EW P ARADIGM arguing to learn with argument-based machine learning

Let‘s see how this works in practice

B ASIC OR A DVANCED ? „basic“ „advanced“

L EARNING D ATA S ET 121 solutions of 62 different exercises teacher labeled each solution as „basic“ or „advanced“ learn data: 91 examples test data: 30 examples

K NOWLEDGE E LICITATION FROM THE T EACHER 1. relevant description language: new attributes 2. consistently labeled examples TEACHER‘S GOALS:

R ESULTS OF K NOWLEDGE E LICITATION FROM THE T EACHER 9 iterations 9 new attributes 9 rules only 1 out of 5 initial attributes remained

A S TUDENT -C OMPUTER I NTERACTIVE L EARNING S ESSION STUDENT‘S TASK obtain rules for distinguishing „basic“ and „advanced“ solutions rules must consist of attributes in teacher‘s final model use teacher‘s descriptive language

A S TUDENT -C OMPUTER I NTERACTIVE L EARNING S ESSION RECOMMENDATIONS TO THE STUDENT use the most important features for explanations use the smallest possible number of features in a single argument try not to repeat the same arguments

T HE F IRST „C RITICAL “ E XAMPLE The question to the student: „Why is this solution advanced?“ Student‘s argument: „Because function zip is present and the number of rows is low.“ Solution = advanced BECAUSE Zip = True AND cRows = low

C OUNTER E XAMPLE IF Zip = True AND cRows = low THEN Solution = advanced The rule is inconsistent with data! Student is presented with counter example: „Compare these two solutions. Why is the second solution a basic one?“

I MPROVING A RGUMENT IF Zip = True AND cRows = low AND LiCom = True THEN Solution = advanced Student‘s extended argument: „Because function zip is present, the number of rows is low, and a list comprehension occurs. “ no more counter examples next iteration

A T THE E ND OF THE I NTERACTIVE S ESSION 5 iterations half an hour 90% accuracy on (previously unseen) testing data several suggestions of new descriptive features

A SSESMENT Results: experiment with 7 students 7.1 iterations 87.1% classification accuracy of obtained rule model 86.7% correctly „manually“ classified (previously unseen) examples very positive qualitative feedback from the students

C ONCLUSIONS New paradigm: arguing to learn with argument-based machine learning Future work: applications in several domains assessing argument‘s quality for improved immediate feedback goal-oriented extension (see our ITS 2012 paper)

Q UESTIONS & D ISCUSSION Thank you! slides: ailab.si/matej enabling arguing to learn