1. 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended Questions: a case study in school algebra ITS’2008

2. Cognitive modeling authoring tool  Problem  Multi-step reasoning, multiple equivalent reasonings  Our approach 1.An expert teacher (or a researcher) defines diagnosis exercises 2.A cognitive engineer implements templates that generalize these particular diagnosis exercises 3.A teacher clones these diagnosis exercises by filling template forms 4.A domain specific application generates the clone and a set of plausible correct and incorrect anticipated solutions matches the student’s reasoning with anticipated solutions 2

3. Outline  An introductory example  Pépite : a specific diagnosis tool  PépiGen : a system to clone Pépite  Author’s and Student’s points of view  Automatic Diagnosis  How does it work ?  Pépinière * Formal processing of expression trees  Conclusion 3 * in French : tree nursery

4. Blandine ValidityIncorrectV3 Use of lettersIncorrectL3 TranslationStep-by-step with incorrect chainsT4 Algebraic Expressions writing Incorrect use of parentheses with memory of meaning EA31 JustificationBy algebra using incorrect rulesJ3

5. Aliou ValidityIncorrectV3 Use of lettersNoL5 TranslationStep-by-stepT2 Algebraic writingNoEA? JustificationBy exampleJ2

6. Definitions  Diagnosis exercise  An exercise (statement and user interface)   an analysis grid to assess every plausible solution anticipated by experts  Clone  A similar exercise has the same kind of statement and user interface gives the same kind of information on students’ competence  an analysis grid to assess every plausible solution automatically generated by the system 6

7. PépiGen  A system to clone the Pépite diagnosis tool  An author (a teacher)  Chooses an exercise to be cloned  Enters the statement of the clone  PépiGen generates  The student’s interface  Each plausible solution (correct or incorrect) and its assessment on several dimensions 7

8. The Author’s interface 8

9. The Student’s interface 9

10. The Automatic Diagnostic 10

11. Outline An introductory example Pépite : a specific diagnosis tool PépiGen : a system to clone Pépite  How does it work ?  Pépinière  Expanding the tree of plausible steps of correct and incorrect algebraic transformations  Walking through the tree to anticipate different solutions and their assessment  Diagnosing the student’s reasoning  Conclusion 11

12. Plausible steps (x+6)*3-3x -2x+18 18 3x+18-3x x*3+6*3-3xx+6*3-3x 3x+18-3x 18x 21x-3x R1 R3 R2 R4 R3 21x-3x 18x R5 Correct rules R1 : (A+B)C AC+BC R3 : AB+AC A(B+C) R2: (A+B)C A+BC R4: AB+C B(A+C) R5: A+B*C (A+B)*C Incorrect rules 18 R3 R4 V1,EA1V3,EA42 V3,EA31 V3,EA31EA42 V3,EA32

13. Analysis grid generation  PépiGen 1.sends the algebraic expression to Pépinière that returns a tree of plausible steps Validity and Algebraic Expression Writing 2.completes the plausible solutions set with Non optimal algebraic 3.completes each solution assessment on the 5 dimensions V, EA, L, T, J 4.saves each algebraic solution and its assessment XML file : solution analysis grid  Note :  arithmetic reasonings are analyzed by the diagnosis system 13

14. Analysis grid (extract) (…) Algebraic proof ; the student interprets the statement as an equation V2,EA1,L1,T1,J1 (x+6)*3-3*x = 18 x*3+6*3-3*x = 18 C,3 x*3+18-3*x = 18 18 = 18 14

15. Automatic diagnosis XM L Diagnosis system loaded Expressions Tree processor Pépinière Equivalent expression tree ? True/False save XM L Analysis grid XM L Student’s reasoning loaded Student’s reasoning+ assessment

16. Diagnosis algorithm  Numerical or algebraic approach?  Loop on each expression of the student’s reasonning  Build the expression tree (ST)  Loop on each Plausible solution in the analysis grid Build the expression tree (PT) If numerical approach -substitute the numerical value in PT If ST  PT -keep :PT, the rule and the comment and stop  At the end  walk through PT to set up the final assessment  save the final assessment, the comment and the applied rules 16

17. Results and tests  On going work  A demonstration prototype implements a complex exercise cloning  Authoring clones  Solving  Diagnosing  Preliminary Tests  assessment of a corpus of 141 students’ solutions Multi-step reasoning Multiple equivalent reasonings  3 teachers tested it in the lab 17

18. Discussion  Diagnosis  compared with model tracing ≈Tree of plausible steps: correct and incorrect rules ≠Emphasis : whole reasoning/step-by-step ≠Several student’s types of reasoning derived from a single solution branch ≠Multidimensional assessment  Authoring  Filling template forms - Limited to specified exercises  Automatic multidimensional diagnosis validated by experts  No programming, no modeling for teachers 18

19. Automatic Multi-criteria Assessment  Our proposal  Teachers clone a diagnosis tool previously designed by experts  The cloning process relies on A preliminary educational study in the domain An implementation of templates of diagnosis exercises A specific application to analyze reasonings that are not pre-formated  Demo: Friday afternoon  http://pepite.univ-lemans.fr 19