1. My talk describes how the detailed error diagnosis and the automatic solution procedure of problem solving environment T-algebra can be used for automatic assessment of student solutions

2. Content 1.What is T-algebra - interactive solution environment with three- stage solution steps, - automatic solver for demo solutions and step hints - detailed error diagnosis 2.How to add assessment facilities - positive part - penalties - evaluation of economy

3. Automatic assessment possibilities in problem solving environment with detailed diagnosis of errors Rein Prank University of Tartu rein.prank@ut.ee

4. What is T-algebra T-algebra is an interactive problem solution environment for Basic School algebra: calculation of the values of numerical expressions; operations with fractions; solving of linear equations, inequalities and linear equation systems; operations with monomials and polynomials. Under development in University of Tartu, 2004-… More than 50 problem types implemented

5. T-algebra environment In T-algebra each solution step consists of 3 sub-steps: 1) selection of the operation from the menu, 2) marking the operand(s) in expression, 3) entering the result of the operation. T-algebra requires precise selection of operands First two substeps are “mixed together”

6. Input modes in T-algebra For entering the result the program has three input modes: 1. Free, 2.Structured, 3.Partial Input mode for each task is fixed in task file

7. What can we diagnose in A-O-I-interface? (1) Application of selected operation is impossible does not correspond to the algorithm In current version of T-algebra first two sub-steps are checked together

8. What can we diagnose in A-O-I-interface? (2) Selection of operands Marked term is not a syntactically correct expression, Marked term is not a proper subexpression (order of operations misunderstood), Marked term has not the form required for selected rule, Operands do not satisfy the compatibility requirements (are not like terms, etc.), Operands do not satisfy the location requirements (do not belong the same sum, fraction, product etc.)

9. What can we diagnose in A-O-I-interface? (3) Entered subexpression (result of conversion) is not a syntactically correct expression, should be preceded by a sign, should be put in parentheses (order of operations), has not the structure required for selected rule and marked operands (not a monomial, wrong number of members etc), is not equivalent with marked part, concrete parts of Input do not have right value selected operation with marked terms is not performed (nothing reduced, terms are not moved to other side, …)

10. Automatic solver The program contains an automatic Solver. The solution steps are applications of T-algebra rules The Solver implements for each problem type the same (textbook) algorithm that the students are expected to use Output is solution (not only the answer)

11. Components of assessment 1.Positive component – how far the student has progressed in creating the solution 2.Negative component – mistakes and use of help 3.Economy component

12. Assessment - positive component (A) A. Static approach (phases of algorithm): 1)The teacher assigns weights to each phase in the “official” solution algorithm for particular problem type 2)T-algebra is able to determine what is the first phase that was not yet completed 3)In addition, it is possible to know from Solver’s solution, what phases of the algorithm did not contain steps in case of the actual task Problem: many elementary task types have algorithms containing only 1-2 phases (combine like terms)

13. Assessment - positive component (B) B. Dynamic approach (T-algebra steps): The program can count a) the number of steps in the full solution created by Solver b) the number of steps neccessary for completing the student’s solution Problem: applications of different rules have different and depending on actual expressions complexity

14. Assessment - negative component T-algebra 1)classifies student errors in 20 categories 2)counts separately the cases of asking help for choosing the rule, for marking the operands, for input and for additional operations The negative component can be calculated by assigning penalties for each of those categories. More detailed assessment of different aspects of skills/mistakes is possible, using directly the results of diagnostics (in modules of rules)

15. Assessment - economy component? The numbers of steps in student’s and Solver’s solution can be compared. But 1)the steps made with different rules of T- algebra are not of the same weight, 2) T-algebra makes in one step calculations (addition, multiplication, …) with arbitrary number of numbers, 3) T-algebra applies the rules mainly in fixed order and sometimes it is quite easy to solve with smaller number of steps.

16. Conclusions Detailed error diagnosis and automated solution facilities of T-algebra create good potential for adding automated assessment without complex programming. There are obvious possibilities to make the assessment of usual positive and negative aspects of solutions adjustable by the teacher Current version of T-algebra is not very well suited for evaluation of economy of solution