2. Virtual presentation. Session: e-Learning, Training, Evaluation and Assessment TEACHERS’ CORRECTION ACTIONS OF FREE WORDED EXERCISE SOLUTIONS FOR MODELING A TEACHER ASSISTING FEEDBACK TOOL Dr. Sonja Ruda Chemnitz University of Technology, Germany Communication on Technology

3. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 1. INTRODUCTION Exercises 1) preset solution possibilities, e.g. multiple choices 2) solution text drafted by the students themselves

4. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 1. INTRODUCTION Exercises 1) preset solution possibilities, e.g. multiple choices  student can only guess the answer 2) solution text drafted by the students themselves  students entirely have to resort to their own knowledge

5. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Marking self worded answers solutions requires of the teachers  special knowledge,  much time and  energy.

6. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Marking self worded answers solutions requires of the teachers  special knowledge,  much time and  energy. Automatic correction: Teaching and learning systems can identify errors simply as formal deviations.

7. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Some problems 1) Ambiguous word combinations: A false solution is though assessed as “correct”. System:What is a ‚turn-taking‘? Student:The previous hearer gives the word to a previous speaker. System: Correct! The previous speaker gives the word to a previous hearer.

8. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Some problems 2) The teacher does not consider correct alternatives, right solutions can nevertheless be marked as “wrong”. System:What is a ‚turn-taking ‘? Student: Change of speakers. System: Unfortunately wrong.

9. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Some problems 3) Student’s answer: “not completely correct”, the corresponding text passages cannot be marked exactly.

10. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Some problems 4)Sample solutions: The teacher has no control whether the student compared the solutions correctly.

11. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Some problems 5)Feedback: “Your answer is unfortunately wrong”. “Congratulations, you get 3 points out of 3!”

12. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda All this demonstrates  technical shortcomings  enormous implementation expenditure of a differentiated feedback  costs

13. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Teacher assisting feedback tool for marking free worded exercise solutions  Combination: advantages of free worded exercise solutions, advantages processed by the computer  Analysis: Teachers’ correction actions and their knowledge of free worded exercise solutions

14. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Some questions  Which links are there between tasks and solution text?  How is the comparison between the lecturer’s and student’s solution text made?  How are mistakes recognized, analyzed, corrected and assessed?  Which actions are performed by all lecturers?  Which actions could be performed by the computer?

15. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 2. METHODOLOGY What do teachers, when correcting free worded exercise solutions?

16. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Text corpus:  offline university examination questions  self worded answers with at least one word, communicative-pragmatic approach (Brinker, 2005, p. 18) ’think aloud method’ (van Someren, Barnard & Sandberg, 1994) ’concurrent record analysis’ and ’focused interview’ (Karbach & Linster, 1990, pp. 84 and 87)

17. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda  14 lecturers produced 15 verbal records  analytical transcripts  each revised between three and seven papers  71 papers from 13 special fields with 488 corrected exercises including subtasks  The tape recordings: from about nine minutes up to almost two hours. The average recording amounted to one hour.

18. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Pragmalinguistic analysis ‘Speech Act Theory’ Austin, J. L. (1962). How to Do Things with Words. Cambridge, Massachusetts: Harvard University Press. Searle, J. R. (1969). Speech Acts. An Essay in the Philosophy of Language. Cambridge: Cambridge University Press.  Brinker, Holly, von Polenz

19. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student.

20. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. A task definition is limited by restrictive attributes, e.g.: Describe briefly...

21. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. A task definition is limited by restrictive attributes, e.g.: Describe briefly... Task definitions are dominated by action predicates, e.g.: mark.

22. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. A task definition is limited by restrictive attributes, e.g.: Describe briefly... Task definitions are dominated by action predicates, e.g.: mark. Topic development patterns → “Help”

23. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Teachers’ correction actions of free worded exercise solutions

24. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Teachers’ correction actions of free worded exercise solutions 1) Grasping the complete student’s solution attempt or filtering the propositions to be dealt with 2) Comparison with the expected solution 3) Assessment of the student’s solution attempt with regard to the definition of the tasks 4) Total assessment of student’s solution and 5) Total assessment of all present solutions of a student.

25. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Potential correction actions  writing down expected solutions,  compiling a criteria grid and  determining the approach – paper by paper or exercise by exercise

26. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Individual assessment of a student’s exercise solution

27. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Individual assessment of a student’s exercise solution 1) Envisioning of the exercise definition 2) Envisioning of the connection between exercise definition and solution 3) Expressing the first impression 4) Grasping the student’s solution attempts 5) Comparison with the expected solution 6) Assessing individual parts of the student’s solution attempt and 7) Correcting.

28. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Problematic and vague solution attempts

29. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Problematic and vague solution attempts problem solving strategies/(speech) acts, e.g.: They  PARAPHRASE from the student’s point of view,  INTERPRET,  SUMMARIZE,  EXPLAIN,  SPECIFY own solution texts,  ELUCIDATE the task definition, and  search for parts of the solution text which may still be considered as correct,  use aids like professional literature,  ASK colleagues/superiors and  MAY MAKE COMPROMISES.

30. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda 4. TEACHER ASSISTING FEEDBACK TOOL  oriented towards free worded exercise solutions  standardized  brief  without grafics  „Borland Delphi 5.0“  Prototype

31. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

36. Rule Assistant (Boole’s operators) AND (UND): Which words/phrases must be included in the solution?

37. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda OR (ODER) and PREFERRED (BEVorzugt): Please enter acceptable alternatives to the required words/phrases. If you prefer certain alternatives, please tick them. (BEV-Operator)

38. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda OR POSSIBLE (ODEREVTL): Now enter solutions that you would accept even though they are not completely correct. AND POSSIBLE (UNDEVTL): Now enter solution text parts that may be included additionally. POINT (PLUS): When would a solution text be particularly good?

39. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda NOT (NICHT): Which words/phrases must not be included in the solution? What would not be completely wrong but can nevertheless not be accepted by you? What would be a serious mistake?

40. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

47. 5. CONCLUSION Feedback tool  supports teachers in their correction work  by making arrangements with them regarding the solution texts and assessments  which requires the acquisition of certain rules and initial training  An automatic correction of all solution texts cannot and should not be expected!

48. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda Current works the use of a semantic word net in order to e.g. capture synonyms and statistical data.

49. Thank you for your attention! Contact: Dr. Sonja Ruda Chemnitz University of Technology Communication on Technology D-09107 Chemnitz sonja.ruda@phil.tu-chemnitz.de