1. STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK

2. STEM 2012 Aim To extend the capabilities of e-assessment systems (online tests) to include questions that require students to draw free-form diagrams in answers. In particular: –Minimal constraints on what might be drawn –Not multiple choice –Support for a variety of diagram types –Automatic marking –Moodle VLE (via OpenMark)

3. STEM 2012 Example Student draws diagram in response to question

4. STEM 2012 What is a diagram? Free-form –Not free-hand –Uses a drawing tool Specialised for the domain by the teacher As unconstrained as the teacher wishes Graph based –Shows relationships between objects

5. STEM 2012 Biological Flow Diagram Illustrates the flow of hormones between glands.

6. STEM 2012 Entity-Relationship Diagram Shows the relationships (links) between entities (boxes)

7. STEM 2012 Use case diagram (UML) Shows the actions that an actor Book Borrower takes part in when using a library

8. STEM 2012 Spray Diagram Illustrates how UK farms have diversified in recent times.

9. STEM 2012 Spatial Relationships It is the relative positions of boxes that determines whether a relationship exists. Adult and Child are inside Person – inside indicates subtype relationship..

10. STEM 2012 Sequences time The messages, represented by arrows, are sequenced in time.

11. STEM 2012 Question in VLE test This is what the user sees in their browser

12. STEM 2012 Run the Drawing Tool Up pops the drawing tool – an applet

13. STEM 2012 Automatic marker provides feedback Feedback from the automatic marker

14. STEM 2012 Get Answer

15. STEM 2012 Automatic Marking The students diagram is compared with the model solution(s) to see how well the objects and the relationships match. Similarity measures of how well each element on the student diagram matches with an element on the model solution. –A mixture of NLP and heuristics A marking scheme is applied. –Designed by the question setter from a template

16. STEM 2012 Does it work? – Experiment 1 394 diagrams from a real exam - marked out of 7. Difference between auto and human markers 00.51.01.52.03.5 Number 38941000 % 98.71.00.25000 Cumulative % 98.799.7100 Main corpus Diagrams only have entity types and relationships Published in 2011

17. STEM 2012 Comparison Human – Auto (Distribution of marks)

18. STEM 2012 Support Tools An authoring tool for teacher to design the question, the model solution(s) and the mark scheme. A specification tool to define a type of diagram. –Used to specialise both the drawing tool and the automatic marker A marking tool for marking sets of diagrams off-line.

19. STEM 2012 Authoring Tool for Questions, Model Solutions and Marking Schemes Question (text) Model Solution (diagram)

20. STEM 2012 Specification Tool A graphical user interface for defining a new type of diagram

21. STEM 2012 Summary A new question type for online tests Free-form diagrams (graph based) Automatic marking Implemented in Moodle Set of support tools

22. STEM 2012 Thank You p.g.thomas@open.ac.uk We always want more examples of types of diagram. Send me your suggestions. There is a website where you can try this technology out. Please contact me for details.