1. Bridging the Gap between Practitioners and E-learning Standards: A Domain-specific Modeling Approach Yongwu Miao, Tim Sodhi, Francis Brouns, Peter Sloep, and Rob Koper Centre for Learning Sciences and Technologies Open University of The Netherlands ECTEL08, Maastricht, Sep. 16-19, 2008

2. Overview Problem and Goal Domain-specific Modelling Peer Assessment Peer Assessment Modelling Language Peer Assessment Modelling Tool Summary and Future Work

3. Problem and Goal Developing a complicated Unit of Learning (UoL) using IMS LD constructors (e.g., roles, learning activities, properties, and conditions) is difficult and even impossible for ordinary practitioners, because the required level of pedagogical knowledge and technical knowledge is significant. Enabling ordinary practitioners to create pedagogically sound and technically executable Uols by providing pedagogy-specific modeling languages

4. Domain-specific Modeling What is Domain-Specific Modeling (DSM) or Domain-specific language (DSL) DSM is a kind of model-driven approach to develop software applications. It raises the level of abstraction beyond programming by specifying the solution in terms of concepts and associated rules culled from the very domain of the problem being solved. The final software products are generated from these high-level specifications. Why to Use DSM DSM are more expressive than and therefore tackle complexity better, making modeling easier and more convenient. More importantly, they allow automatic, full code generation, similar to the way today's compilers generate Assembler from a programming language like JAVA.

5. What are differences between DMS and other approaches different users different languages different abstraction levels Domain-specific Modeling LD & QTI model Map to LD & QTI interpret

6. Domain-specific Modeling How to adopt DSM approach to develop software Developing a meta-model Develop executable components Developing mapping functions to transform domain model to components

7. Peer Assessment Definitions: Peer assessment is a process whereby groups rate their peers (Falchikov 2001) Peer assessment is an arrangement for peers to consider the level, value, worth, quality or successfulness of the products or outcomes of learning of others of similar status (Topping, Smith, Swanson, & Elliot, 2000). Peer assessment models: Pure peer assessment (two learners do assignment and then assess the outcome of his/her peers) With additional stages (design assessment and react to feedback) Group/collaborative assessment (more candidates and reviewers) With involvement of the tutor Different types of assignment tasks Different distribution patterns ……

8. design assignment writing article1 reviewing article2 improving article1 writing article2 reviewing article1 improving article2 final assessment2final assessment1 learner1learner2tutor article2 feedback1 improved a2 paper1 article1 feedback2 improved a1 paper2 Peer Assessment A peer assessment example tutor’s feedback1tutor’s feedback2 assessment form1assessment form2

9. Peer Assessment A typology of peer assessment (Topping 1998)

10. Peer Assessment Modelling Language Four kinds of stages In each stage, people with a certain role perform activities In each activity, there may be input and output documents

11. Peer Assessment Modeling Language Guide to make design decisions by showing guidance and consequence Make decisions by selecting default or pre-defined options

12. Peer Assessment Authoring Tool A peer assessment authoring tool will be based on the peer assessment modeling language (meta-model). Users will be guided to specify a peer assessment model by making a series of design decisions. The consequences of the decision-making will be presented as a diagram. The tool is underdevelopment. The design and use of the tool will be explained (in a mock-up UI) by using an example

13. File Edit View Tool Template Help docroleservicelanestage Peer assessment Here shows the guidance to make design decisions State bar user’s choices (decision options) Specify role structure, document, and service Show and edit peer assessment model

14. File Edit View Tool Template Help docroleservicelanestage Peer assessment How many stages in the peer assessment process? Please make a decision user’s decisions Default (three stages) With design assessment stage With additional feedback stage More complicated setting

15. File Edit View Tool Template Help docroleservicelanestage Peer assessment How many roles will be involved? Five stages have been defined design assessmentdo assignmentgive feedback react to feedbackgive feedback 2 Default (two persons) With the involvement of tutor More complicated setting

16. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to define role structure? reviewerRdesignerRcandidateR Default role setting and with the involvement of a tutor learner1RtutorRlearner2Rlearner1Rlearner2RtutorR

17. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to define a typical activity for each stage ? reviewerRdesignerRcandidateR Please make a decision learner1RtutorRlearner2Rlearner1Rlearner2RtutorR Default setting More complicated setting

18. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to define a typical activity for each stage ? reviewerRdesignerRcandidateR Nine activities are defined and roles are assigned learner1RtutorRlearner2Rlearner1Rlearner2RtutorR learner1 learner 2 learner1learner2tutor

19. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Which stages will the tutor be involved? reviewerRdesignerRcandidateR Please make a decision learner1RtutorRlearner2Rlearner1Rlearner2RtutorR learner1 learner 2 learner1learner2 Give feedback 1 Give feedback 2 More complicated setting tutor

20. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Which stages will the tutor be involved? reviewerRdesignerRcandidateR The tutor is assigned to work in the stage of “give feedback 2” learner1RtutorRlearner2Rlearner1Rlearner2RtutorR learner1 learner 2 tutor

21. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to create documents ? reviewerRdesignerRcandidateR Please make a decision learner1RtutorRlearner2Rlearner1Rlearner2RtutorR Default setting More complicated setting learner1 learner 2 tutor

22. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to create typical documents ? document are created learner1 learner 2 tutor feedback1.1Dfeedback1.2Doutcome1Doutcome2Dassignment1Dassignment2Dimproved outcome1Dimproved outcome2Dfeedback2.1Dfeedback2.2D

23. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Could you please change the titles of documents and define documents? document are created learner1 learner 2 tutor feedback1.1Dfeedback1.2Doutcome1Doutcome2Dassignment1Dassignment2Dimproved outcome1Dimproved outcome2Dfeedback2.1Dfeedback2.2D finished

24. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Could you define document? document are created learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2D finished learner 2 Review Form How do you think about this article? accept accept with minor/major modification? reject Comments: assessment form assessment editor assessment formD

25. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Could you define document? document are created learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2D finished assessment formD

26. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 How will the documents be distributed? Please make a decision learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2Dassessment formD Default setting More complicated setting

27. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 How will the documents be distributed? documents are distributed by default learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2Dassessment formD

28. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 How will the documents be distributed? doc-flow are customized learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2Dassessment formD

29. Summary and Future Work Summary: Outline an approach to apply the domain-specific modelling paradigm to the task of the development of peer assessment Develop a peer assessment modeling language Propose a method to enable the end-user to model peer assessment processes through making a series of design decisions Future work: Develop executable components Design and implement mapping algorithm Implement the tool and conduct evaluation

30. Thanks for your attention! Questions?