FREMA: e-Learning Framework Reference Model for Assessment David Millard Yvonne Howard Learning Technology Group University of Southampton, UK

Structure Overview Workshop: Eliciting Common Usage Patterns Introduction to FREMA What is a Reference Model? Views on the Model Do we have the right Personas – are you there? Do our Concept Maps make sense to you? Workshop: Eliciting Common Usage Patterns What are the most common tasks within assessment? What is (and is not) being addressed by current software? What do you consider to be the core elements of the assessment domain?

Introduction What is FREMA? What is the ELF? JISC funded Project between Southampton, Strathclyde and Hull Part of the e-Learning Framework (ELF) effort What is the ELF? Service Oriented Architecture for e-learning systems Layered Web Services (Domain services over Common Services) Dynamic and evolving FREMA will develop a Reference Model for the Assessment domain Supporting design-time activities Supporting run-time activities Virtual organisations and lifelong learning What is a Reference Model? A description of how services behave within a particular domain A community resource

Assessment and ELF Not enough to describe and define these services Need a proper audit trail of decision making Start by defining the domain Work up through the services to a reference implementation This is an ELF Reference Model

Anatomy of a Reference Model Domain Definition Overview of the domain, and how projects and standards fit within it Identifying Common Usage Patterns Scoping the FREMA Project Developing Use Cases Formal descriptions of usage patterns Gap Analysis Mapping of Use Cases to the Services in ELF Reference Impl’ Service Profiles Formal descriptions of those services Service Profiles Gap Analysis Reference Implementation Of key/core services Examples Validation Resource Use Cases Common Usage Patterns Assessment Domain Definition

Assessment Domain Definition What does it look like? An evolving, cross-referenced, searchable web site Indexed resources and narrative descriptions of the domain UML Use Cases and Scenario documents Service descriptions, narrative and WSDL Service implementations to download (Java/.NET) Different gateways into the model according to how you want to use it Reference Impl’ Service Profiles Gap Analysis Use Cases Common Usage Patterns Assessment Domain Definition

Assessment Domain Definition How might you use it? Use the Reference Implementation Build on some or all of the developed services Use the Service Profiles To develop your own services that will fit into the framework Use the Use Cases To help understand usage patterns within the domain Develop new Service Profiles and thus Services Use the Domain Definition To develop a context for your own work Understand how existing work fits together Identify standards Locate experts Reference Impl’ Service Profiles Gap Analysis Use Cases Common Usage Patterns Assessment Domain Definition

Use Case for a Service Oriented Reference Model (SORM)

A More Concrete Description An ontology for the resources of the domain A knowledge base of the resources Concept Maps that define the domain and aid searching Processes (think of verbs) Classes of Objects (things, think of nouns) And many more gateways into resources Visualisations of Concepts Populations of resources over concepts Searchable, flexible, dynamic website

Process concept map (verbs)

Classes concept map (nouns)

Another look at the SORM Use Case How would these actors choose to view the domain?

Personifying the actors Will, Web services developer Scenario: ‘I want to lookup use cases and scenarios to help me design my application. This will help me to define my footprint in the assessment domain. I see there are some web services I could download but some are missing. What standards can I use when writing my own web services to ensure that I can interoperate with the web services I’ve chosen?’ Yvonne, Institutional Resource Manager Scenario ‘I want an overview of what this domain is all about. I want to know what standards are applicable in the domain to ensure that we comply with quality assurance requirements. I want to examine use cases and scenarios to understand the available footprints. I also want to know who the key players are and what the key projects are.’ Other Actors Early Adopter Toolkit developer Course developer And there are others . . .

Concept modelling What mental models might these and other actors use to orient their searches? How do they visualise the assessment domain?

Structure Overview Workshop: Eliciting Common Usage Patterns Introduction to FREMA What is a Reference Model? Discussion Do we have the right Personas? Do our Concept Maps make sense? Workshop: Eliciting Common Usage Patterns What are the most common tasks within assessment? What is (and is not) being addressed by current software? What do you consider to be the core elements of the assessment domain?

Another concept map – based on a domain expert view Items Validation Final Grade Structuring Assessment Marking Grade book Multiplicity Self assess Mark Learning Peer Group Student Feedback Evidence Tracking Artefacts Course Group Member Transcript Schedule Delivery Plagiarism Authorise Authenticate Student Appeal Help Authoring QA Workflow Failover