WP5: Validation Anne De Roeck Diane Evans The Open University, UK

WP5 Who? Objectives Challenges & Risks Scenario Based Evaluation Why now? Progress Plan

Who? All Partners Open University (WP5 Lead) –Distance: 200,000 students – 90% on-line –Research and Teaching OU Centre for Research in Computing –CRC = Computing + KMi + IET UserLab –NLP: Anne De Roeck; Donia Scott, … –E-learning: Josie Taylor; Patrick McAndrew; Diane Evans, …

WP5 Objectives Aim: –to validate if eLearning systems (LMS) are improved by adding new functionalities based on language technology tools and ontology-based (multilingual) retrieval. Objectives –to assess the extent to which the integration of these new functionalities affects the effectiveness of the eLearning process.

Challenges Focus: –Quantitative vs Qualitative Evaluation –Metrics vs User Experience –Functionalities vs Software Development

Challenges Focus: –Quantitative vs Qualitative Evaluation –Metrics vs User Experience –Functionalities vs Software Development

Challenges Focus: –Quantitative vs Qualitative Evaluation –Metrics vs User Experience –Functionalities vs Software Development Challenge 1: Methodology: –Scenario based validation –Functionality & user context centred –User questionnaires & interviews

Challenges Focus: –Language Technology Tools –Ontology retrieval –Improve effectiveness! Challenge 2: Search & Retrieval Scenarios –Embed functionality (teachers & learners?) –Establish base-line –Refine scenarios

Challenges Focus: –Multi-lingual experience Challenge 3: Multilingual Scenarios –Multiple communities –Multi-lingual scenario –Chart the territory!

Risks Content! Assessment?? Performance of Components? Interface vs Functionality Cross-cultural issues

Risks Content! Assessment? Performance of Components? Interface vs Functionality Cross-cultural issues Technical deliverable vs user requirements

Methodology User Scenarios –‘a story focused on a user or group of users, which provides information on the nature of the users, the goals they wish to achieve and the context in which the activities will take place’.

Methodology User Scenarios –Step beyond Software Development Use Cases –Qualitative evaluation of functionalities … –…in context of user roles –Accessible to all stakeholders –Developed from high level descriptions into scenario instantiations during implementation –Used in learning applications (eg mobile learning) & Software Development

Why now? Functionalities & System development LT4eL = New technology Co-evolve system & validation –Process of scenario instantiation: –Explore/refine ideas on possible usage –Clarify user requirements –Establish common ground between pedagogists, developers, NLPers, users –Concurrent with design & development

Progress Aim: –Prepare & support user scenario validation –Clarify objectives –Mitigate risk Background literature search and survey: –Keyword extraction & validation methods –Ontologies and deployment in search –Scenario-based validation

Progress Contribution WP1: –Identifying and supplying English Content within the domain of e-learning in educational establishments. Contribution WP2/3: –running early version of keyword extractor with English content and feeding back to WP2 leader. –liasing with Malta on English content –identifying qualitative validation tests for Keyword Extractor application.

Progress Start WP5: –Monitor content (ongoing: high risk!) –Draft validation plan for ILIAS + new functionality –Delivered high level user scenario (starting point of instantiation) –Preliminary Questionnaire (establish qualitative base-line)

Progress/Plan Questionnaire: –slide presentation supporting 'focus group' –questionnaire (in Word format) covering search motivation and behaviours –81 individual student responses from 6 partners

Progress/Plan Questionnaire Findings on: –Reasons to search in non Native language –Difficulties encountered when searching –Selecting search criteria –Handling search results –Other issues

Plan Process Questionnaire Challenge2 –Refine search & retrieval scenarios (Workshop – March) –Integration (WP4) Challenge 3 –Multi-lingual dimension to functionalities –Uncharted territory !

Plan 0. Preliminary Questionnaire (M12) - Partners 1. Pilot Validation (M16-18) – OU 2. Main Validation (M19-23) – Partners (ILIAS Community part) 3. Final Validation (M26-28) Reports M24, M30