WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 WP4: Pilots with the Integrated System Validation of the Project UPF 8, ALTRAN 7, LogicaCMG 3, OUNL 3, GIUNTI 5, CERTH 4, SU 4, SYN 2 Dai Griffiths, Josep Blat, UPF

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Aims of the (integration) activity To provide realistic scenarios and corresponding pilots which constitute an effective environment for evaluation of the project, whereby the systems resulting from the research and development can be challenged and validated To provide the overall project evaluation strategy

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Tasks, deliverables, timetable (I) T1. Define initial pilot scenarios (m6) T2. Define the pilot evaluation plan (m6) T5. Develop an overall evaluation plan (m6) D4.1: Report (m 6) on: –scenarios and pilots definition –user requirements and validation criteria for the overall project –evaluation strategies and initial evaluation plan implementation

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Tasks, deliverables, timetable (II) T3. Set up the initial pilots (m6-12) T4. Validate the initial system through the evaluation of the Digital Cinema pilot (m12-18) T6. Generate user requirements for cycle-2 pilots (m12-18) D4.2: Report on results of first evaluation of pilots (month 18) –Suggested interim report on m12 to submit for the review

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Pilots envisaged Pilot 1 Training for digital movie production (UPF) Pilot 2 Training for continuous education of health workers (UPF) Pilot 3 Unesco-IHE – Institute for Water Education (?) Pilot 4 The Antwerp Lifelong Learning City (?) –Request an initial description

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Pilots (I) T1. Define initial pilot scenarios (up to m6) –Pilot 1: Digital movie production Partners (?) –Visual effects and other UPF postgraduate courses –Others from possible partners: Mediapro, Brainstorm, Thomson, Barco, XDC, Filmlight, DVS, Pandora (MediaSalles,...) –Commitments definition: elaborate on association document User requirements, validation criteria, evaluation Target groups definition: learning providers, learning designers, content, tutors, learners... availability: STRESS THE IMPORTANCE OF A COMMUNITY OF USERS Plan for the implementation: define competencies based syllabi,... INSERT HERE MATERIAL FROM IP-RACINE – DISCUSS TRANSFORMATION INTO COMPETENCE ORIENTED – DISCUSS THE CHALLENGES POSED BY THE PILOT –Establish a model for the other pilots

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Pilots (II) T1. Define initial pilot scenarios (up to m6) –Pilot 2: Training for continuous education of health workers Partners (?) Hospital Cl í nic, Doyma (others) Gastroenterology/hepatology, Pediatrics, Colo-rectal cancer screening,... User requirements, validation criteria, evaluation Target groups definition: learning providers, learning designers, content, tutors, learners... availability

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Pilots (III) T1. Define initial pilot scenarios (up to m6) –Pilot 3: Unesco-IHE – Institute for Water Education ?

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Pilots (IV) T1. Define initial pilot scenarios (up to m6) –Pilot 4 The Antwerp Lifelong Learning City ?

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Evaluation planning and initial implementation T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6) T5. Develop an overall evaluation plan (up to m6) Validation focus along cycles –'proof of concept' during the first cycle through the Digital Cinema pilot –'usability' during the second cycle through Health Workers pilot, the UNESCO-IHE pilot, the Antwerp Lifelong Learning City pilot, and an extension of the Digital Cinema pilot. –' wider applicability and sustainability' during the final cycle.

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Evaluation planning (I) T5. Develop an overall evaluation plan (up to m6) –objectives addressed (including gender issues), –criteria and indicators: quality criteria effectiveness in context technical criteria components and functions to be assessed methodologies in context of the project the different actors and “ reference users ” in the process and their tasks the coordination and procedures, defining a timeline for evaluation

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Sample evaluation grid from TA Activities Aspect RTD activitiesTechnical Design & Implementation activities Pilots & Validation activities experimentsExperimentspilotsdemonstrators ObjectiveTechnical verification and validation of the functional and usability aspects of the tools Verification of the integration and the interoperability between the components Validation of the environment integrated and of the individual components. Focus: - functionality - operationally - usability Validation of the environment integrated and of the individual components. Focus: - sustainability Feeds intoThe concerning activities Technical Design & Implementation + Aspect RTD activities Requirements & Analysis activities Period‘continuously’ after completion of each integration loop: 3 times after the conclusion of the experiments of the first two integration loops: 2 times. Round 1: small scale Round 2: large scale after the conclusion of the experiment of the final integration loop: 1 time

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 On users/actors and methodologies Users / actors –learning providers, learning designers, tutors, learners... (different from different areas?) Methodologies to be used: –Ethnographic study, Interviews, Focus groups, Questionnaires, Observations, Experiment, Peer review, Think-alouds, User diaries, System log data, Textual data, Cost analysis, Reviews, Usability inspection,... Components / areas (to be defined for a good plan)

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Evaluation planning in detail T5. Develop an overall evaluation plan (continued) –more detail needed for planning and initial implementation –evaluation contact in each organisation – not only this WP partners –define components to be evaluated; provide initial evaluation sheet for each component (model provided) –Initial scenarios for pilots probably needed to plan for the whole period –UPF in charge of coordination and assistance, every partner responsible of carrying out their own evaluation, evaluation committee to supervise overall evaluation and validation –Everybody to send a list of components, and an evaluation sheet for each component

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Evaluation sheet model Component to be evaluated Areas to be assessed Parameters to be assessed Actors / partners Role of the actors Evaluation procedure / material Starting date Immediate action (and date) Intermediate action (and date) End date Conclusion (general / for each area) Overall comments Evaluation committee comments

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Pilot evaluation plan and implementation T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6) –Define an implementation plan for the first pilot –Model for the coming plans (which have to be initially defined)

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Deliverable reminder; contributions D4.1: Report (m 6) on: –scenarios and pilots definition –user requirements (?) and validation criteria for the overall project –evaluation strategies and initial evaluation plan implementation Contributions of the different partners (to be defined now for the first 6 months of the project) –OUNL ? –ALTRAN ? –LogicaCMG ? –GIUNTI ? –CERTH ? –SU ? –SYN ? –Anybody missing? Any contribution missing?

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Evaluation representatives (all partners) OUNL ? ALTRAN ? LogicaCMG ? GIUNTI ? CERTH ? SU ? SYN ? L3S ? INSEAD ? UB ? UvA ? SURF ? Evaluation committee?

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Reminder of coming tasks T3. Set up the initial pilots (m6-12) –Initial pilot on Digital Cinema; start earlier if possible to be able to validate; otherwise, make it a phased pilot, where validation can start early T4. Validate the initial system through the evaluation of the Digital Cinema pilot (m12-18) –Phased work or earlier start T6. Generate user requirements for cycle-2 pilots (m12-18) –On the basis of early work on pilots definition D4.2: Report on results of first evaluation of pilots (month 18) –Suggested interim report on m12 to submit for the review

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 Ourselves Dai Griffiths (WP manager) Josep Blat (in charge of DC pilot) Sergio Sayago (evaluation specialist) and others!

WP4 Presentation, TenCompetence Kick-off, Dec 8 th – 9 th 2005 By way of summary Evaluation, validation, user requirement activities are probably everybody ’ s task Components and overall evaluation/validation strategies to be defined: contributions expected from everybody; not an easy task Pilot 1 (Digital Cinema) to be planned and carried out during first period Pilots 2, 3, 4 to be planned during m12-18; but initial scenarios important for the whole user reqs, validation, evaluation planning due by month 6. A second updated deliverable by month 12? And don ’ t forget the periphery of the project! Possible extra pilots, associates feedback...