Davinia Hernández-Leo, Eloy D. Villasclaras, Juan I. Asensio-Pérez, Yannis Dimitriadis GSIC / EMIC group University of Valladolid, Spain TENCompetence.

Slides:



Advertisements
Similar presentations
By: Edith Leticia Cerda
Advertisements

Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
June 20, Your Presenters  Ken Mattingly – Rockcastle County  Jessica Addison – Todd County/KDE.
Episode 3 / CAATS II joint dissemination event Lessons Learnt Episode 3 - CAATS II Final Dissemination Event Philippe Leplae EUROCONTROL Episode 3 Brussels,
Group of teachers and PhD Students who teach Research Methods in Education using ICT and study them. At this moment, we have different research projects.
Chapter 2 – Software Processes
Dr Jim Briggs Masterliness Not got an MSc myself; BA DPhil; been teaching masters students for 18 years.
Introduction to Research Methodology
Characteristics of on-line formation courses. Criteria for their pedagogical evaluation Catalina Martínez Mediano, Department of Research Methods and Diagnosis.
GRIDCOLE: a Grid Collaborative Learning Environment M.L. Bote-Lorenzo, L.M. Vaquero-González, G. Vega-Gorgojo, J.I. Asensio-Pérez, E. Gómez-Sánchez, Y.A.
CAP 252 Lecture Topic: Requirement Analysis Class Exercise: Use Cases.
COLLAGE, a Collaborative Learning Design Editor Based on Patterns COLLAGE, a Collaborative Learning Design Editor Based on Patterns Davinia Hernández-Leo.
CLD WC CW Cohere Patterns and PLs Designs Tools, resources and other mediating artifacts to be used at the design subcycle of the OER cycle CompendiumLD.
CSCL Scenarios based on Combinations of CL Flow Patterns: A Case Study Davinia Hernández-Leo GSIC / EMIC group University of Valladolid, Spain OTEC, OUNL.
Four Dark Corners of Requirements Engineering
(Computer Supported) Collaborative Learning patterns Yannis Dimitriadis University of Valladolid, Spain EMIC/GSIC research group
COLLAGE, a Collaborative Learning Design Editor Based on Patterns COLLAGE, a Collaborative Learning Design Editor Based on Patterns Davinia Hernández-Leo,
Research Methods for Business Students
Project Workshops Results and Evaluation. General The Results section presents the results to demonstrate the performance of the proposed solution. It.
Combining Perceptions and Prescriptions in Requirements Engineering Process Assessment Presented By: Hector M Lugo-Cordero, MS EEL
Damian Gordon.  Summary and Relevance of topic paper  Definition of Usability Testing ◦ Formal vs. Informal methods of testing  Testing Basics ◦ Five.
User Experience Design Goes Agile in Lean Transformation – A Case Study (2012 Agile Conference) Minna Isomursu, Andrey Sirotkin (VTT Technical Research.
User requirements for UK e-Science grid environments Bruce Beckles University of Cambridge Computing Service.
Teachers have a significant role in developing and implementing the most effective teaching and learning strategies in their classroom and striving for.
The Knowledge Resources Guide The SUVOT Project Sustainable and Vocational Tourism Rimini, 20 October 2005.
Professional Growth= Teacher Growth
An investigation of the impact of student support initiatives on the retention of computer science students Clem O’Donnell 1, James Murphy 2, Abdulhussain.
Evaluation and analysis of the application of interactive digital resources in a blended-learning methodology for a computer networks subject F.A. Candelas,
VIRTUAL BUSINESS RETAILING
Introducing small-group workshops as formative assessment in large first year psychology modules Suzanne Guerin School of Psychology, University College.
Problem Based Learning (PBL) David W. Dillard Arcadia Valley CTC.
Interuniversity Center for Educational Research and Advanced Training Paolo Tosato, Juliana Raffaghelli European Distance and E-Learning Network Teachers’
RESEARCH A systematic quest for undiscovered truth A way of thinking
© L.A.C.E. Research Group, 2003 University of Cadiz Eudoxos Project Teaching Science with a Robotic Telescope EVALUATION OF THE EUDOXOS PROJECT The evaluation.
UNIVIRTUAL FOR INSTRUCTIONAL DESIGN Versione 00 del 29/07/2009.
Chapter 8 Architecture Analysis. 8 – Architecture Analysis 8.1 Analysis Techniques 8.2 Quantitative Analysis  Performance Views  Performance.
The Audit Process Tahera Chaudry March Clinical audit A quality improvement process that seeks to improve patient care and outcomes through systematic.
Dissertation Theme “The incidence of using WebQuests on the teaching-learning process of English Foreign Language (EFL) for students attending the seventh.
A Model to Facilitate Effective Blended E-learning within Universities in Developing Countries B. Aguti, R. J. Walters, G. B. Wills Electronics and Computer.
SiTEL LMS Focus Group Executive Summary Prepared: January 25, 2012.
INTERNATIONAL LABOUR ORGANIZATION Conditions of Work and Employment Programme (TRAVAIL) 2012 Module 13: Assessing Maternity Protection in practice Maternity.
EVALUATING PAPERS KMS quality- Impact on Competitive Advantage Proceedings of the 41 st Hawaii International Conference on System Sciences
SOFTWARE DESIGN.
What is a Business Analyst? A Business Analyst is someone who works as a liaison among stakeholders in order to elicit, analyze, communicate and validate.
2007 © Giunti Labs – No reproduction without written permission Visualizing IMS Learning Design for easier editing Marco Luccini, Giunti Labs R&D Project.
Cluster Management Scorecard FITT (Fostering Interregional Exchange in ICT Technology Transfer)
Lecture 7: Requirements Engineering
OVERVIEW PRESENTATION
Sharing Design Knowledge through the IMS Learning Design Specification Dawn Howard-Rose Kevin Harrigan David Bean University of Waterloo McGraw-Hill Ryerson.
Analyze Design Develop AssessmentImplement Evaluate.
Intel ® Teach Program International Curriculum Roundtable Programs of the Intel ® Education Initiative are funded by the Intel Foundation and Intel Corporation.
Chapter 2 – Software Processes Lecture 1 Chapter 2 Software Processes1.
Dept of Science and Technology Education, Faculty of Education
Working in Partnership
1 TOOL5100-CSCL Presentation of 2 papers by Jan Are Otnes Stahl (2000): A Model of Collaborative Knowledge-Building Stahl (2000): A Model of Collaborative.
The FDES revision process: progress so far, state of the art, the way forward United Nations Statistics Division.
Agenda Debrief on past module development Tools for online content development Module development template Timeline Suggested guidelines for developing.
An Agile Requirements Approach 1. Step 1: Get Organized  Meet with your team and agree on the basic software processes you will employ.  Decide how.
Oman College of Management and Technology Course – MM Topic 7 Production and Distribution of Multimedia Titles CS/MIS Department.
INFOMGP Student names and numbers Papers’ references Title.
User Interface Evaluation Introduction Lecture #15.
Company LOGO. Company LOGO PE, PMP, PgMP, PME, MCT, PRINCE2 Practitioner.
ICAJ/PAB - Improving Compliance with International Standards on Auditing Planning an audit of financial statements 19 July 2014.
Fifth Edition Mark Saunders, Philip Lewis and Adrian Thornhill 2009 Research Methods for Business Students.
Day 8 Usability testing.
Design for usability E6: Human Factors Design IB Technology.
DATA COLLECTION METHODS IN NURSING RESEARCH
Research Methods for Business Students
MASTER IN ENGLISH LANGUAGE AUTONOMOUS LEARNING ENVIRONMENTS 2013
Teaching with Instructional Software
Presentation transcript:

Davinia Hernández-Leo, Eloy D. Villasclaras, Juan I. Asensio-Pérez, Yannis Dimitriadis GSIC / EMIC group University of Valladolid, Spain TENCompetence Workshop Barcelona, June 22 nd Some findings from the evaluation of Collage authoring tool

2 Our approach (I) Challenges 1.Designing scripts that potentially elicits the desired interactions How can teachers create “potentially effective” “computer-interpretable” collaboration scripts according to the necessities of their situation? 3. Hiding computational representations of scripts to teachers 2. Formalizing the scripts so that they are computer-interpretable

3 Our approach (II) 1. Not easy to design a script that potentially elicits the desired interactions Reusing the “essence” of commonly used CSCL scripts (formulated as patterns) CLFPs, Collaborative Learning Flow Patterns: Broadly accepted techniques repetitively used by practitioners when structuring the flow of types of (collaborative) learning activities (E.g. Jigsaw, Pyramid (Snowball) CLFPs)

4 Our approach (IV) 2. Formalizing the scripts so that they are computer-interpretable IMS Learning Design (LD) IMS Learning Design (LD)

5 Our approach (V) 3. Computational representations are not familiar to educators Integrate in authoring tools “a pattern-based design process for CSCL macro-scripts computationally represented with IMS LD” which allows teachers to create their own computer-interpretable script by reusing CLFPs as visual LD templates that can be assembled and refined COLLAGE (COLaborative LeArning desiGn Editor) (Based on Reload Editor. IMS-LD level A compliant)

6 Evaluation methodology (I) Multicase study (Stake, 2005) Multicase study (Stake, 2005) Authentic experiences with different audiences (contextual issues, achievement of educational benefits, impact of software systems) Cross-case analysis Not aiming at accurate generalizations but at valid conclusions Mixed method combining qualitative and quantitative data gathering techniques (Martínez et al., 2003) Mixed method combining qualitative and quantitative data gathering techniques (Martínez et al., 2003) Triangulation: Comparative analysis of the data obtained with the different techniques

7 Evaluation methodology (II) Multicase study (Stake, 2005) QUITAIN: The proposed pattern-based design process for CSCL macro-scripts computationally represented with IMS LD ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements? Creating CSCL scripts based on CLFPs using Collage ISSUE: Can we use Collage for creating a script representing a scenario proposed by a third-party? Solving a third-party scenario ISSUE: Can we use CSCL scripts created with Collage in real situations ? Putting into practice a CSCL script created with Collage “Collage workshops” Case Study Lab. uni. of Valladolid and Cádiz, Spain Hands-on sessions Mixed method Questionnaires Observations Focus groups Mini-cases “Planet game” Case Study ICALT conference Pre-work Presenting Discussing Achieved CSCL script, papers, discussion “Network Managment” Case Study Two f2f and a distance session Lab. Uni. Of Valladolid Mixed method

8 Evaluation methodology (III) Two main experiences with the target audience (teachers with interest in CL and CS) University of Cádiz, Spain [UCA], University of Valladolid, Spain [UVA] University of Cádiz, Spain [UCA], University of Valladolid, Spain [UVA] Creation of the script (for Network Management course) Creation of the script (for Network Management course) Two smaller experiences (mini-cases): CSCL practitioners [GSIC/EMIC], Educational technologist [UNFOLD] CSCL practitioners [GSIC/EMIC], Educational technologist [UNFOLD] Several different scripts Several different scripts Data : Scripts, questionnaires, observations, discussion groups

9 Sample creation of a script (I) “CTM2” script (applied in the “Network Management” case study) Optional undergraduate course on Network Management technologies Optional undergraduate course on Network Management technologies Teacher collaborative understanding of a complex long technical paper that can be divided into 3 different sections (3 versions of a network management protocol). I want the students to discuss and reach agreement on the main ideas of the paper… I want to design a collaboration script that guides the students in the collaborative understanding of a complex long technical paper that can be divided into 3 different sections (3 versions of a network management protocol). I want the students to discuss and reach agreement on the main ideas of the paper… (Evaluation methodology)

10 Sample creation of a script (II) Checking educational benefits, types of problems, complexity Checking educational benefits, types of problems, complexity Selecting the CLFPs Reading information and examples Reading information and examples (Evaluation methodology)

11 Authoring a CLFP-based LD Combining the CLFPs Combining the CLFPs Sample creation of a script (III) (Evaluation methodology)

12 Sample creation of a script (IV) Authoring a CLFP-based LD (Evaluation methodology) Refining the Refining theCLFPs

13 Evaluation methodology (IV) ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements? Pattern- based design process Focus on CSCL critical elements Use of Collage Characteristics of the potential audience Four topics of interest

14 Evaluation results Pattern-based design process Is the selection of the CLFP-based LD templates and their representation useful and satisfactory? Does the design process achieves a satisfactory trade-off between the reuse of the CLFPs and the creation of scripts contextualized according to the situational needs? Support data “… a minimum formation of patterns is necessary, for which Collage is helpful [UVA-focus]” Selection phase is critical and promotes the understanding of the patterns “Collage systematizes the selection of the patterns [UCA-quest-final]” All the GSIC/EMIC, 5 UNFOLD, 3 UVA and 10 UCA partici- pants agree on that the CLFPs are significant. 1 UVA and 3 UCA interviewees state that they are other relevant techniques but these are OK. The rest (2 UNFOLD, 1 UVA and 1 UCA participants) indicate that they do not know any CL technique. The CLFPs are significant but there are more well-known strategies (e.g. for assessment) Support data “… I miss a pattern that I employ in my classes, which is randomly selecting a member of a group to explain what the group has done [UVA-quest-final]” The interviewees’ ideas of the CLFPs coincide with what is presented in Collage Support data “… I think that the patterns are perfectly transferred to the user workspace, reproducing the needed roles and activities for their execution [GSIC/EMIC]” The combinations of patterns provides design flexibility Support data “… it would be nice if other patterns could be added. However, as patterns can be combined, these already offer quite a lot of flexibility [UNFOLD]” Support data “… no example is directly transferable…” “… the examples are useful especially if results are available… as in a community of practice…” [UVA-focus] Pattern-based templates are probably more useful in the process of customizing a new situation than ready-to-run templates, but complete examples are also useful Support data “… no example is directly transferable…” “… the examples are useful especially if results are available… as in a community of practice…” [UVA-focus] Satisfactory trade-off between flexibility, keeping the essence captured in the CLFPs, hiding LD-specific technological details and providing a clear (but limited) set of design options “… it saves the teacher a lot of specification workload [UVA-quest-final]” “I do not find the process too contrained since it enables the combination of patterns… and I think that it is easier for a novice to have an already structured model [UVA-focus]” Support data

15 Evaluation results Use of Collage Can the teachers use successfully Collage? How can Collage be improved? Support data Most of the participants find Collage user- friendly and intuitive and are able to create partly completed examples during the workshops. Very few problems appear by they point out the importance of applying the scripts in real situations. “I am satisfactorily astonished by the user-friendliness of the tool; I thought it was going to be more difficult [UCA-quest-final] ” “.. the participants create the designs without difficulties… [UVA-observer] ” Collage is easier to use, specific to CL and it is the first IMS-LD editor providing pattern- based templates “… at last, patterns in practice!” “… less complicated and more intuitive than Reload on its own…” “makes Reload much easier to use for specific purposes ” [UNFOLD] Support data Incorporating more CLFPs and other types of patterns (also non-pattern based elements), which would provide more flexible design options. (Level B and C of IMS-LD) “Preliminary view” of the script. The integration of Collage with instantiation and delivery systems should support flexibility. “… it may happen that advanced users cannot specify things that they would like [UVA-quest-final]” “… the incorporation of activities that are not part of a pattern but that may be complementary [GSIC/EMIC]” “… if some students abandon the course in the third session, then I need to re-structure [UVA-focus]” Support data

16 Evaluation results Characteristics of the potential audience [UCA- focus-group] Which are the characteristics and motivations of the potential audience of Collage? -Interested in designing CL processes to be used with an LMS in f2f, distant or blended situations -teachers, novice or with experience in CL -some experience and interest in the use of ICT, but it is not necessary to be expert (IMS-LD) technologist “… it is going to be necessary to use this type of tools in our educational system. The EEES will also push to use these tools [UCA-quest-final].” “… for non-experts on CL this is really an eye-opener [UNFOLD]” Support data

17 Discussion (I) A systematic multicase study Focus on Collage workshops (2 experiences with target audience and 2 mini-cases), using a mixed evaluation method Focus on Collage workshops (2 experiences with target audience and 2 mini-cases), using a mixed evaluation method Other two cases: Other two cases: ICALT 2006 workshop (third-party scenario) Experience with students in a real situation (Hernández-Leo et al., IEEE Transactions on Education, to be published in November 2007)

18 Discussion (II) Positive results with respect to: Reuse and contextualization Reuse and contextualization Focus on CSCL critical elements Focus on CSCL critical elements Support of practitioners (teachers) Support of practitioners (teachers) But also pointers for several enhancements and new research efforts

Davinia Hernández-Leo

20 Evaluation results Focus on CSCL critical elements Does the design process: help to determine the learning objectives, task-type and expected interaction that will be developed? help to understand and determine the structure regarding the flow of activities and hierarchy of groups? support also the definition of group- size, resource distribution, computer support and structure within activities? The “selection phase” helps to determine the objectives related to CL that will be promoted and to select the task-type that will be solved by the students Support data “… it helps to determine the CLFP more suitable to foster particular objectives [UVA-quest-final]” “… it helps to envisage how the interactions will develp [UCA-quest-final]” Many arguments support that the design process is very satisfactory concerning this element Support data “I think that the most important aspect of Collage is that it helps to structure and systemtize the different types of activities… Collage has helped me to structure what I already do but I do not organize well [UCA-focus]” Support data “… it hightlights the interrelations among activities” “… the structure of the (learning) process helps to check the roles that each group plays in each moment” [UCA-quest-final] “… Collage helps to conceptualize the groups [UVA-focus] “ I helps to specify these elements, but it does Not provide suggestion of recommended content or tools and the eventual structure within activities is textually-defined (or supported by tools) “… Since Collage guides the design process…, we are pushed to reflect on each element that shapes the activity [UVA-quest-final]” “the design implies the previous determination of all these elements and their organization [UCA-quest-final]” Support data