Davinia Hernández-Leo, Eloy D. Villasclaras, Juan I. Asensio-Pérez, Yannis Dimitriadis GSIC / EMIC group University of Valladolid, Spain TENCompetence Workshop Barcelona, June 22 nd Some findings from the evaluation of Collage authoring tool

2 Our approach (I) Challenges 1.Designing scripts that potentially elicits the desired interactions How can teachers create “potentially effective” “computer-interpretable” collaboration scripts according to the necessities of their situation? 3. Hiding computational representations of scripts to teachers 2. Formalizing the scripts so that they are computer-interpretable

3 Our approach (II) 1. Not easy to design a script that potentially elicits the desired interactions Reusing the “essence” of commonly used CSCL scripts (formulated as patterns) CLFPs, Collaborative Learning Flow Patterns: Broadly accepted techniques repetitively used by practitioners when structuring the flow of types of (collaborative) learning activities (E.g. Jigsaw, Pyramid (Snowball) CLFPs)

4 Our approach (IV) 2. Formalizing the scripts so that they are computer-interpretable IMS Learning Design (LD) IMS Learning Design (LD)

5 Our approach (V) 3. Computational representations are not familiar to educators Integrate in authoring tools “a pattern-based design process for CSCL macro-scripts computationally represented with IMS LD” which allows teachers to create their own computer-interpretable script by reusing CLFPs as visual LD templates that can be assembled and refined COLLAGE (COLaborative LeArning desiGn Editor) (Based on Reload Editor. IMS-LD level A compliant)

6 Evaluation methodology (I) Multicase study (Stake, 2005) Multicase study (Stake, 2005) Authentic experiences with different audiences (contextual issues, achievement of educational benefits, impact of software systems) Cross-case analysis Not aiming at accurate generalizations but at valid conclusions Mixed method combining qualitative and quantitative data gathering techniques (Martínez et al., 2003) Mixed method combining qualitative and quantitative data gathering techniques (Martínez et al., 2003) Triangulation: Comparative analysis of the data obtained with the different techniques

7 Evaluation methodology (II) Multicase study (Stake, 2005) QUITAIN: The proposed pattern-based design process for CSCL macro-scripts computationally represented with IMS LD ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements? Creating CSCL scripts based on CLFPs using Collage ISSUE: Can we use Collage for creating a script representing a scenario proposed by a third-party? Solving a third-party scenario ISSUE: Can we use CSCL scripts created with Collage in real situations ? Putting into practice a CSCL script created with Collage “Collage workshops” Case Study Lab. uni. of Valladolid and Cádiz, Spain Hands-on sessions Mixed method Questionnaires Observations Focus groups Mini-cases “Planet game” Case Study ICALT conference Pre-work Presenting Discussing Achieved CSCL script, papers, discussion “Network Managment” Case Study Two f2f and a distance session Lab. Uni. Of Valladolid Mixed method

8 Evaluation methodology (III) Two main experiences with the target audience (teachers with interest in CL and CS) University of Cádiz, Spain [UCA], University of Valladolid, Spain [UVA] University of Cádiz, Spain [UCA], University of Valladolid, Spain [UVA] Creation of the script (for Network Management course) Creation of the script (for Network Management course) Two smaller experiences (mini-cases): CSCL practitioners [GSIC/EMIC], Educational technologist [UNFOLD] CSCL practitioners [GSIC/EMIC], Educational technologist [UNFOLD] Several different scripts Several different scripts Data : Scripts, questionnaires, observations, discussion groups

9 Sample creation of a script (I) “CTM2” script (applied in the “Network Management” case study) Optional undergraduate course on Network Management technologies Optional undergraduate course on Network Management technologies Teacher collaborative understanding of a complex long technical paper that can be divided into 3 different sections (3 versions of a network management protocol). I want the students to discuss and reach agreement on the main ideas of the paper… I want to design a collaboration script that guides the students in the collaborative understanding of a complex long technical paper that can be divided into 3 different sections (3 versions of a network management protocol). I want the students to discuss and reach agreement on the main ideas of the paper… (Evaluation methodology)

10 Sample creation of a script (II) Checking educational benefits, types of problems, complexity Checking educational benefits, types of problems, complexity Selecting the CLFPs Reading information and examples Reading information and examples (Evaluation methodology)

11 Authoring a CLFP-based LD Combining the CLFPs Combining the CLFPs Sample creation of a script (III) (Evaluation methodology)

12 Sample creation of a script (IV) Authoring a CLFP-based LD (Evaluation methodology) Refining the Refining theCLFPs

13 Evaluation methodology (IV) ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements? Pattern- based design process Focus on CSCL critical elements Use of Collage Characteristics of the potential audience Four topics of interest

14 Evaluation results Pattern-based design process Is the selection of the CLFP-based LD templates and their representation useful and satisfactory? Does the design process achieves a satisfactory trade-off between the reuse of the CLFPs and the creation of scripts contextualized according to the situational needs? Support data “… a minimum formation of patterns is necessary, for which Collage is helpful [UVA-focus]” Selection phase is critical and promotes the understanding of the patterns “Collage systematizes the selection of the patterns [UCA-quest-final]” All the GSIC/EMIC, 5 UNFOLD, 3 UVA and 10 UCA partici- pants agree on that the CLFPs are significant. 1 UVA and 3 UCA interviewees state that they are other relevant techniques but these are OK. The rest (2 UNFOLD, 1 UVA and 1 UCA participants) indicate that they do not know any CL technique. The CLFPs are significant but there are more well-known strategies (e.g. for assessment) Support data “… I miss a pattern that I employ in my classes, which is randomly selecting a member of a group to explain what the group has done [UVA-quest-final]” The interviewees’ ideas of the CLFPs coincide with what is presented in Collage Support data “… I think that the patterns are perfectly transferred to the user workspace, reproducing the needed roles and activities for their execution [GSIC/EMIC]” The combinations of patterns provides design flexibility Support data “… it would be nice if other patterns could be added. However, as patterns can be combined, these already offer quite a lot of flexibility [UNFOLD]” Support data “… no example is directly transferable…” “… the examples are useful especially if results are available… as in a community of practice…” [UVA-focus] Pattern-based templates are probably more useful in the process of customizing a new situation than ready-to-run templates, but complete examples are also useful Support data “… no example is directly transferable…” “… the examples are useful especially if results are available… as in a community of practice…” [UVA-focus] Satisfactory trade-off between flexibility, keeping the essence captured in the CLFPs, hiding LD-specific technological details and providing a clear (but limited) set of design options “… it saves the teacher a lot of specification workload [UVA-quest-final]” “I do not find the process too contrained since it enables the combination of patterns… and I think that it is easier for a novice to have an already structured model [UVA-focus]” Support data

15 Evaluation results Use of Collage Can the teachers use successfully Collage? How can Collage be improved? Support data Most of the participants find Collage user- friendly and intuitive and are able to create partly completed examples during the workshops. Very few problems appear by they point out the importance of applying the scripts in real situations. “I am satisfactorily astonished by the user-friendliness of the tool; I thought it was going to be more difficult [UCA-quest-final] ” “.. the participants create the designs without difficulties… [UVA-observer] ” Collage is easier to use, specific to CL and it is the first IMS-LD editor providing pattern- based templates “… at last, patterns in practice!” “… less complicated and more intuitive than Reload on its own…” “makes Reload much easier to use for specific purposes ” [UNFOLD] Support data Incorporating more CLFPs and other types of patterns (also non-pattern based elements), which would provide more flexible design options. (Level B and C of IMS-LD) “Preliminary view” of the script. The integration of Collage with instantiation and delivery systems should support flexibility. “… it may happen that advanced users cannot specify things that they would like [UVA-quest-final]” “… the incorporation of activities that are not part of a pattern but that may be complementary [GSIC/EMIC]” “… if some students abandon the course in the third session, then I need to re-structure [UVA-focus]” Support data

16 Evaluation results Characteristics of the potential audience [UCA- focus-group] Which are the characteristics and motivations of the potential audience of Collage? -Interested in designing CL processes to be used with an LMS in f2f, distant or blended situations -teachers, novice or with experience in CL -some experience and interest in the use of ICT, but it is not necessary to be expert (IMS-LD) technologist “… it is going to be necessary to use this type of tools in our educational system. The EEES will also push to use these tools [UCA-quest-final].” “… for non-experts on CL this is really an eye-opener [UNFOLD]” Support data

17 Discussion (I) A systematic multicase study Focus on Collage workshops (2 experiences with target audience and 2 mini-cases), using a mixed evaluation method Focus on Collage workshops (2 experiences with target audience and 2 mini-cases), using a mixed evaluation method Other two cases: Other two cases: ICALT 2006 workshop (third-party scenario) Experience with students in a real situation (Hernández-Leo et al., IEEE Transactions on Education, to be published in November 2007)

18 Discussion (II) Positive results with respect to: Reuse and contextualization Reuse and contextualization Focus on CSCL critical elements Focus on CSCL critical elements Support of practitioners (teachers) Support of practitioners (teachers) But also pointers for several enhancements and new research efforts

Davinia Hernández-Leo

20 Evaluation results Focus on CSCL critical elements Does the design process: help to determine the learning objectives, task-type and expected interaction that will be developed? help to understand and determine the structure regarding the flow of activities and hierarchy of groups? support also the definition of group- size, resource distribution, computer support and structure within activities? The “selection phase” helps to determine the objectives related to CL that will be promoted and to select the task-type that will be solved by the students Support data “… it helps to determine the CLFP more suitable to foster particular objectives [UVA-quest-final]” “… it helps to envisage how the interactions will develp [UCA-quest-final]” Many arguments support that the design process is very satisfactory concerning this element Support data “I think that the most important aspect of Collage is that it helps to structure and systemtize the different types of activities… Collage has helped me to structure what I already do but I do not organize well [UCA-focus]” Support data “… it hightlights the interrelations among activities” “… the structure of the (learning) process helps to check the roles that each group plays in each moment” [UCA-quest-final] “… Collage helps to conceptualize the groups [UVA-focus] “ I helps to specify these elements, but it does Not provide suggestion of recommended content or tools and the eventual structure within activities is textually-defined (or supported by tools) “… Since Collage guides the design process…, we are pushed to reflect on each element that shapes the activity [UVA-quest-final]” “the design implies the previous determination of all these elements and their organization [UCA-quest-final]” Support data