Multilevel analysis of collaborative activities based on 1 Multilevel analysis of collaborative activities based on a Mobile Learning Scenario for real classrooms Irene-Angelica Chounta, Adam Giemza, H. Ulrich Hoppe Collide, University of Duisburg-Essen {chounta, giemza, hoppe}@collide.info

Mobile learning: a special case of learning.... Learning on the move, within context Activities supported by: „location-based“ scenarios(museum, outdoor trips etc) smartphones, tablets etc. Qualitative analysis of the activity

Objectives of the study Introduction of a mobile learning scenario in traditional school classrooms Use of common learning analytics within a mobile learning context Study of collaborative learning activities on multiple levels of analysis

a mobile learning scenario for the classroom Computer Kit: „Learn how to assemble a Personal Computer with Mobilogue“

Mobilogue Architecture Authoring environment Activity logs Online Management Environment

a mobile learning scenario for the classroom QR codes (learning stations) marking computer parts in a localized setting Text & Pictures Video Quizzes

Method of the study Two case studies: Case A: preliminary study, 17 students Case B: main study, 24 students Task: Group of students working together to assemble a pc Each group was supported by one mobile device The activities were analyzed with respect to: students practice (individual and group level) group characteristics learning outcome 13 – 15 years old

Method of the study For the analysis we used: Activity metrics from logfiles Average time gap among consequent actions Average response time to quizzes Quiz score Experts observations Activity transcripts (recorded by students) Users questionnaires (collaborative experience) Pre and Post Knowledge Tests (ten questions each)

Case study A - Results 17 students randomly grouped in 5 groups Activity duration ≈ 90 minutes Group profiles based on pre-tests: One „strong“ team (team A) One „weak“ team (team E) Three average teams (teams B, C, D)   Pretest Scores avg_group max_idv teamA 6.25 8 teamB 3.75 5 teamC 4.33 teamD 2.67 4 teamE 1 heterogeneous homogeneous

Case study A - Results Greatest improvement  teams of average performance (on group and individual level) The “strong” team maintained the same score on average but scored less on the individual level - homogeneity increased The “weak” team slightly improved – heterogeneity increased Heterogeneous teams increase homogeneity and vice versa   Pre-test Scores avg_group max_idv teamA 6.25 8 teamB 3.75 5 teamC 4.33 teamD 2.67 4 teamE 1 Post-test Scores avg_group max_idv 6.25 7 8 5.67 6.00 2.00 3

Case study A - Results Quiz score reflects the knowledge test performance The average time gap correlates negatively with quiz score Teams with good knowledge background and high heterogeneity move faster Activity metrics avg_timegap avg response time Quiz score 27.63 32.02 120 36.56 56.96 37.14 23.70 100 37.91 31.55 90 38.43 57.43 80   Pre-test Scores avg_group max_idv teamA 6.25 8 teamB 3.75 5 teamC 4.33 teamD 2.67 4 teamE 1

Case study B - Results 24 students – 6 groups of four Groups were created by the teacher Activity duration ≈ 90 minutes Additions / Modifications: Experts monitored the activity Students kept transcripts of the group activity Students filled in questionnaires regarding their collaborative experience

Case study B - Results Group profiles based on pre-tests: One strong team (team 03) Two weak teams (team 00 and team 05) Three average teams (teams 01, 02, 04) The Homogeneity of teams according to pre-test confirmed teacher‘s perception   Pre-test Scores avg_group max_idv team 00 2 4 team 01 5.75 9 team 02 4.25 5 team 03 6.5 8 team 04 5.5 7 team 05 1 heterogeneous homogeneous

Case study B - Results Group performance improved on average (47%) and individual (31%) Max knowledge gain  weak team (team 05) Min knowledge gain  strong team (team 03) Knowledge „loss“ for the „strong“ students Homogenity increases for heterogenous groups and vice versa (confirming case study A)   Pre-test Scores avg_group max_idv team 00 2 4 team 01 5.75 9 team 02 4.25 5 team 03 6.5 8 team 04 5.5 7 team 05 1 Post-test Scores avg_group max_idv 3.5 7 7.5 8 6 6.5 7.25 9

Case study B - Results Heterogeneous teams achieved higher Quiz scores The average time gap correlates negatively with quiz score (also observed case study A) Heterogeneous teams move faster, i.e. Individuals take the lead Activity metrics avg_timegap avg response time Quiz score 20.89 51.88 120 23.42 49.02 24.00 45.73 110 25.84 61.99 90 40.44 72.51 70 27.75 82.56   Pre-test Scores avg_group max_idv team 00 2 4 team 01 5.75 9 team 02 4.25 5 team 03 6.5 8 team 04 5.5 7 team 05 1

Case study B - Results The students undertook three roles throughout the scenario: the operator of the mobile device the scriber and the assembler 4 out of 6 teams followed no strict role distribution Teams following strict role distribution appeared to have lowest knowledge gain & dissatisfaction towards collaboration

Discussion Learning Outcome A mobile scenario to support collaborative activities for traditional classrooms Multilevel analysis of the activity (various methods and multiple units of analysis) to enhance the overall outcome Learning Outcome The setting was more effective for weak and average students Strong students/ teams had not significant knowledge gain The students performance increased

Group Characteristics & Learning Analytics Discussion Group Characteristics & Learning Analytics Heterogeneous teams achieved higher quiz scores Quiz score did not correlate to knowledge gain or knowledge background Heterogeneous teams moved faster through the stages of the scenario Homogeneous teams tend to increase their heterogeneity and vice versa

Discussion Collaboration Team members undertook roles with no particular plan In the case a strict plan was followed, it led to frustration and low knowledge gain Strong students stated they would have done better on their own Weak/average students considered the activity as helpful

Conclusion and Future Work Multilevel analysis is needed to fully understand group tactics and enhance mobile learning Extend the field of study to further investigate: the use of various scenarios varying degrees of freedom to classroom activities concurrent use of multiple devices within a team

Questions. Thank you  chounta@collide.info Thank you 