Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly GaLA Game and Learning Alliance The European Network of Excellence on Serious Games SIG3.5 – SGs for Humanities & Heritage user studies Michela Mortara, Chiara Eva Catalano, CNR – IMATI Jeffrey Earp, CNR-ITD

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Overview Icura [ongoing, 50%] Playing History – The Plague [started, 20%] Playing History – The Plague [started, 10%]

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly USER STUDY #1: ICURA

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Game description Educational objective: cultural awareness - Japanese etiquette Genre: adventure Environment: 3D Platform: PC Intended educational context: informal (at home) Developer: TU-Wien Duration: less than 1 h Target: secondary school and higher Previous studies: evaluation done by the developers at the Wien university on a group of 20 students. Reports on papers and PhD. thesis

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Previous evaluation Pre-questionnaire (demographic data & game habits) Pre-test with 12 questions ( on language basics, behaviour & etiquette, culture and society) Game session Post-test (same 12 questions) Final questionnaire about user satisfaction Observations: On average from +5 to +10 correct answers btw pre and post –Selective attention bias Learning gain linked to the way the content is communicated (information agent vs quest) Lacking estimate of retention

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Our setting Same phases but different questions in the pre-test to avoid the selective bias Higher participant number (CNR, RWTH, ORT, DIBE) Open answers to assess higher level in Bloom’s taxonomy (post questionnaire) e.g.: What have you learnt from the game and how (through what mechanics)? Did the game increase your interest towards Japan/ Japanese culture? Would you read more about Japan? Would you visit it? How would you summarise Japanese culture in 3 adjectives? What can you deduce about Japanese culture from the facts you learnt in the game? How would you compare Japanese and Western culture? Plan to repeat the post test after a few months to evaluate retention

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Current status CNR session –10 university students –Colleagues&friends RWTH session –17 university students ORT session –12 people –students DIBE session

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Partial outcomes

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Partial outcomes

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Partial outcomes

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly PLAYING HISTORY – THE PLAGUE

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly The game Educational objective: history, historical reconstruction Genre: adventure Environment: 3D Platform: browser (Unity Web player) Intended educational context: formal (at school) Developer: Serious Games Interactive Duration: nearly 3 hours Target: 8-13 Previous studies: evaluation done by a master student with two primary school classes in Denmark. Report in her Master thesis (in Danish).

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly The experience: a Joint Research Action JRA involving CNR-IMATI (SIG3.5), CNR-ITD (TC2.8, T6.3), SGI and DIS (Deledda International School) in Genova Objectives: –explore the suitability and effectiveness of serious games in the field of humanities and cultural heritage in schools; –evaluate the educational efficacy of the game and the level of student engagement and enjoyment triggered by the learning activities; –investigate critical aspects of SG deployment in formal ed./school contexts, especially teacher’s role –examine cultural/educational issues related to game’s adoption in Italy (game produced in northern Europe and set in medieval Italy)

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly The context School: English-speaking school in Genoa following International Baccalaureate & IB Middle Years programmes Classes: 2 classes Year 7 (11-12 year olds). Approx. 35 pupils Deployment settings: computer lab, classroom, home access

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Research framework Approach: qualitative, design based research – DBR CollectiveDBR Collective Tools: –question matrix for gathering teachers’ GBL/ICT experience & attitudes, plus their expectations, intentions and goals for the experience –pedagogical planning tool (teachers document intentions/goals of intervention and plan learning activity sequence) –researchers’ observation sheets for monitoring classroom activities –pupils’ pre/post tests (in teacher’s resource pack from SGI) –mind map recording pupils’ collective pre/post knowledge/awareness –teachers’ monitored focus group –teacher questionnaire –reflection sheet for recording researchers’ attitudes, intentions, goals and interventions in the experience

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Activity Phases 1 & 2… 1. Teacher preparation –introduction to the experiment, the game and its teaching resource pack –teachers’ group discussion of matrix questions (researcher compiled) –tech. training session (game’s teacher functions; ITD pedagogical planner) –teachers consolidate pedagogical plans 2.Deployment –pupil’s individual pre-test & collective brainstorming of pre-knowledge (making collective mind map) –class briefing: game presentation + play –gameplay sessions in pairs (plus possible home play) –pupil summative test & debriefing (re-examining collective mind map) =total 4/5 school periods with researcher monitoring & support

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly … Activity phases Teacher debriefing –focus group (inc. re-examination of matrix questions and pre- deployment responses, and of pedagogical plan/s) –final questionnaire 4.Analysis of data gathered from phases Follow up –Repeat of post test to evaluate pupil retention 6.Final data analysis and reporting

Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Current status Series of organisational meetings held involving CNR, DIS & SGI Activity phase 1 commenced –introduction to teachers –discussion / compiled of teachers’ of question matrix Issues with running game on DIS network have delayed commencement of deployment sessions with pupils Solution: DIS agreed to move students to CNR (2 sessions in May)