(1) Makahiki and SGSEAM: A Serious Game Framework for Sustainability and Assessment Method Yongwen Xu Collaborative Software Development Laboratory Department of Information and Computer Science University of Hawai`i at Mānoa

(2) Sustainability is “development that meets the needs of the present without compromising the ability of future generations to meet their own needs”. Brundtland Report, 1987 Introductio n

(3) Sustainability Behavior “We are not always rational decision makers” Psychology of Sustainable Behavior, Manning, 2009 Energy is “invisible” Water is “everywhere”

(4) Sustainability Education “Education as an essential tool for achieving sustainability” –Agenda 21, Earth Summit 1992, U.N. “Surprising lack of sustainability curriculum initiatives in institutions of higher education” – Sendall 2010

(5) Chelsea Hodge, 2010 Campus Sustainability Survey Collegiate Sustainability Competitions

(6) Information Technology for Sustainability Education?

(7) A. Use a "minimal tech" solution such as a web page and manual posting of data and results Harvard Green Cup

(8) B. Out-source to a commercial provider Oberlin Campus Resource Monitoring System Lucid Design Group

(9) 3 weeks competition C. Build your own custom in-house solution The Kukui Cup 2011 University of Hawai`i at Manoa 2 years software development

(10) D. Use an Open Source Serious Game Framework

(11) 1.Design a serious game framework for sustainability education 2.Design a method for assessing serious game frameworks Research Goals

(12) Serious Game “a mental contest, played with a computer in accordance with specific rules that uses entertainment to further government or corporate training, education, health, …..” Zyda, Michael 2005

(13) Serious Games in Learning and Behavior Motivation Accomplishment Enforcement Engaging Social

(14) Power House Reeves, James, 2012 Related work

(15) Vermontivate Vermont town energy competition, 2012 Related work

(16) “a framework comprised of a collection of di ff erent tools, utilities, and interfaces that hide the low-level details of the various tasks that make up a video game.” - Sherrod 2007 Game Framework

(17) Makahiki: An Open Source Serious Game Framework for Sustainability

(18) Architecture Game and Mechanics Library APIAPI APIAPI Green Button Data Wattdepot Data Manual Data Smartgrid game Water game Raffle game Raffle game Badge Quest Energy game Authentication Facebook Integration Game Analytics Game Analytics Resource Manager MemCache Database Responsive Web UI System Design Topscore game Topscore game Participation Referral CAS LDAP Django Score Manager

(19) What makes Makahiki a Framework? 1.Configurable game elements 2.Configurable resources 3.Extending new content and games 4.Built-in analytics for game evaluation 5.Mobile support 6.Flexible local or cloud deployment

(20) 1. Configurable Game Elements

(21) 2. Configurable Resource

(22) 3. Extending new content and games

(23) 4. Built-in Analytics for Game Evaluation

(24) 5. Mobile support Laptop view Mobile view Responsive Web Design

(25) 6. Flexible local or cloud deployment

(26) Experience with Makahiki Framework Upcoming 2014

(27) Instances of the Framework UHM’12HPU’12EWC’12HPU’13 ResourceEnergy Energy & Water Energy Smart metersYes NoYes Duration9 months3 weeks2 weeks3 weeks Eligible players Learning actions PrizesYes NoYes Quest MechanicsYes NoYes AuthenticationCASLDAPCAS & InternalLDAP DeploymentCloudLocalCloudLocal

(28) Assessing Serious Game Framework How do we know if a serious game framework is correctly designed ?

(29) Serious Game Assessment Related work

(30) Assessment Tools GEQ (Game Engagement Questionnaire) QUIS (Questionnaire for User Interaction Satisfaction) Related work

(31) SGSEAM: Serious Game Stakeholder Experience Assessment Method

(32) SGSEAM Overview

(33) Assessment Vs Evaluation Evaluation: - is to “determine the quality of a program by formulating a judgment” »Hurteau et al., Evaluation 2009 SGSEAM is an assessment method - not to judge - not compare with others

(34) SGSEAM Goals Identify the strengths and weaknesses of the framework by assessing the stakeholders’ experiences for the purpose of self improvement

(35) SGSEAM Serious Game Stakeholders  Players participate in the game  System admins install and maintain the game infrastructure  Game designers design the content and game mechanics  Game managers manage the game during the game period  Game developers extend, enhance the game

(36) Serious Game Stakeholders NOT in SGSEAM  Researchers  Spectators  Funding organizations

(37) SGSEAM Assessment Approaches

(38) Multiple Assessment Approaches Different levels of rigor in-lab experiment has more data points post-hoc interview has data in production Additive, the more the better

(39) SGSEAM Evaluation 1.Apply SGSEAM to Makahiki 1.Apply SGSEAM to Lucid BuildingOS

(40) Makahiki Assessment Status

(41) Player Pre-post Study Assessment Approach Energy literacy survey before and after the game Energy usage comparison: before and after the game

(42) Pre-post Study Result 2011 UH Kukui Cup [2] Energy literacy - 18% (p=0.056) improvement (24 players) Energy usage: - ~2% reduction (20 teams) [2] Robert Brewer PhD dissertation Assessment Outcome: effectiveness in literacy improvement not enough evidence in behavior change

(43) Player Engagement Metrics Assessment Approach Analyze game log to calculate: Participation rate Daily players Daily play time Submission rate Social interaction rate Game error rate

(44) Engagement Metrics Assessment Result 2012 UH Kukui Cup (n=1038) Evidence of successful player engagement

(45) In-lab experiments ICS691 Spring 2013 Serious game development course 8 students (CS major) Assessment: - System admin - Game designer - Game developer

(46) Experiment Design Student assignments: - Install the framework - Design games - Develop enhancements Use google form to record: - Time spent - Problem encountered Write blog posts to describe: - Problems in depth - Experiences

(47) System admin In-lab installation experiment Result Avg time (minutes) for installation steps Spring ICS691 (n=8) Weakness: database configuration, no enough automation

(48) Game designer In-lab design experiment Result Avg time (minutes) for design tasks Spring ICS691 (n=8) Weakness: Smartgrid game predicate design, bugs in design interface

(49) Game developer In-lab development experiment Result 2013 Spring ICS691 (n=8) completed “Hello world” widget 8 students completed all 5 enhancements 1 students Main problem reported lack of documentation

(50) Makahiki SGSEAM Assessment Preliminary Outcome SGSEAM Assessment Report

(51) Makahiki SGSEAM Assessment Preliminary Outcome SGSEAM Assessment Report (Cont.)

(52) Lucid BuildingOS Assessment Timeline: Campus Conservation Nationals competition (CCN 2014) Collaborated with Lucid Design Group software platform used by CCN

(53) BuildingOS SGSEAM Assessment Proposal

(54) Lucid Assessment Steps Provide SGSEAM user guide and initial Lucid assessment plan Identify the stakeholders for BuildingOS Choose stakeholders from several participants of CCN Carry out assessment Produce strength/weakness report

(55) Anticipated Contributions 1.Makahiki: an open source serious game framework for different organizations to create sustainability related serious games 2.SGSEAM: an method for assessing serious game framework 3.Insights into the real world experiences of using serious game framework for sustainability education

(56) Anticipated Contributions (cont.) 4.Evidence regarding the strengths and weaknesses of Makahiki 5.Evidence regarding the strengths and weaknesses of BuildingOS 6.Evidence of strengths and weaknesses of SGSEAM

(57) Publications from this research 1. SGSEAM: Assessing serious game frameworks from a stakeholder experience perspective (Gamification 2013) 2. Energy Feedback for Smart Grid Consumers: Lessons Learned from the Kukui Cup (Energy 2013) 3. Makahiki+WattDepot: An open source software stack for next generation energy research and education (ICT4S 2013) 4. Beyond kWh: Myths and fixes for energy competition game design (Meaningful Play 2012) 5. Lights Off. Game On. The Kukui Cup: A Dorm Energy Competition (CHI 2011 Workshop on Gamification)

(58) Time lines (completed)

(59) Time lines (planned)

(60) Mahalo ! Makahiki Project: