1. (1) Makahiki and SGSEAM: A Serious Game Framework for Sustainability and Assessment Method Yongwen Xu Collaborative Software Development Laboratory Department of Information and Computer Science University of Hawai`i at Mānoa

2. (2) Sustainability is “development that meets the needs of the present without compromising the ability of future generations to meet their own needs”. Brundtland Report, 1987 Introductio n

3. (3) Sustainability Behavior “We are not always rational decision makers” Psychology of Sustainable Behavior, Manning, 2009 Energy is “invisible” Water is “everywhere”

4. (4) Sustainability Education “Education as an essential tool for achieving sustainability” –Agenda 21, Earth Summit 1992, U.N. “Surprising lack of sustainability curriculum initiatives in institutions of higher education” – Sendall 2010

5. (5) Chelsea Hodge, 2010 Campus Sustainability Survey Collegiate Sustainability Competitions

6. (6) Information Technology for Sustainability Education?

7. (7) A. Use a "minimal tech" solution such as a web page and manual posting of data and results Harvard Green Cup

8. (8) B. Out-source to a commercial provider Oberlin Campus Resource Monitoring System Lucid Design Group

9. (9) 3 weeks competition C. Build your own custom in-house solution The Kukui Cup 2011 University of Hawai`i at Manoa 2 years software development

10. (10) D. Use an Open Source Serious Game Framework

11. (11) 1.Design a serious game framework for sustainability education 2.Design a method for assessing serious game frameworks Research Goals

12. (12) Serious Game “a mental contest, played with a computer in accordance with specific rules that uses entertainment to further government or corporate training, education, health, …..” Zyda, Michael 2005

13. (13) Serious Games in Learning and Behavior Motivation Accomplishment Enforcement Engaging Social

14. (14) Power House Reeves, James, 2012 Related work

15. (15) Vermontivate Vermont town energy competition, 2012 Related work

16. (16) “a framework comprised of a collection of di ff erent tools, utilities, and interfaces that hide the low-level details of the various tasks that make up a video game.” - Sherrod 2007 Game Framework

17. (17) Makahiki: An Open Source Serious Game Framework for Sustainability

18. (18) Architecture Game and Mechanics Library APIAPI APIAPI Green Button Data Wattdepot Data Manual Data Smartgrid game Water game Raffle game Raffle game Badge Quest Energy game Authentication Facebook Integration Game Analytics Game Analytics Resource Manager MemCache Database Responsive Web UI System Design Topscore game Topscore game Participation Referral CAS LDAP Django Score Manager

19. (19) What makes Makahiki a Framework? 1.Configurable game elements 2.Configurable resources 3.Extending new content and games 4.Built-in analytics for game evaluation 5.Mobile support 6.Flexible local or cloud deployment

20. (20) 1. Configurable Game Elements

21. (21) 2. Configurable Resource

22. (22) 3. Extending new content and games

23. (23) 4. Built-in Analytics for Game Evaluation

24. (24) 5. Mobile support Laptop view Mobile view Responsive Web Design

25. (25) 6. Flexible local or cloud deployment

26. (26) Experience with Makahiki Framework 2012 2013 Upcoming 2014

27. (27) Instances of the Framework UHM’12HPU’12EWC’12HPU’13 ResourceEnergy Energy & Water Energy Smart metersYes NoYes Duration9 months3 weeks2 weeks3 weeks Eligible players1067198129197 Learning actions144394139 PrizesYes NoYes Quest MechanicsYes NoYes AuthenticationCASLDAPCAS & InternalLDAP DeploymentCloudLocalCloudLocal

28. (28) Assessing Serious Game Framework How do we know if a serious game framework is correctly designed ?

29. (29) Serious Game Assessment Related work

30. (30) Assessment Tools GEQ (Game Engagement Questionnaire) QUIS (Questionnaire for User Interaction Satisfaction) Related work

31. (31) SGSEAM: Serious Game Stakeholder Experience Assessment Method

32. (32) SGSEAM Overview

33. (33) Assessment Vs Evaluation Evaluation: - is to “determine the quality of a program by formulating a judgment” »Hurteau et al., Evaluation 2009 SGSEAM is an assessment method - not to judge - not compare with others

34. (34) SGSEAM Goals Identify the strengths and weaknesses of the framework by assessing the stakeholders’ experiences for the purpose of self improvement

35. (35) SGSEAM Serious Game Stakeholders  Players participate in the game  System admins install and maintain the game infrastructure  Game designers design the content and game mechanics  Game managers manage the game during the game period  Game developers extend, enhance the game

36. (36) Serious Game Stakeholders NOT in SGSEAM  Researchers  Spectators  Funding organizations

37. (37) SGSEAM Assessment Approaches

38. (38) Multiple Assessment Approaches Different levels of rigor in-lab experiment has more data points post-hoc interview has data in production Additive, the more the better

39. (39) SGSEAM Evaluation 1.Apply SGSEAM to Makahiki 1.Apply SGSEAM to Lucid BuildingOS

40. (40) Makahiki Assessment Status

41. (41) Player Pre-post Study Assessment Approach Energy literacy survey before and after the game Energy usage comparison: before and after the game

42. (42) Pre-post Study Result 2011 UH Kukui Cup [2] Energy literacy - 18% (p=0.056) improvement (24 players) Energy usage: - ~2% reduction (20 teams) [2] Robert Brewer PhD dissertation Assessment Outcome: effectiveness in literacy improvement not enough evidence in behavior change

43. (43) Player Engagement Metrics Assessment Approach Analyze game log to calculate: Participation rate Daily players Daily play time Submission rate Social interaction rate Game error rate

44. (44) Engagement Metrics Assessment Result 2012 UH Kukui Cup (n=1038) Evidence of successful player engagement

45. (45) In-lab experiments ICS691 Spring 2013 Serious game development course 8 students (CS major) Assessment: - System admin - Game designer - Game developer

46. (46) Experiment Design Student assignments: - Install the framework - Design games - Develop enhancements Use google form to record: - Time spent - Problem encountered Write blog posts to describe: - Problems in depth - Experiences

47. (47) System admin In-lab installation experiment Result Avg time (minutes) for installation steps -- 2013 Spring ICS691 (n=8) Weakness: database configuration, no enough automation

48. (48) Game designer In-lab design experiment Result Avg time (minutes) for design tasks -- 2013 Spring ICS691 (n=8) Weakness: Smartgrid game predicate design, bugs in design interface

49. (49) Game developer In-lab development experiment Result 2013 Spring ICS691 (n=8) completed “Hello world” widget 8 students completed all 5 enhancements 1 students Main problem reported lack of documentation

50. (50) Makahiki SGSEAM Assessment Preliminary Outcome SGSEAM Assessment Report

51. (51) Makahiki SGSEAM Assessment Preliminary Outcome SGSEAM Assessment Report (Cont.)

52. (52) Lucid BuildingOS Assessment Timeline: Campus Conservation Nationals competition (CCN 2014) Collaborated with Lucid Design Group software platform used by CCN

53. (53) BuildingOS SGSEAM Assessment Proposal

54. (54) Lucid Assessment Steps Provide SGSEAM user guide and initial Lucid assessment plan Identify the stakeholders for BuildingOS Choose stakeholders from several participants of CCN Carry out assessment Produce strength/weakness report

55. (55) Anticipated Contributions 1.Makahiki: an open source serious game framework for different organizations to create sustainability related serious games 2.SGSEAM: an method for assessing serious game framework 3.Insights into the real world experiences of using serious game framework for sustainability education

56. (56) Anticipated Contributions (cont.) 4.Evidence regarding the strengths and weaknesses of Makahiki 5.Evidence regarding the strengths and weaknesses of BuildingOS 6.Evidence of strengths and weaknesses of SGSEAM

57. (57) Publications from this research 1. SGSEAM: Assessing serious game frameworks from a stakeholder experience perspective (Gamification 2013) 2. Energy Feedback for Smart Grid Consumers: Lessons Learned from the Kukui Cup (Energy 2013) 3. Makahiki+WattDepot: An open source software stack for next generation energy research and education (ICT4S 2013) 4. Beyond kWh: Myths and fixes for energy competition game design (Meaningful Play 2012) 5. Lights Off. Game On. The Kukui Cup: A Dorm Energy Competition (CHI 2011 Workshop on Gamification)

58. (58) Time lines (completed)

59. (59) Time lines (planned)

60. (60) Mahalo ! yxu@hawaii.edu Makahiki Project: http://makahiki.io