Evaluating the Effectiveness of a Virtual Economy System Within an Advanced Scientific Cyberinfrastructure Directed Project, WMP in Technology A. Nedossekina. Chair: Dr. M. Sutton

The age of virtual economy A system of economic activities of users within a virtual community. A virtual marketplace for real goods and real money Social network sites where virtual trading & rewards help build communities MMOG, virtual gaming communities trading virtual goods Grid computing projects using market-based resource allocation Total value of assets in virtual worlds (including online games) approaching US$2 billion

nanoHUB.org Online simulation and more Serving a scientific community of over 90,000 users around the globe

nanoHUB.org A resource to the entire nanotechnology discovery and learning community Online simulation and more 137 simulation tools (plus 97 in development) Over 1400 “and more” resources 819 resource reviews 433 questions & answers

nanoHUB.org Resources come from over 600 contributors, community rates them and asks questions Online simulation and more

The site has over 24,000 total registered users, but on average only about 2% contribute 2.5% of registered users contributed a resource < 1% of registered users participated in the Answers forum < 2% of registered users contributed a review Problem: low contribution rates

Introducing virtual reward points as a way to motivate contributions Earn by contributing Spend on merchandise and high-end services Solution: virtual economy

nanoHUB virtual economy Resources Reviews Questions & Answers … Merchandise Faster computation cycles More data storage More interactive sessions …

nanoHUB virtual economy Goal: to help virtual community growth and sustainability

nanoHUB virtual economy Resources Reviews Questions & Answers … Merchandise Faster computation cycles More data storage More interactive sessions … Current development stage

2002 nanoHUB startedPoints introduced Jun 2007Apr 2008 Answers forum launched

What was the impact on user contribution activity? Did the system change user behavior? Do users contribute more than before? ? Why? To improve current system To help future development To know users better Need for an evaluation model

PRE 2007 Jul – Sep (A) Oct – Dec (B) Jul – Dec (C) POST 2008 Jul – Sep (A) Oct – Dec (B) Jul – Dec (C) Our project looked for an answer in usage data, measuring impact on activity in the Answers forum Jun 2007Apr 2008 ? What was the impact on user contribution activity?

Closer look at our users Simulation users Minimal Expert Average Heavy PRE POST Number of simulation runs

What percentage of users contributed Q&A? Percentage of users who contributed Q&A increased in all categories.

How big is the change in percentage of users who contributed Q&A? 8.3 increase among Heavy simulation users!

What are the ratios of Q&A per user? Ratios of Q&A contributions per user increased in all categories, With expert users having the highest Q&A/user ratios

How big is the change in the ratios of Q&A per user? 9.6 increase among Heavy simulation users!

How does the impact on Q&A activity compare with changes in other contribution activities? Change in percentage of contributing users Q&A Reviews Resources

How does the impact on Q&A activity compare with changes in other contribution activities? The impact on Q&A contribution activity among Heavy users is obvious! Change in percentage of contributing users Change in ratio of contributions per user Q&A Reviews Resources

Research process Data analysis Descriptive statistics Ratio analysis in user categories Data collection Correlation analysis Spearman rank correlation between contribution and usage levels

Correlation analysis examined the relationship between usage level and contribution activity Data for each user in a group: Usage N of simulation runs N of sessions CPU Session time N of downloads N of web clicks N of unique module views Login time N of logins usage contributions nanoHUB Contributions N of Q&A N of resources N of reviews Simulation usage level 1 Anderson-Darling Test for normality a.non-normal distribution b.need Spearman Rank 2Spearman Rank correlation (Pearson correlation on ranked data) 3Correlation coefficients compared in PRE & POST for several user groups Non-simulation usage level Correlation?

Correlation analysis showed increase of positive correlation between usage levels and Q&A contributions in POST periods for ALL studied user groups Yet, the correlation coefficients were statistically insignificant (<0.3)

Conclusions about the evaluation model Two data analysis approaches utilized Correlation analysisRatio analysis in user categories Both methods provided some valuable insight into user behavior Both may be used for future system assessment Taking advantage of complex statistical tests Looking at effect on overall population Unbiased assessment Complex procedures Studied impact saturated Robust assessment Comprehensive comparison of pre and post situation Looking at impact on different user categories Arbitrary user classification, thus more bias May appear oversimplified

What’s next? Expand system to earn points on reviews Tie Q&A with non-simulation resources to increase participation of non-simulation, minimal and average simulation users Repeat assessment when new components are deployed: both for testing of evaluation approaches & for new insight about impact of system on user behavior May publish in Journal of Virtual Worlds Research Special issue on virtual economies Deadlines: Abstract - June15, Full manuscript - November 1, 2009 Publication Date: December 15, 2009

The end.

Studied User Groups