Evaluation Methods for Social Systems Joan DiMicco IBM Research Center for Social Software

IBM’s Center for Social Software Cambridge, MA Within IBM Research – social software – Collaboration – enterprise communication – social visualization I manage the Visual Communication Lab

Background We build social communities Measure success of these communities Our challenge as researchers: quantifying essentially qualitative experiences Overview of 5 different analysis approaches

Social Software Community We built a social network site for IBM employees How do we get people to join and then continue to use? – Reputation, skills, your merit – Photos – Commenting / sharing – Incentive systems (e.g. points) and honorable roles

Broad summaries from interview data Asked of us: What is the ROI on social networking? We asked: Why are people using our SNS site? Focused subject interviews on this question (~20 ppl) – Coded each interview statement based on the type of motivation – Themes emerged: 1. social/personal connecting, 2. career climbing, and 3. spreading marketing message of pet projects – Then looked content on site for alignment with these themes JM DiMicco, DR Millen, W Geyer, C Dugan, B Brownholtz, M Muller. (2008) “Motivations for Social Networking at Work.” Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW 2008). `

Using surveys to measure success Question: What impact does our SNS have on IBM? (similar to ROI question) We used “social capital” as a concept to measure how employees have more access to information and expertise – Through existing survey instruments, we measured social capital of employees. – Based on their reported behavior on the SNS, we correlated (regression analysis) behavior on site with greater social capital. C Steinfeld, JM DiMicco, N Ellison, C Lampe. (2009) “Bowling Online: Social Networking and Social Capital within the Organization. Fourth International Conference on Communities and Technologies (C&T 2009).

Measuring an intervention We deployed as new feature on the SNS (rewarding points for content). How do we measure impact? – Controlled the deployment: only half of people saw the feature – Compared the two groups – Fully instrumented the system to observe behavior R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “Results from Deploying a Participation Incentive Mechanism within the Enterprise.” Proceedings of CHI 2008.

Using a survey to gather ground truth Does your behavior on the SNS reflect your feelings about the people? We designed a survey “game” asking “How strong is your relationship with these people?” Used this ground truth to compare with behavior on site – Regression analysis demonstrated correlations between closeness and certain behaviors A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

Using interviews to inform quantitative analysis A website for viewing legislation: how are people using it? Logs of which features were used and how long each user stayed – Determined few power users, many casual users – Interviewed 6 power users to answer our questions raised during our log analysis of “why?” A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

Where from here? What is your question to answer? How can you measure it? – Can you measure it by proxy? – Do you have the data or do you need to collect? Surveys System logging Interviews Controlled study