Evaluation Methods for Social Systems Joan DiMicco IBM Research Center for Social Software.

Slides:



Advertisements
Similar presentations
A GUIDE TO CREATING QUALITY ONLINE LEARNING DOING DISTANCE EDUCATION WELL.
Advertisements

Insert the title of your presentation here Presented by Name Here Job Title - Date Social networking sites as a means to encourage sustainable mobility.
A Power Point for School Planning. Students in Alberta schools should have access to an effective school library program that is integrated with instructional.
©2011 1www.id-book.com Data analysis, interpretation and presentation Chapter 8.
Chapter 8 Data Analysis, Interpretation and Presentation.
Digital Firm (II) Soetam Rizky. Before we start……… Digital firm ? B2B, B2C, C2C ? E-Commerce advantages ? Prospectus of digital firm ?
Genre Shift: Instructor Presence and its Impact on Student Satisfaction in Online Learning.
Consistency of Assessment
S-STEM Program Evaluation S-STEM PI Meeting Arlington, VA October 2012.
Who is non-pro user? certain experience and skill-set not necessarily computer-related tied to the medium computer skills play a role, but maybe they shouldn’t.
Copyright © hutchinson associates 2005 The Knowledge is in the Network Patti Anklam June Holley Valdis Krebs Using Network Analysis to Understand and Improve.
Enterprise social bookmarking - in a community of practice in IBM, Denmark by Joachim Florentz Boye and Marianne Lykke Nielsen Royal School of Library.
Administrivia Turn in ranking sheets, we’ll have group assignments to you as soon as possible Homeworks Programming Assignment 1 due next Tuesday Group.
BSBMKG506B Plan market research Session 1
RESEARCH METHODS Lecture 19
Knowledge is Power Marketing Information System (MIS) determines what information managers need and then gathers, sorts, analyzes, stores, and distributes.
©2007 Prentice Hall Organizational Behavior: An Introduction to Your Life in Organizations Chapter 19 OB is for Life.
Title I Needs Assessment/ Program Evaluation Title I Technical Assistance & Networking Session October 5, 2010.
Curriculum Development For Adult Learners. Basics Educational objective- – What they will learn – Learning needs assessment or gap analysis Learner outcomes-
1 Classroom-Based Research: How to Be a Researcher in Your Classroom Basic Skills Initiative Teaching and Learning Workshop October 2009 Darla M. Cooper.
Company LOGO GET ACEC2012-USING REMOTE INTERNET- BASED LECTURE-CAPTURE TO IMPROVE TEACHING PERFORMANCE Dr. Loren B. Naffziger, Dr. Kenneth Fawson, and.
Research Process Research Process Step One – Conceptualize Objectives Step One – Conceptualize Objectives Step Two – Measure Objectives Step Two – Measure.
Evaluation of a Laptop Program Deborah L. Lowther Steven M. Ross Gary R. Morrison.
Research Process Step One – Conceptualization of Objectives Step Two – Measurement of Objectives Step Three – Determine Sampling Technique Step Four –
Margaret J. Cox King’s College London
Developing You Personal Brand: How to Market Yourself in Interviews and Beyond Mark Presnell Director, Career Center Johns Hopkins University.
Use a plan review Keep Score Provide incentives for positive behavior Manage knowledge retention & transfer Follow up for Sustainable Results Cox Ch 7.
Crowdsourcing Predictors of Behavioral Outcomes. Abstract Generating models from large data sets—and deter¬mining which subsets of data to mine—is becoming.
+ Principals Leadership, Learning and Improvement Project Review Process Environmental and Zoo Education Centres 2013 Principals Conference Dr Natalie.
Sue Huckson Program Manager National Institute of Clinical Studies Improving care for Mental Health patients in Emergency Departments.
Data analysis, interpretation and presentation
SITE Conference – 3/ Successful Online Assessment, Evaluation, and Interaction Techniques Presented by: Dr. Barbara K.Mckenzie – Dept. of MIT, State.
Messaging Nautilus Workshop Overview A two-day workshop designed to transform the way b-to-b marketers create effective content and connect with buyers.
Learnings from Classroom Connections Findings and Conclusions from Two Studies on the Statewide Laptop Initiative Dr. Wade Pogany – South Dakota DOE –
INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Using Evidence for Educational Technology Success.
Quantitative Research Qualitative Research? A type of educational research in which the researcher decides what to study. A type of educational research.
Comments on Communications panel? instruction observation reflections – points to share? (example from Barbara) self evaluation / reflection on instruction.
The Process of Conducting Research
The Literature Search and Background of the Problem.
A System Wide Look at Professional Development Opportunities for Afterschool Professionals Afterschool in Oregon: Professional Development.
Quantitative and Qualitative Approaches
Social Media for Business Communication
A grassroots demonstration by Nokwanda Mbusi. Features  A method of communication with students.  Allows interaction between lecturer and students and.
Chapter 5 Social Network Analysis: Techniques to Discover How Work Really Gets Done.
IMPACT OF QUALITY ASSURANCE SYSTEM IN EFFECTIVENESS OF VOCATIONAL EDUCATION IN ALBANIA IMPACT OF QUALITY ASSURANCE SYSTEM IN EFFECTIVENESS OF VOCATIONAL.
Beyond Data Why does evaluation seen so foreign to most?
Writing Software Documentation A Task-Oriented Approach Thomas T. Barker Chapter 5: Analyzing Your Users Summary Cornelius Farrell Emily Werschay February.
Quantitative & Qualitative Approaches Harlina Nathania Lukman.
MG462 Qualitative Methods Focus Groups Dr. Meredith Rolfe Week 9.
EE573 Introduction to VLSI Systems. Course Objectives (Target Expertise) –Extraction of specification of embedded systems and SoC therein –Methodology-based.
Week 2 The lecture for this week is designed to provide students with a general overview of 1) quantitative/qualitative research strategies and 2) 21st.
Lincoln Community Learning Centers A system of partnerships that work together to support children, youth, families and neighborhoods. CLC.
Social Media for Business Communication Chapter 8 © 2016 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not.
Documenting Completion of your PDP
Quantitative Research Qualitative Research? A type of educational research in which the researcher decides what to study. A type of educational research.
Esri UC2013. Technical Workshop. Technical Workshop 2013 Esri International User Conference July 8–12, 2013 | San Diego, California ArcGIS for Law Enforcement:
Type author names here Social Research Methods Chapter 28: E-research: Internet research methods Alan Bryman Slides authored by Tom Owens.
Quantitative and Qualitative research
Monitoring and Evaluation in the GMS Learning Program 7 – 18 May 2012, Mekong Institute, Khon Kaen, Thailand Randy S. Balaoro, CE, MM, PMP Data Collection.
Developing a Metric for Evaluating Discussion Boards Dr. Robin Kay University of Ontario Institute of Technology 2 November 2004.
Communities and Portals Lan Zhang School of Information University of Texas at Austin.
MScSED Thesis Workshop: Introduction June 12 th, 2015.
OCLC Online Computer Library Center 1 Using Library Perception Information and Impact Data.
Social Media Tools Building a company blog presence presented by Tom Swift Wednesday Nov 18, 2009: 10:45 AM (PST) – 11:30 AM Building Websites and Web.
1 Common knowledge: an exploration of social learning in distributed organisations.
1 KULIAH 2B Survey Design PM Abdul Majid Ismail Ir.M Nawawiy Ir.Dwira.
International Qualitative Conference STUDENT ACHIEVEMENT USING WEB 2.0 TECHNOLOGIES: A MIXED METHODS STUDY By: Dallas R. Malhiwsky.
Creating an Anti-Bias Learning Community
Creating an Anti-Bias Learning Community
Data analysis, interpretation
Presentation transcript:

Evaluation Methods for Social Systems Joan DiMicco IBM Research Center for Social Software

IBM’s Center for Social Software Cambridge, MA Within IBM Research – social software – Collaboration – enterprise communication – social visualization I manage the Visual Communication Lab

Background We build social communities Measure success of these communities Our challenge as researchers: quantifying essentially qualitative experiences Overview of 5 different analysis approaches

Social Software Community We built a social network site for IBM employees How do we get people to join and then continue to use? – Reputation, skills, your merit – Photos – Commenting / sharing – Incentive systems (e.g. points) and honorable roles

Broad summaries from interview data Asked of us: What is the ROI on social networking? We asked: Why are people using our SNS site? Focused subject interviews on this question (~20 ppl) – Coded each interview statement based on the type of motivation – Themes emerged: 1. social/personal connecting, 2. career climbing, and 3. spreading marketing message of pet projects – Then looked content on site for alignment with these themes JM DiMicco, DR Millen, W Geyer, C Dugan, B Brownholtz, M Muller. (2008) “Motivations for Social Networking at Work.” Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW 2008). `

Using surveys to measure success Question: What impact does our SNS have on IBM? (similar to ROI question) We used “social capital” as a concept to measure how employees have more access to information and expertise – Through existing survey instruments, we measured social capital of employees. – Based on their reported behavior on the SNS, we correlated (regression analysis) behavior on site with greater social capital. C Steinfeld, JM DiMicco, N Ellison, C Lampe. (2009) “Bowling Online: Social Networking and Social Capital within the Organization. Fourth International Conference on Communities and Technologies (C&T 2009).

Measuring an intervention We deployed as new feature on the SNS (rewarding points for content). How do we measure impact? – Controlled the deployment: only half of people saw the feature – Compared the two groups – Fully instrumented the system to observe behavior R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “Results from Deploying a Participation Incentive Mechanism within the Enterprise.” Proceedings of CHI 2008.

Using a survey to gather ground truth Does your behavior on the SNS reflect your feelings about the people? We designed a survey “game” asking “How strong is your relationship with these people?” Used this ground truth to compare with behavior on site – Regression analysis demonstrated correlations between closeness and certain behaviors A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

Using interviews to inform quantitative analysis A website for viewing legislation: how are people using it? Logs of which features were used and how long each user stayed – Determined few power users, many casual users – Interviewed 6 power users to answer our questions raised during our log analysis of “why?” A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

Where from here? What is your question to answer? How can you measure it? – Can you measure it by proxy? – Do you have the data or do you need to collect? Surveys System logging Interviews Controlled study