1. DRAFT: Opinion Space Pilot Study Topic: Reputation of the American Automotive Industry Participants: Automotive Industry Online Community Research Period: Fall 2010 – Spring 2011 May 2011 Rick Wagner, Joyce Salisbury, Ken Goldberg, David Wong UC Berkeley Logo

2. Pilot Study Goals 1)Sample Existing Perceptions of the Image of the American Automotive Industry 2)Collect Ideas about How to Improve the Image of the American Automotive Industry and Identify the Most Insightful Ideas 3)Study how Opinion Space compares with Existing Survey Methods in Terms of Participant Engagement 4)Consider Demographic Variations among Participants 5)Consider Applications for gm.com

3. Executive Summary 1)Sample Size (# Participants): 1198 2)Avg Number of Minutes Spent on Site: 36 3)Avg Number of Peer-to-Peer Evals: 90 4)% Who are “Extremely Likely” to Participate Again: 95 1)Average Age of Least Active Participants: 44 2)Average Age of Most Active Participants: 50 (!) 3)Key Issues: 4)20 Most Insightful Ideas

4. Executive Summary Discussion Question: Improving Industry Image 1)Integrity: Honesty, Responsiveness to Flaws 2)Value: Quality, Price, Safety, Warranty 3)Only 2 of top 20 mention “bailout” 4)Listen more to consumers 5)Don’t over-power vehicles 6)Fewer “Gee-Whiz” features 7)Become leaders in innovation 8)Don’t charge too much for GPS 9)LED Headlights 10)Develop Autos that run on Natural Gas!

5. Executive Summary Opinion Space was very effective in engaging AIOC participants –Average total time spent on the system: 36 minutes. –Average number of ratings per user: 90 insight and agreement ratings. –Much excitement expressed to participate again (see figure). Top Insights –AIOC members suggested many things GM is already working on. Next steps –Host another Opinion Space on GM.com to engage the public in generating ideas. –Reengage the AIOC to better communicate what GM is working on and solicit ideas on how to improve. Q150. How willing would you be to participate in other versions of this system (on related topics)? Not likely Extremely likely

6. Opinion Space

7. Our Approach 1. Visualization2. Level Playing Field 3. Wisdom of the Crowds4. Game Structure

8. "Opinion Space will harness the power of connection technologies to provide a unique forum for international dialogue. This is...an opportunity to extend our engagement beyond the halls of government directly to the people of the world.” - U.S. Secretary of State Hillary Clinton

9. “The world spends over 110 billion minutes per month on social networks and blog sites.” - NielsenWire, June 2010

10. “ We’re moving from an Information Age to an Opinion Age.” - Warren Sack, UCSC

11. 20 sec. per comment X 35,387 comments = 8 days

13. goals For Organizations Understand the diversity of their community Engage their community –Solicit feedback and creative suggestions –Rapidly identify patterns, insightful ideas For Community Members Understand relationships with other community members Engage with a diversity of viewpoints and ideas Express ideas, and be heard hybrid vigor

14. opinion.berkeley.edu

15. Step 1: Enter your opinions and response

16. Step 2: Visualize your position

17. Step 3: Read and rate others users

19. System Design (Aug-Dec 2010) Baseline Questions: When deciding on which major product to buy, I often consider where it was made in making my decision. Reducing oil consumption should be a higher priority in the U.S. I am willing to pay more for a product that comes from a socially responsible corporation. The U.S. government should not give loans to corporations. New technologies are causing more problems than they are solving. Discussion Question: What is the Most Important Step or Steps that US Automotive Manufacturers could take to Improve Their Image?

20. System Stats Duration 20 Dec 2010 – 21 Jan 2011 Announcement, 2 Reminders 1198 participants (57% response rate) 1,148 Responses 96,000 Insight Ratings 96,000 Agreement Ratings

21. Opinion Space

22. Users and sessions per day Unique visits per day Active users per day

23. Session duration Zoomed out Zoomed in

24. Ratings per day

25. Distribution of ratings All responses are rated on two scales: 1)How much do you agree with the response? 2)How insightful is this response?

26. Ratings and views per comment

27. Insight ratings per user Zoomed out Zoomed in

28. Agreement ratings per user Zoomed out Zoomed in

29. Statement Statistics 1)When deciding on which major product to buy, I often consider where it was made in making my decision. 2)Reducing oil consumption should be a higher priority in the U.S. 3)I am willing to pay more for a product that comes from a socially responsible corporation. 4)The U.S. government should not give loans to corporations. 5)New technologies are causing more problems than they are solving.

30. Statement 1: When deciding on which major product to buy, I often consider where it was made in making my decision.

31. Statement 2: Reducing oil consumption should be a higher priority in the U.S.

32. Statement 3: I am willing to pay more for a product that comes from a socially responsible corporation.

33. Statement 4: The U.S. government should not give loans to corporations.

34. Statement 5: New technologies are causing more problems than they are solving.

35. Statements in relation to position in Opinion Space For each statement, a participant’s point will move in the direction and angle indicated by the arrow as they drag the slider.

36. Participant Responses Participant responses were ranked using the insightfulness ratings from other participants. Two main algorithms were used to rank responses. –Spatial ranking: ratings of a response are weighted by the spatial distance between the author of the response and the rater. –Confidence interval ranking: responses are ranked by the lower bound of a 95% confidence interval of the mean. See Appendix for a verbatim list of the top 10 responses

37. Topics and Insights from the Top 18 Participant Responses Topic: US Automotive Corporations - Achieve more integrity - Streamline production to reduce costs - Renegotiate union contracts to keep prices low - Find out what the customer really wants - Engage in product development with the goal of long term use - Market value to consumers using measurable and verifiable methods - Pay back government loans - Lessen environmental impact Topic: Price - Remove extraneous, meaningless features that increase the price and don’t add significant value. - GPS cited as an example* - Keep prices low and competitive with foreign companies - Affordability is the bottom line for most people *participant1329’s response (rank #1): “When offering new technology, they must keep the price unchanged or slightly higher. Newer technology is often exponentially more than its value. For example, remember the GPS? Automakers were charging $1500-2500 for an installed GPS when you could buy one for $300. This makes the automaker look greedy…”

38. Topics and Insights from the Top 18 Participant Responses Topic: What participants want in quality and value - Durable, reliable, dependable, well- built, no manufacturing defects, built with stronger and lighter materials - Safety - Fuel-saving, fuel-efficient, and good fuel-economy and mileage - Increased warranties that reflect higher quality cars - Performance - Comfortable for larger families Topic: Electric cars - Stop building electric cars and focus on increasing gas mileage - Make electric cars with higher performance that can go longer distances - Make cars large enough to fit larger families* *participant2053’s response (rank #5): “…Make vehicles that contain fuel efficient technologies in more then just the small car genre. We are a family of 5 and need to transport a wheelchair for our middle son. Most super economical cars wont carry us as a family. I like the idea of electric vehicles but would need to see them able to go greater distances before needing to be charged…”

39. Top terms from participant responses

40. Top co-occurring terms from participant responses

41. Post Survey Analysis

42. Q105. What was your overall opinion of the Opinion Space site?

43. Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space activity?

44. Q130_1. I was motivated to consider responses from people far from my own response on the opinion map

45. Q130_2. The size of each point was a reasonable measure of how insightful the corresponding response was

46. Q135. How do you feel about using two sliders (agreement, insightfulness) to rate responses?

47. Q140_1. I would have liked to comment on other participants' responses

48. Q140_2. I would have liked to have been notified when my response was rated by other participants

49. Q150. How willing would you be to participate in other versions of this system (on related topics)?

50. Q110. What aspects of the Opinion Space interface did you like? Main Ideas: -Ability to explore others’ ideas and see how similar or different you are from others - Learning a lot from reading the opinions of others; opened up from previous way of thinking - The ability to rate others opinions - Allowed ongoing participation over time so participants could come back - Sense of being heard - Interesting change of pace; fun, fast, different, unique, and easy to use - Enjoyed the combination of statements and free form answer box - Enjoyed the anonymity -The UI is very visually pleasing, uncluttered, and intuitive - The ability to see who has rated you - Topics were of high interest - The star metaphor with larger points meaning more insightful idea - Engaging and made excited to rate the opinions of others “I really enjoyed being able to read and rate other participants' responses. I found that there were many with which I didn't necessarialy agree with but were well thought out and concisely argued. That gave me the chance to rate highly an opinion other than my own and also opened my eyes to other sides of "the argument" so to speak.”

51. Q110. What aspects of the Opinion Space interface did you like? “I liked the way we could change screens to get to the ones we didn't rate yet, instead of it all being on one page in some kind of a continuous boring list…” “I loved reading other members ideas and/or opinions - some were very interesting and I even learned something from some! Other ideas/opinions were unique and things that I never thought about.” “…the idea of an incentive and/or prize (not to mention just the morale-boost of having others treat my own opinions with respect and interest) was itself an incentive to read as many entries as I could. It was a contest of sorts, but a most supportive and kind one, not the cutthroat, nasty kind that I try to steer clear of. I was so sorry when it ended!...”

52. Q115. What aspects of the Opinion Space interface could be improved (and how)? Main Ideas: - Include a tutorial, more instructions, and/or a demo video on how to use the system and interface - See more details on each participant - Allow commenting on your response - Ability to see who has rated you and get feedback on your ideas - Ability to create categories and clusters - Want a dedicated page to visit as opposed to finding link through email - Keep the system online longer - Add a spell checker - Include background info on discussion topics - More real time information and interaction with other participants - Want a way to view the higher level trends in the system Main Ideas: - Include a tutorial, more instructions, and/or a demo video on how to use the system and interface - See more details on each participant - Allow commenting on your response - Ability to see who has rated you and get feedback on your ideas - Ability to create categories and clusters - Want a dedicated page to visit as opposed to finding link through email - Keep the system online longer - Add a spell checker - Include background info on discussion topics - More real time information and interaction with other participants - Want a way to view the higher level trends in the system - Use a mouse-over as opposed to clicking to view a participant’s response - Organization of who you’ve rated and who rated you; a list here makes sense - Have system give suggestions of whose ideas are similar/different than your own - Add additional questions to make conversation more in-depth - Have more reminder emails with notifications of the status of the system (e.g. new responses to be rated). - Increase size of the stars - Increased features for navigation (see all stars, zoom in and out). - Rewards for everyone who participates - Larger viewing box for long responses

53. Q145. What new features would you like to see in Opinion Space?

54. Subgroup Analysis of Participation 100 Least Active vs. 100 Most Active Reviewers

55. Age of 100 least active reviewers MeanMedianStdev 43.4840.513.29772913 100 Least Active vs. 100 Most Active Reviewers Age Age of 100 most active reviewers MeanMedianStdev 49.765111.17149945

56. 100 Least Active vs. 100 Most Active Reviewers Gender Gender of 100 least active reviewers MaleFemaleRatio (M:F) 49510.96:1 Gender of 100 most active reviewers MaleFemaleRatio 56441.27:1

57. 100 Least Active vs. 100 Most Active Reviewers Education Education of 100 least active reviewersEducation of 100 most active reviewers MedianMax College Graduate (4 years) MedianMax College Graduate (4 years)

58. 100 Least Active vs. 100 Most Active Reviewers Income Income of 100 least active reviewers Median $75,000 to $99,999 Income of 100 most active reviewers Median $75,000 to $99,999

59. 100 Least Active vs. 100 Most Active Reviewers Region Region of 100 least active reviewersRegion of 100 most active reviewers

60. 100 Most Active vs. 50 Most Active Reviewers Subgroup Analysis of Participation

61. 100 Most Active vs. 50 Most Active Reviewers Age Age of 100 most active reviewers MeanMedianStdev 49.765111.17149945 Age of 50 most active reviewers MeanMedianStdev 50.885310.69885975

62. 100 Most Active vs. 50 Most Active Reviewers Gender Gender of 100 most active reviewers MaleFemaleRatio 56441.27:1 Gender of 50 most active reviewers MaleFemaleRatio 30201.5:1

63. 100 Most Active vs. 50 Most Active Reviewers Education Education of 100 most active reviewers Education of 50 most active reviewers MedianMax College Graduate (4 years) MedianMax College Graduate (4 years)

64. 100 Most Active vs. 50 Most Active Reviewers Income Income of 100 most active reviewers Median $75,000 to $99,999 Income of 50 most active reviewers Median $75,000 to $99,999

65. 100 Least Active vs. 100 Most Active Reviewers Region Region of 100 most active reviewers Region of 50 most active reviewers

66. 50 Most Active Reviewers vs. 50 Top Authors Subgroup Analysis of Participation

67. 50 Most Active Reviewers vs. 50 Top Authors Age Age of 50 most active reviewers MeanMedianStdev 50.885310.69885975 Age of 50 top authors MeanMedianStdev 47.7499.428149341

68. Gender of 50 most active reviewers MaleFemaleRatio 30201.5:1 50 Most Active Reviewers vs. 50 Top Authors Gender Gender of 50 top authors MaleFemaleRatio 28221.27:1

69. Education of 50 most active reviewers 50 Most Active Reviewers vs. 50 Top Authors Education Education of 50 top authors MedianMax College Graduate (4 years) MedianMax College Graduate (4 years)Postgraduate College

70. Income of 50 most active reviewers Median $75,000 to $99,999 50 Most Active Reviewers vs. 50 Top Authors Income Income of 50 top authors Median $75,000 to $99,999

71. Region of 50 most active reviewers 50 Most Active Reviewers vs. 50 Top Authors Region Region of 50 top authors

72. Subgroup Analysis of Participation Post Survey Results *red text indicates statistically significant differences 100 Least Active Reviewers 100 Most Active Reviewers 50 Most Active Reviewers 50 Top Authors Aggregate Results Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space activity? 100 Least Active Reviewers 100 Most Active Reviewers 50 Most Active Reviewers 50 Top Authors Aggregate Results Q105. What was your overall opinion of the Opinion Space site?

73. Q130(1). I was motivated to consider responses from people far from my own response on the opinion map 100 Least Active Reviewers 100 Most Active Reviewers 50 Most Active Reviewers 50 Top Authors Aggregate Results Q130 (2) The size of each point was a reasonable measure of how insightful the corresponding response was 100 Least Active Reviewers 100 Most Active Reviewers 50 Most Active Reviewers 50 Top Authors Aggregate Results Subgroup Analysis of Participation Post Survey Results *red text indicates statistically significant differences

74. Q150. How willing would you be to participate in other versions of this system (on related topics)? 100 Least Active Reviewers 100 Most Active Reviewers 50 Most Active Reviewers 50 Top Authors Aggregate Results Subgroup Analysis of Participation Post Survey Results *red text indicates statistically significant differences

75. Of all statistically significant differences (95% confidence interval) Subgroup Analysis of Statement Ratings

76. Subgroup Analysis of Statement Ratings (statistically significant differences) Statement 1: When deciding on which major product to buy, I often consider where it was made in making my decision. Education = Postgraduate Income = $125k - $150k Aggregate Results Statement 2: Reducing oil consumption should be a higher priority in the US Income = $15k - $25k Region = Northeast Aggregate Results Region = West

77. Statement 3: I am willing to pay more for a product that comes from a socially responsible corporation. Age = 30 - 39 Age = 40 - 49 Aggregate Results Education = College Graduate Gender = Male Region = West YAM YEF Subgroup Analysis of Statement Ratings (statistically significant differences)

78. Statement 4: The U.S. government should not give loans to corporations. Education = High School Education = Some College Aggregate Results YAM Income = $25k - $35k Income = $35k - $50k Subgroup Analysis of Statement Ratings (statistically significant differences)

79. Statement 5: New technologies are causing more problems than they are solving. Aggregate Results Age = 18 - 29 Age = 50 - 59 Education = High School Gender = Female Gender = Male Income = $125k - $150k Income = $150k - $200k Income = $25k - $35k Income = $50k - $75k Subgroup Analysis of Statement Ratings (statistically significant differences)

80. Subgroup Analysis of Post Survey Ratings Of all statistically significant differences (95% confidence interval)

81. Subgroup Analysis of Post Survey Ratings (statistically significant differences) Q105. What was your overall opinion of the Opinion Space site? Aggregate Results Education = Postgraduate Education = Trade/Vocational School Income = $100k - $125k Region = Northeast YAM

82. Subgroup Analysis of Post Survey Ratings (statistically significant differences) Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space activity? Aggregate Results Age = 50 - 59 Education = Postgraduate Region = West Q130(1). I was motivated to consider responses from people far from my own response on the opinion map Aggregate Results Age = 60 - 70 Education = College Graduate

83. Subgroup Analysis of Post Survey Ratings (statistically significant differences) Q130 (2) The size of each point was a reasonable measure of how insightful the corresponding response was Aggregate Results Age = 30 - 39 Age = 60 – 70 YAM Q135. How do you feel about using two sliders (agreement, insightfulness) to rate responses? Aggregate Results Age = 60 - 70 Age = 30 - 39

84. Subgroup Analysis of Post Survey Ratings (statistically significant differences) Q150. How willing would you be to participate in other versions of this system (on related topics)? Aggregate Results Gender = Female Gender = Male Income = $100k - $125k Income = $15k - $25k Income = $50k - $75k

85. Summary of Subgroup Findings

86. Young Affluent Males (YAM)

87. Aggregate rating statistics Subgroup rating statistics Statements: 1)When deciding on which major product to buy, I often consider where it was made in making my decision. 2)Reducing oil consumption should be a higher priority in the U.S. 3)I am willing to pay more for a product that comes from a socially responsible corporation. 4)The U.S. government should not give loans to corporations. 5)New technologies are causing more problems than they are solving.

88. Young Educated Females (YEF)

89. Aggregate rating statistics Subgroup rating statistics Statements: 1)When deciding on which major product to buy, I often consider where it was made in making my decision. 2)Reducing oil consumption should be a higher priority in the U.S. 3)I am willing to pay more for a product that comes from a socially responsible corporation. 4)The U.S. government should not give loans to corporations. 5)New technologies are causing more problems than they are solving. Young Educated Females (YEF)

90. Incentive Analysis

91. Subgroup Response to Q120: “How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space activity?” SubgroupQ120 MeanQ120 Stdev Top 100 Reviewers3.4505494511.206791553 Top 50 Reviewers3.3695652171.340971349 income-$125,000 to $149,9993.0384615381.181506076 age-18-293.0126582281.237531955 Top 50 Authors31.268761639 YEF2.9489051091.234073238 YAM2.9393939391.179095543 age-30-392.9330143541.21604544 income-$50,000 to $74,9992.9090909091.231660693 region-Northeast2.8947368421.263556556 100 least active reviewers 2.6266666671.230429013 …

92. Measurements of Participant Activity MeanStdev Insight ratings per user45.56142.52 Agreement ratings per user46.03142.57 Session Length (min)12.6911.25 Average Session Count3.005.89 User Events46.46 Total comments rated105,861.00 Total comments viewed118,737.00 Total Sessions3,147.00 Comments viewed per session 37.73 Insight ratings per viewed comment 0.81 Agreement ratings per viewed comment 0.82 AIOC Opinion Space – All Users MeanStdev Insight ratings per user42.48132.37 Agreement ratings per user42.95132.44 Session Length (min)12.5911.13 Average Session Count3.005.90 User Events44.68 Total comments rated90,017.00 Total comments viewed110,804.00 Total Sessions3,101.00 Comments viewed per session 35.73 Insight ratings per viewed comment 0.80 Agreement ratings per viewed comment 0.81 AIOC Opinion Space – Rogue Raters Removed

93. Measurements of Participant Activity MeanStdev Insight ratings per user3.2018.84 Agreement ratings per user3.3418.89 Session Length (min)16.1111.41 Average Session Count2.001.54 User Events15.21 Total comments rated7,520.00 Total comments viewed20,326.00 Total Sessions1,755.00 Comments viewed per session 11.58 Insight ratings per viewed comment 0.31 Agreement ratings per viewed comment 0.32 Unilever Opinion Space – 1939 Total users MeanStdev Insight ratings per user5.0410.78 Agreement ratings per user5.1110.67 Session Length (min)12.2411.08 Average Session Count2.001.29 User Events43.99 Total comments rated2,072.00 Total comments viewed9,881.00 Total Sessions390.00 Comments viewed per session 25.34 Insight ratings per viewed comment 0.19 Agreement ratings per viewed comment 0.19 Opinion Space 3.0 (state.gov/opinionspace) – 370 Total users

94. Measurements of Participant Activity MeanStdev Insight ratings per user4.8922.05 Agreement ratings per user4.8521.58 Insight ratings per viewed comment 0.19 Agreement ratings per viewed comment 0.18 Opinion Space 2.0 previously at state.gov/opinionspace – 5711 Total users

95. Rogue Rater Analysis

96. Summary

97. Appendix

99. Google Moderator