Computational Campaign Coverage with PollyVote.com Andreas Graefe graefe.andreas@gmail.com Tow Center September 13, 2016

PollyVote History The PollyVote started in 2004 as a forecasting research project and was applied Prospectively to U.S. presidential elections in 2004, 2008, 2012, the 2013 German federal election Retrospectively to U.S. presidential elections in 1992, 1996, and 2000.

PollyVote Goals Demonstrate the power of combining forecasts for improving accuracy Assess the relative accuracy of different methods over time Develop models that aid decision-making Validate (new) methods for forecasting elections Transparency and accessibility Distribute PollyVote to other countries Provide automated campaign coverage (texts and visualizations) Make the project sustainable

How does the PollyVote work? Mechanical combination of forecasts Within different components of methods by calculating unweighted averages. Indicate that this is an example of 2 methods. Also show the averages for each/

How does the PollyVote work? Mechanical combination of forecasts Within Across different components of methods by calculating unweighted averages. Combined forecasts are accurate because They use more information Errors of individual forecasts cancel out Indicate that this is an example of 2 methods. Also show the averages for each/

Relative accuracy of forecasting methods On average across the last 100 days prior to each of the past 6 elections… Polls performed worst (MAE: 2.8 percentage points) Expectations (citizen forecasts) performed best (MAE: 1.2 percentage points) Performance of methods varies (within and) across elections Mean absolute error 1992 1996 2000 2004 2008 2012 Average Polls 4.8 4.7 2.7 1.5 1.7 1.3 2.8 Prediction markets 0.8 1.2 0.6 Econometric models 3.6 3.9 3.2 1.1 2.4 Expert judgment 1.4 Index models 0.4 2.3 Citizen forecasts 0.7 1.8 COLORS AGAIN

Power of combining forecasts On average across the last 100 days prior to each of the past 6 elections the PollyVote missed the final election outcome on average by 1 percentage point. was more accurate than each of the (combined) component forecasts. Also, since 2004, the PollyVote always predicted the correct winner, at any given day. Mean absolute error 1992 1996 2000 2004 2008 2012 Average Polls 4.8 4.7 2.7 1.5 1.7 1.3 2.8 Prediction market 0.8 1.2 0.6 Econometric models 3.6 3.9 3.2 1.1 2.4 Expert judgment 1.4 Index models 0.4 2.3 Citizen forecasts 0.7 1.8 PollyVote 0.9 1.0

PollyVote is a resource for election observers Provides accurate forecasts of the election outcome

PollyVote is a resource for election observers Provides accurate forecasts of the election outcome Allows for Comparing methods/models Discovering “interesting” stories

Our project Goals: Make our content more accessible by Visualizing data Providing first drafts of a story  Develop automated news Based on data from our component methods / models In English and German language

Automated („Robot“) Journalism Process of using software or algorithms to automatically generate news stories without human intervention …after the initial programming of the algorithm

AJ in newsrooms Leading media outlets have started to experiment with automated news. Early expansion phase Few news organizations Limited topics (sports, finance, crime, weather) No prior work for political news (but see WaPo‘s Heliograf)

Status quo The good news Completely automated process, from collecting the forecasting data to publishing texts in both languages More than 13,000 texts published since April But… Many texts contain errors Some texts are missing, others are published many times This is due to errors In the raw data With the data export / import In the algorithm

Key learnings to date (I) Data management Clean and accurate data are most crucial But difficult to maintain

Key learnings to date (II) Article quality Simple texts are easy to automate That is, texts based on a single row in the data Example: Description of poll results Tradeoff: adding complexity (i.e., insights) vs. risk of introducing new errors Additions to texts had to be delayed due to constant error fixing Source of error can be difficult to detect

Key learnings to date (III) Algorithm development Large onboarding effort to educate the NLG provider about our project NLG software difficult to use (but constantly improving) Very challenging to set the rules for the algorithm, for example, How to refer to the margin in polls? When does a candidate have a momentum? When is there a trend in the data?  Boundaries of automation?