Name of presentation

DNC Voter File 2006-2009 National Swap Agreement with State Parties, so that DNC provides data services and all voter file data collected co-owned by the DNC and Parties DNC provides national uniform interface - VAN Presidential Primary Candidates and Obama campaign appended millions of IDs National Database maintained at the DNC so that no data is lost and analytics projects possible

DNC Data Assets Voter File # of voters Maps of IDs (Maybe over time) – by type Users / committees 44 States with 2008 History (map)

Almost every state has 2008 history

DNC Data Assets Donor, Volunteers and Web

Current Voter File Process Voter files from Secretaries of State Standardize/Hygiene NCOA/ InfoUSA/ APT/ geocode appends Send to VoteBuilder Load to National Database Field Data from VAN E.T.L is the glue of the voter file process.

Current DNC Infrastructure 3 TB Data Warehouse Appliance Address cleansing software USPS NCOA (national change of address) software Sunopsis/Now Oracle Data Integrator ETL Product Geocodes VAN sync

Current Data Ecosystem Voter file Web Data Volunteer Data NGP data VAP data? Registration data MIKE TO DRAW DIAGRAM FOR THIS SLIDE

Successes and Challeges What we do well Custom solutions for state specific source file flaws and peculiarities Great at servicing local party and candidate needs Well integrated VAN/DNC team Mapping almost all source data into a usable form in VoteBuilder Fundraising modeling Challenges Updating voter files takes too long Incomplete integration with all DNC systems Need a national person identifier across the country ETL Vendor has been acquired by Oracle No real-time services for other applications

Where we need to be 2010 and 2012 Build Upon Successes with solutions to challenges National Person Identifier with services for other applications Faster Voter File Refresh Process enabling more frequent updates Real Time Record Updating Expanded Reporting Capability Real Time Microtargeting

Internal Development Strategy

Goal: Build upon existing infrastructure to support the State Parties, Organizing for America, and Democratic Campaigns in 2010 and 2012.

Assessment DNC Commissioned ThoughtWorks to do a voter file assessment Lasted six weeks

Two Projects ETL Replacement – enabling further processing automation National ID and Services Implementation

What do ETL Products do – Extract, Transform, Load

1. ETL Replacement Allow additional automation without sacrificing state-specific data customizations Replace with lower-cost and state of the art technology solution Lowering barrier to entry for Voter File Staff Useful for integrating other DNC data silos – Donor data, Web data

recommendations Narrowed down to two different ETL vendors – both Open Source Commercial options Split voter file team between Operations and Development – and continually swap roles Additional state-specific processes Additional file preprocessing

2. National ID & API De-duping national voter and person database Provide service which is arbiter of unique person records

De-duping across databases and nationally DIAGRAM OUTLINING HOW DEDUPING ACROSS DATABASES AND STATES WORKS

Exposed Services Match Person to National Database Match or Add Person to National Database Give or get external ID for Person Add/Update Contact Info for Person Get Person Properties

Architecture Diagram

Person service Vote For Change VF Updates BSD/Online Tools NGP/New Donors Welcome to the DNC’s Person Service, let me see if I know you… Yes No What can I do for you today? Update your info? Return more data? Link to other systems? Welcome, let’s get your in as a new person and store it for all time… Need to check real quick (<1 second) that you are allowed to have this information… You were served in <1 second. Come back again! Yes

Who are the new registrants so we can add them to our GOTV universe? With an automated ETL process, we can add them to the VAN in a couple days But how likely are they to vote for us? Are they all GOTV-able? With services that return their microtargeting scores, we can tell you right away

Give me the email addresses from BSD of all the people I am about to put in the mail

Is this person registered to vote already? No? Ok, lets give them a National Person ID, store their info, and send them to VoteBuilder so field can chase them

Person service Vote for Change Is John Smith registered? Service Layer

New Voter File Process

Team: ETL Migration and National ID Architect Senior Developer Developer Thoughtworks Iteration Manager (Part-time) ETL Vendor Developer DNC Developer

Integration with Labs program User-friendly data tools for State Parties and campaigns Stories.Barack Obama.com National ID Service Call Congress Tools My.Barack Obama.com

DNC Staffing

Risks Timeline slipping – interferes with 2010 campaign programs Budget slipping – extra development hours Can’t continue to staff voter file project Duplicate energy in the progressive community End up deciding to use Catalist for re-elect in 2012

Budget for Capital Investments

Timeline

Why? Attention to detail in voter file processing Control and ownership of data Control and direction of infrastructure Lack of dependence on a single vendor that can’t be changed (ala BSD, VAN) Flexibility – Building can turn on a dime.

DELETED SLIDES

What we do well Custom solutions for state source file flaws and peculiarities Great at servicing local party and candidate needs Well integrated VAN/DNC team Mapping almost all source data into a usable form in VoteBuilder Fundraising modeling

State of Voter Files

Challenges Updating voter files takes too long Incomplete integration with all DNC systems Need a national person identifier across the country ETL Vendor has been acquired by Oracle Need more real-time services for other applications

Current DNC Process Collect voter files from Secretaries of State around the country Standardize files Hygiene the file Append Phone Numbers & Consumer Data Match against National Database Upload to VAN