1. Name of presentation

2. DNC Voter File
National Swap Agreement with State Parties, so that DNC provides data services and all voter file data collected co-owned by the DNC and Parties
DNC provides national uniform interface - VAN
Presidential Primary Candidates and Obama campaign appended millions of IDs
National Database maintained at the DNC so that no data is lost and analytics projects possible

3. DNC Data Assets Voter File
# of voters
Maps of IDs (Maybe over time) – by type
Users / committees
44 States with 2008 History (map)

4. Almost every state has 2008 history

5. DNC Data Assets Donor, Volunteers and Web

6. Current Voter File Process
Voter files from Secretaries of State
Standardize/Hygiene
NCOA/ InfoUSA/ APT/ geocode appends
Send to VoteBuilder
Load to National Database
Field Data from VAN
E.T.L is the glue of the voter file process.

7. Current DNC Infrastructure
3 TB Data Warehouse Appliance
Address cleansing software
USPS NCOA (national change of address) software
Sunopsis/Now Oracle Data Integrator ETL Product
Geocodes
VAN sync

8. Current Data Ecosystem
Voter file
Web Data
Volunteer Data
NGP data
VAP data?
Registration data
MIKE TO DRAW DIAGRAM FOR THIS SLIDE

9. Successes and Challeges
What we do well
Custom solutions for state specific source file flaws and peculiarities
Great at servicing local party and candidate needs
Well integrated VAN/DNC team
Mapping almost all source data into a usable form in VoteBuilder
Fundraising modeling
Challenges
Updating voter files takes too long
Incomplete integration with all DNC systems
Need a national person identifier across the country
ETL Vendor has been acquired by Oracle
No real-time services for other applications

10. Where we need to be 2010 and 2012
Build Upon Successes with solutions to challenges
National Person Identifier with services for other applications
Faster Voter File Refresh Process enabling more frequent updates
Real Time Record Updating
Expanded Reporting Capability
Real Time Microtargeting

11. Internal Development Strategy

12. Goal: Build upon existing infrastructure to support the State Parties, Organizing for America, and Democratic Campaigns in 2010 and 2012.

13. Assessment DNC Commissioned ThoughtWorks to do a voter file assessment
Lasted six weeks

14. Two Projects
ETL Replacement – enabling further processing automation
National ID and Services Implementation

15. What do ETL Products do – Extract, Transform, Load

16. 1. ETL Replacement
Allow additional automation without sacrificing state-specific data customizations
Replace with lower-cost and state of the art technology solution
Lowering barrier to entry for Voter File Staff
Useful for integrating other DNC data silos – Donor data, Web data

17. recommendations
Narrowed down to two different ETL vendors – both Open Source Commercial options
Split voter file team between Operations and Development – and continually swap roles
Additional state-specific processes
Additional file preprocessing

18. 2. National ID & API De-duping national voter and person database
Provide service which is arbiter of unique person records

19. De-duping across databases and nationally
DIAGRAM OUTLINING HOW DEDUPING ACROSS DATABASES AND STATES WORKS

20. Exposed Services Match Person to National Database
Match or Add Person to National Database
Give or get external ID for Person
Add/Update Contact Info for Person
Get Person Properties

21. Architecture Diagram

22. Person service Vote For Change VF Updates BSD/Online Tools
NGP/New Donors
Welcome to the DNC’s Person Service, let me see if I know you…
Yes No
What can I do for you today? Update your info? Return more data? Link to other systems?
Welcome, let’s get your in as a new person and store it for all time…
Need to check real quick (<1 second) that you are allowed to have this information…
You were served in <1 second. Come back again!
Yes

23. Who are the new registrants so we can add them to our GOTV universe?
With an automated ETL process, we can add them to the VAN in a couple days
But how likely are they to vote for us? Are they all GOTV-able?
With services that return their microtargeting scores, we can tell you right away

24. Give me the email addresses from BSD of all the people I am about to put in the mail

25. Is this person registered to vote already?
No?
Ok, lets give them a National Person ID, store their info, and send them to VoteBuilder so field can chase them

26. Person service
Vote for Change
Is John Smith registered?
Service Layer

27. New Voter File Process

28. Team: ETL Migration and National ID
Architect
Senior Developer
Developer
Thoughtworks Iteration Manager (Part-time)
ETL Vendor Developer
DNC Developer

30. Integration with Labs program
User-friendly data tools for State Parties and campaigns
Stories.Barack Obama.com
National ID Service
Call Congress Tools
My.Barack Obama.com

31. DNC Staffing

33. Risks Timeline slipping – interferes with 2010 campaign programs
Budget slipping – extra development hours
Can’t continue to staff voter file project
Duplicate energy in the progressive community
End up deciding to use Catalist for re-elect in 2012

34. Budget for Capital Investments

35. Timeline

36. Why? Attention to detail in voter file processing
Control and ownership of data
Control and direction of infrastructure
Lack of dependence on a single vendor that can’t be changed (ala BSD, VAN)
Flexibility – Building can turn on a dime.

37. DELETED SLIDES

38. What we do well
Custom solutions for state source file flaws and peculiarities
Great at servicing local party and candidate needs
Well integrated VAN/DNC team
Mapping almost all source data into a usable form in VoteBuilder
Fundraising modeling

39. State of Voter Files

40. Challenges Updating voter files takes too long
Incomplete integration with all DNC systems
Need a national person identifier across the country
ETL Vendor has been acquired by Oracle
Need more real-time services for other applications

41. Current DNC Process
Collect voter files from Secretaries of State around the country
Standardize files
Hygiene the file
Append Phone Numbers & Consumer Data
Match against National Database
Upload to VAN