Broadband Session Michael Byrne

Broadband Map Technical Details Data Integration Map Presentation Since Launch

Data Integration – What we did Review data for format and content Poured data into 1 container Summary Statistics – Feedback to awardees Analyze data for completeness Incorporating speed data from Overview Wireline Statistics (any geography, population, hhs) Wireless Statistics (any geography, population, hhs)

Data Integration – Challenges / Innovations Challenge #1 – Data integrity problems –Feedback; communication; client side cleaning scripts Challenge #2 – Speed in a different geography –Substantial hand matching of providers to all records Challenge #3 – Spatial integrity of data –Parallel software processing; OGC Validation tools Challenge #4 – Spatial data complexity –Custom code for dissolving features (2/1/11) Challenge #5 – Calculate availability (pop / hhs) –Custom code for spatially distributing households

Map Presentation – What we did 1 st requirement was entirely built on RESTful services Wanted to embody the spirit of the internet (innovation, fast response, full volume of data) Wanted to serve multiple users Required to serve out 100% of data (that we could) Wanted to make data (all of it) easily consumable Wanted to make a.gov look like a.com Searchable and interactive

Map – Challenges / Innovations Challenge #1 – Time (we, like the awardees didnt have any) –Build a prototype; re-use code; –Agile development cycle –Rapid Application Development; –Separate the application development into component parts; –Focus on foundation architecture; –Use the right pieces for the solution

Map – Challenges / Innovations Challenge #2 – Provide multiple points of entry –REST –Develop an architecture to translate codes to english –Put english words in the REST url –Develop an API information architecture to support ANY combination of geography and metric –Build on standards (e.g. no plug-ins); this site is only html, CSS and JavaScript –Build reusable modules in Web Tier (e.g. WordPress) – 475 modules

Map – Challenges / Innovations Challenge #3 – how can you serve this volume of data this quickly ( Some will say we didnt ) –Cache, cache, cache We pre-cache the geospatial tiles to level 10, then cache on the fly for every spatial zoom We implemented an open source caching software to cache queries (Varnish – it returns 89% of all hits) –Develop right approach for search; and map Search is point in poly Map is dissolve –Single color maps –Focus on what we control; leave basemap to others

Map – Challenges / Innovations Challenge #3a – how do you serve this fast

Map – Challenges / Innovations Challenge #3a – how do you architect this back end –Focus on the data to make it consumable for your audience (23 tables; ~120 view; views => cached tiles) –Table structure 22 million records over 23 tables 2 tables make up 93% of all records Organized by unique values –Type of geography; Type of technology; Number of providers; Holding companies Reduce data to common denominators –Provider by Block; Technology type; download speed; upload speed –Know someone else can do it better than us

Since Launch First 11 hours –158,123,884 hits (89 served by cache) –862 GB served –Peak requests (8,970 per second; average 1,095 per second) –~500,000 visits

Since Launch

Site Launch 30,000 user feedback votes

Since Launch nal-Broadband-Map-jQuery-Demohttp:// nal-Broadband-Map-jQuery-Demo Code for America US Department of Education

The bottom line Developers who are thinking about integrating with your platform want three things. First, they want to improve their application. Second, they want to get more users. Third, they want the first two things to take as little time as possible.

Scaling to billions … Search Engine Optimization Level 1 – create enough buzz around title of project – keyword national broadband map Level 2 – optimize for combination of key words + location (broadband map and new york) Level 3 – educational and consumer oriented (broadband map speed 10 mbs) Level 4 – national and strategic keywords + provider name (national broadband map + verizon)

Scaling to billions … Level 1 – keyword national broadband map –Reached #1 w/i 48 hours of launch Level 2 – combination of key words + location –Reached spots 1 – 10 in first week of launch Level 3 – educational and consumer oriented –Reached #1 and 1 st page for 4 of our pages (Classroom, Rank, map) Level 4 – national and strategic keywords + provider –Reached #1 w/ our APIs and first page w/ Search

Thanks Michael Byrne Geographic Information Officer