Big Data Symposium: Analytics and Applications for Federal Big Data - FEMA Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist.

Slides:

Advertisements

Similar presentations

Federal Transparency.gov As Data For the Digital Government Strategy Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Advertisements

What's on My Dashboard Today?: Governmentwide Acquisition Contract Dashboard Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist.

Lab 3 Data Sources: Locating, Importing, Manipulating URBPL 5/6010: Urban Research University of Utah Pam Perlich Revised 9/10/2006.

Data Science for Natural Medicines: Dead Doctors Don't Lie Radio

OMB Data Visualization Tool Requirements Analysis: Information Builders Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Data Science for Tackling the Challenges of Big Data

Director and Senior Data Scientist/Data Journalist

OMB Data Visualization Tool Requirements Analysis: Oracle Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Who Tweets the most about Gov20? Dr. Brand Niemann Director and Senior Data Scientist Semantic Community July 5,

OMB Data Visualization Tool Requirements Analysis: Birst Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Data.gov Wiki: A Semantic Web Approach to Government Data Li Ding, Dominic DiFranzo, Sarah Magidson, Jim Hendler Tetherless World Constellation Aug 7,

A Search for Veterans Benefits Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community December 22,

Microsoft Access Exporting Access Data and Mail Merging.

Data Science for MyFamilySearch.org Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community My Personal Family History.

OMB Data Visualization Tool Requirements Analysis: Logi Analytics Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

OMB Data Visualization Tool Requirements Analysis: QlikView Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

OMB Data Visualization Tool Requirements Analysis: Microsoft Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

NLM-Semantic Medline Data Science Data Publication Commons Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data.

Big Data and Social Media & Web Analytics Innovation Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

NIST Scientific Data for Data Science United Nations Open Data / Open Government Conference, April 26-28, Abu Dhabi

EPA Big Data Analytics: Data Science for EPA Fracturing Data Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Linked Data Visualizations for Eurostat Linked Data Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

OMB Data Visualization Tool Requirements Analysis: SAP Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

3 Round Stones: All Content As Big Data Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

Big Data Conference: Analytics and Applications for Federal Big Data Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic.

A TEDMED Data Reveal: Big and Little Dr. Brand Niemann Director and Senior Data Scientist Semantic Community AOL Government.

Imagine Everything is Before You: Past, Present, and Future Paper and Demonstration for the 2014 Family History Technology BYU Dr. Brand Niemann.

A Spotfire Demo Gallery with Data Science Dr. Brand Niemann Director and Senior Data Scientist Semantic Community November 13, 2011 DRAFT 1.

Information Sharing Begins With Me Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

GIS Data Science for Collaboration Across Communities: GIScience 2.0 and Beyond Dr. Brand Niemann Director and Senior Data Scientist Semantic Community.

Data Science Publication for NSF Polar Cyberinfrastructure Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Using Data Science as Evidence in Public Policy With Big Data and Elections Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist.

EPA Indicators of Our Health and Environment Updated and Improved Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic.

Big Data Symposium: Analytics and Applications for Federal Big Data – Bureau of Justice Statistics Dr. Brand Niemann Director and Senior Enterprise Architect.

Farm Data Dashboards: USDA and Microsoft Innovation Challenge Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data.

Data Science for Agency Initiatives 2015 Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Data Science for RDA Climate Change Data Challenge and Meetup Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data.

Data Science for DataBay DataBay "Reclaim the Bay" Innovation Challenge: August 1-3, 2014, Smithsonian Environmental Research Center, 647 Contees Wharf.

Data Science ESIP Publication Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Data Science for USGS Minerals Big Data Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data Science Data Science.

Data Science for DTIC Data Ecosystem Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

The 2012 EuroStat Regional Yearbook for Semantic Interoperability Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic.

Why Doesn't EPA Have a Self- Contained Statistical Unit?: A Tribute to Doug Engelbart Dr. Brand Niemann Director and Senior Data Scientist Semantic Community.

Data Science for EPA Big Data Analytics: Oregon Data Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Open DATA METI: All Content As Big Data Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

Data Science for Migration Data Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Health Datapalooza IV: Child and Adolescent Health Data App Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Build the NY Times Subject Headings and Topics in the Cloud Dr. Brand Niemann Director and Senior Data Scientist Semantic Community July 4,

SmartGrid and Spotfire Cloud Computing - Similarities in Innovation Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic.

Research on US Federal Government Handling of Data Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

Binary Group Knows What It Knows Because of It’s Information Attitude Brand Niemann Senior Enterprise Architect and Data Scientist August 26,

An Internet of Things: People, Processes, and Products in the Spotfire Cloud Library Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist.

Data Science for the NOAA Chief Data Officer Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Harnessing Data to Address Diabetes in the US Dr. Brand Niemann Director and Senior Data Scientist Semantic Community AOL.

Data Science for HealthCare.gov Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Department of Commerce App Challenge: Big Data Dashboards Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community.

Data Science for DoI BSEE Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data Science Data Science for DoI BSEE.

Data Science for Joint Doctrine Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data Science Data Science for Joint.

Data Science for FDA RFI Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Data Science for Conservation International's Big Ecosystem Data Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community.

NGA Demo Participant Collaboration Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

Cross Information Sharing and Integration for the Intelligence Community: 13 th SOA for eGovernment Conference Dr. Brand Niemann Director and Senior Enterprise.

NIEM 3.0 Data Analytics App Dr. Brand Niemann Director and Senior Data Scientist Semantic Community AOL Government Blogger.

Data Science for NIST Big Data Framework Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

Data Science for Global Ebola Response Data Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community Data Science Data Science.

HealthIT.gov Dashboard: Spotfire not Flash Dr. Brand Niemann Director and Senior Data Scientist Semantic Community

Data Science for the National Big Data R&D Initiative Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community

PA Crash Information Tool (Unbelted Injuries)

Spotfire 5 Users Guide Dashboard

Title: Build EPA Apps in the Cloud

Presentation transcript:

Big Data Symposium: Analytics and Applications for Federal Big Data - FEMA Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community AOL Government Blogger March 5-6,

Begin With the End in Mind Ms. Jo Strang, Associate Administrator, Safety, Federal Railroad Administration, Department of Transportation “Open Gov 2.0 and Safety.Data.Gov” – New safety data sources and challenge from the National Institute of Justice and new data from Open FEMA Open FEMA – Could not find with Google search – Start with Fema.gov Safety.Data.Gov – Could not find at that Web site – Start with Bureau of Justice Statistics 2

My Process Found Data Feeds at Fema.gov Documented the Web Site, Metadata, and Data Sources for: – USGS Earthquakes – World Map – FEMA Disaster Declarations Summary – FEMA Hazard Mitigation Program Summary – Public Assistance Subgrantee Funded Projects Detail – Public Assistance Funded Projects Summary Used Google Earth for the FEMA and USGS KML Files Imported the Above Data Sets Into Spotfire Created a Separate Visualization Tab for Each Data Set in Spotfire Started a Unified Data Architecture Submitted a Question About the Data Definitions to FEMA Documented My Data Science Work in PowerPoint Slides Provided Conclusions and Recommendations 3

Fema.gov Home Page 4 Data Feeds

Fema.gov Data Feeds 5

Fema.gov Data Feeds Examples RSS: – Disaster Declarations Query Form and List of Links KMZ: – Disaster Declared Counties - National View Designated Counties META and XLS: – Disaster Declarations Summary Same as RSS Disaster Declarations – Hazard Mitigation Program Summary Web Page – Public Assistance Funded Projects Detail No Web Page – Public Assistance Funded Projects Summary Web Page KML and CSV: – Worldwide M1+ Earthquakes, Past 7 Days (USGS) KMZ and TXT (No Web Page) My Note: But I found a Way Cool One! – 6

RSS: Disaster Declarations 7

META: Disaster Declarations Summary 8 XLS

XLS: Disaster Declarations Summary 9 View / Download / PrintView / Download / Print (xls 9004K) Instructions Metadata

Hazard Mitigation Program Summary 10

META: Hazard Mitigation Program Summary 11 XLS

XLS: Hazard Mitigation Program Summary 12 View / Download / PrintView / Download / Print (xls 16563K)

META: Public Assistance Funded Projects Detail 13 XLS

XLS: Public Assistance Funded Projects Detail 14 View / Download / PrintView / Download / Print (xls K)

Public Assistance Funded Projects Summary 15

META: Public Assistance Funded Projects Summary 16

XLS: Public Assistance Funded Projects Summary 17 View / Download / PrintView / Download / Print (xls 18935K) Instructions Metadata Projects 1-9 Applicants

Worldwide M1+ Earthquakes, Past 7 Days (USGS) 18 TXT not CSV Spotfire handles that!

Import All Data Sets (15) Into Spotfire 19

Default View in Spotfire 20 Illustrates Data Visualization Design Using Shneiderman’s Mantra: Overview First, Zoom and Filter, Then Details-on-Demand

KMZ: FEMA Designated Counties and USGS Worldwide M1+ Earthquakes, Past 7 Days (USGS) 21

USGS Earthquakes: Spotfire 22 My Note: It is hard to see at this scale, but the data points are all related in each visualization. My Note: Could Spotfire do this? Can USGS do what Spotfire does?

World Map and Data Tables: Spotfire 23 My Note: I needed a world map and to see how to integrate it with the Earthquake and other data tables. This Spotfire functionality helps me do this and is the beginning of a “unified data architecture”(Scott Gnau, President, Terra Data Labs).

FEMA Disaster Declarations Program Summary: Spotfire 24 My Note: The visualizations are connected!

FEMA Hazards Mitigation Program Summary: Spotfire 25 My Note: The visualizations are connected!

FEMA Public Assistance Funded Projects Detail: Spotfire 26

FEMA Public Assistance Funded Projects Summary: Spotfire 27

Conclusions and Recommendations USGS Earthquakes: – Could Spotfire do this? Can USGS do what Spotfire does? Google Earth of FEMA and USGS: – Need a way to access Google Earth within Spotfire. World Map: – One can easily find and import geospatial data files (Shape) into Spotfire. Unified Data Architecture: – Spotfire provides a way to produce this by displaying all the data tables and their columns. FEMA Disaster Declarations Summary: – I submitted a question about the data definitions to FEMA and am waiting for an answer. FEMA Hazard Mitigation Program Summary: – There is county data, but the names do not match those used in the standard Shape file for county maps so I had to use the State designation. Public Assistance Subgrantee Funded Projects Detail: – Parts 1 and 2 of the Data Set need to be merged. Public Assistance Funded Projects Summary: – The nine additional tabs of spreadsheet data should be used in a separate Spotfire file. 28