Data Science Publication for NSF Polar Cyberinfrastructure Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist Semantic Community November 4,
Preface Some prep work is already underway (if you scour the Open Science Codefest site you will find some) to prepare some datasets of relevance to the Polar community. We will provide some of this prepared data to interested parties ahead of the workshop in the next few weeks in case folks want to start hacking early. We will tweet under the hash tag: #nsfpolardatavis 2 Source: Chris Mattmann
Overview Build the knowledge base (in MindTouch) and spreadsheet (in Excel) first, which then makes the Spotfire (data browser) application easier to “storify” the results. Follow the Cross-Industry Data Mining Standard by: – 1 Business Understanding (of the Hackathon), – 2 Data Understanding (by mining the Sessions), – 3 Data Preparation (by screen scraping and downloading), – 4 Modeling (enough data for statistical significance?), – 5 Evaluation (How collected?, Where stored?, What results?, and Believe them?; and – 6 Deployment (Story and Demo). The documentation will be in the form of the Data Science Publication for NSF Polar Cyberinfrastructure. My goal is to see if I can integrate and federate these multiple data sources. 3
Data Science for Business: Data Mining Process 4 Source: Data Science for Business: Chapter 2, 2014
Data Science for NSF Polar Cyberinfrastructure: Knowledge Base 5 Data Science for NSF Polar Cyberinfrastructure
Possible Data Sets Scour the Open Science Codefest: – codefest/issues/26 codefest/issues/26 The Polar Data Catalogue (YES): – BCO-DMO (YES): – Polar Hub (NO): – The AMRC at University of Wisconsin-Madison (YES): – ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/ ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/ 6
Open Science Codefest: NASA/NSF/NSIDC Data Sets NASA Antarctic Master Directory – A master directory for arctic data sets tal=amd&KeywordPath=Parameters|CRYOSPHERE&Metadat aType=0&lbnode=mdlb2 tal=amd&KeywordPath=Parameters|CRYOSPHERE&Metadat aType=0&lbnode=mdlb2 NSF ACADIS Gateway – NSF data repository for arctic/polar data NSIDC Arctic Data Explorer – National Snow and Ice Data Center repository 7 Source: Link to presentation given at Open Science CodefestLink to presentation given at Open Science Codefest
Polar Data Catalogue: Home Page 8
Polar Data Catalogue: Collections 9
Polar Data Catalogue: Search 10
Polar Data Catalogue: Canadian Lake Ice Database MB MDB
Polar Data Catalogue: Sea Ice Thickness in Southern Beaufort Sea 12 Downloaded 5 Files 4 Text and 1 ZIP (Shape) 1.3 MB
Polar Data Catalogue: Spreadsheet 13 Canadian Lake Ice Database Sea Ice Thickness in Southern Beaufort Sea
BCO-DMO 14 Tutorial PDF
Data Access Tutorial 2014 OCB PI Summer Workshop How to Submit Data Data access: TEXT-BASED SEARCH scenario 1: – You have a general idea of what you are looking for. Data access: MAP BROWSE scenario 2: – You are interested in data from a particular geographic region. Data access: MAP KEYWORD SEARCH scenario 3: – You are interested in data of a particular type from a particular geographic area. Data access: MAP SEMANTIC SEARCH scenario 4: – You have an idea what you are looking for, but you do not know the Program, Project, or Deployment name. Glossary of Terms Acknowledgments Follow BCO-DMO 15 My Question: Could Spotfire do all of this?
BCO-DMO Datasets 16
BCO-DMO MapServer Geospatial Interface 17
Polar Hub: A Global Hub for Polar Data Discovery 18 My Question: Where is the Data?
The AMRC at University of Wisconsin-Madison NameSizeDate Modified [parent directory] Ant_IR_area/9/30/14, 8:18:00 PM Ant_IR_netCDF/9/30/14, 8:37:00 PM AWS_dat_MAY_2014/Text Files: Spotfire?9/30/14, 3:23:00 PM AWS_q10_MAY_2014/Text Files: Spotfire?9/30/14, 3:24:00 PM AWS_q1h_MAY_2014/Text Files: Spotfire?9/30/14, 3:25:00 PM AWS_q3h_MAY_2014/Text Files: Spotfire?9/30/14, 3:27:00 PM AWS_r_MAY_2014/Text Files: Spotfire?9/30/14, 3:22:00 PM readme.txt4.6 kB10/6/14, 6:37:00 PM 19 Index of /pub/requests/DVPC/ ftp://amrc.ssec.wisc.edu/pub/requests/DVPC/
Data Science for NSF Polar Cyberinfrastructure: Spreadsheet Knowledge Base 20
Data Science for NSF Polar Cyberinfrastructure: Spotfire Cover Page 21 Web Player
Data Science for NSF Polar Cyberinfrastructure: Spotfire Visualizations 22 Web Player