Download presentation
Presentation is loading. Please wait.
Published byMervyn Garrison Modified over 8 years ago
1
Earth Science Datacasting Informed Pull of GHRSST data using Really Simple Syndication (RSS) http://datacasting.jpl.nasa.gov http://datacasting.jpl.nasa.gov Earth Science Datacasting Informed Pull of GHRSST data using Really Simple Syndication (RSS) http://datacasting.jpl.nasa.gov http://datacasting.jpl.nasa.gov Andrew Bingham Bob Deen, Kevin Hussey Tim Stough, Sean McCleese, Nick Toole Andrew Bingham Bob Deen, Kevin Hussey Tim Stough, Sean McCleese, Nick Toole Funded through NASA ROSES/ACCESS Program
2
Overview Overview of RSS and podcasting Datacasting approach Demo GDAC use and request to data providers to consider setting up Datacasting Feeds Overview of RSS and podcasting Datacasting approach Demo GDAC use and request to data providers to consider setting up Datacasting Feeds
3
The Datacasting Concept Enable discover and/or automatic download of only the data files that meet a predefined need Need all data for a specific event (flood, fire, storm, oil spill etc) Need only cloud free data Need only data that meet (or fail) a quality threshold Datacasting allows users to easily and simply get at the data they need to complete a specific task: Eliminates costly manual data searches Automatically download data that is needed Provides automated data thinning and consequently reduces data volume Enables subscriptions Builds on common web-based standards (RSS) A possible solution for integrating and correlating disparate information sources Enable discover and/or automatic download of only the data files that meet a predefined need Need all data for a specific event (flood, fire, storm, oil spill etc) Need only cloud free data Need only data that meet (or fail) a quality threshold Datacasting allows users to easily and simply get at the data they need to complete a specific task: Eliminates costly manual data searches Automatically download data that is needed Provides automated data thinning and consequently reduces data volume Enables subscriptions Builds on common web-based standards (RSS) A possible solution for integrating and correlating disparate information sources
4
Really Simple Syndication (RSS) NYT > Home Page http://www.nytimes.com/index.html New York Times > Breaking News, World News en-us Copyright 2007 The New York Times Company Sun, 21 Oct 2007 03:05:01 GMT NYT > Home Page http://graphics.nytimes.com/images/section/NytSectionHeader.gif http://www.nytimes.com/index.html Tighter Border Delays Re-entry by U.S. Citizens http://www.nytimes.com/2007/10/21/us/21border.html After decades of being waved through, Americans returning from Mexico are increasingly being checked and questioned like foreigners, leading to wait times of two hours or more. JULIA PRESTON http://www.nytimes.com/2007/10/21/us/21border.html Sun, 21 Oct 2007 02:55:12 GMT In Myanmar, Fear Is a Constant Companion http://www.nytimes.com/2007/10/21/world/asia/21myanmar.html Beneath the ominous calm that has settled since the recent uprising is anger, uncertainty and hopelessness. CHOE SANG-HUN http://www.nytimes.com/2007/10/21/world/asia/21myanmar.html Sun, 21 Oct 2007 02:57:29 GMT ` RSS is a family of web feeds for publishing frequently updated content such as blog entries, news headlines or podcasts
5
Aggregation Services
6
Podcasting All About Everything http://www.example.com/podcasts/everything/index.html en-us A show about everything John Doe A show about everything Shake Shake Shake Your Spices John Doe A short primer on table spices This week we talk about salt and pepper shakers http://example.com/podcasts/archive/aae20050615.m4a Wed, 15 Jun 2005 19:00:00 GMT 7:04
7
Datacasting Approach Datacasting contributes to the RSS family through publishing of updated Earth Science data streams Scenario Data providers create RSS feeds for a data stream - update each time a new data file is available Feed uses the RSS standard, with extensions that describe The web location of the file for downloading The description & structure of the files for data extraction The content & meaning of the data for filtering Users subscribe to RSS feeds and are notified when new data files are available Users create “filters” based on a need, which in turn selects and download the files After the file is on the users hard drive, the user has the option to visually inspect the data Datacasting contributes to the RSS family through publishing of updated Earth Science data streams Scenario Data providers create RSS feeds for a data stream - update each time a new data file is available Feed uses the RSS standard, with extensions that describe The web location of the file for downloading The description & structure of the files for data extraction The content & meaning of the data for filtering Users subscribe to RSS feeds and are notified when new data files are available Users create “filters” based on a need, which in turn selects and download the files After the file is on the users hard drive, the user has the option to visually inspect the data
8
RSS Feeds for Datacasting GHRSST-L4 Hurricane Sea Surface Temp http://podaac.jpl.nasa.gov/hurricanes Daily Sea Surface Temperatures from GHRSST for Hurricanes GHRSST: UKMO's OSTIA, NCDC's AVHRR and AMSR+AVHRR Wed, 17 Oct 2007 22:52:12 GMT Datacasting Feed Publishing Tools http://datacasting.jpl.nasa.gov/datacasting.html NARI at 2007-09-17 12:00 GMT http://podaac.jpl.nasa.gov/cgi-bin/hurr/storm.pl?id=12W&date=20070917&inst=GI Mon, 17 Sep 2007 00:00:00 GMT Mon, 17 Sep 2007 23:59:00 GMT 15.000000 120.000000 40.000000 145.000000 <enclosure url="http://podaac.jpl.nasa.gov/hurricanes/dataGI/OSTIA/2007/20070917-UKMO-L4HRfnd-GLOB-v01-fv02-OSTIA- 12W.nc.gz" length="299728" type="application/x-gzip”/ http://podaac.jpl.nasa.gov/hurricanes/OSTIA/2007/20070908-UKMO-L4HRfnd-GLOB-v01-fv02- OSTIA12W.jpg Wed, 19 Sep 2007 00:55:00 GMT PO.DAAC Hurricanes GHRSST-L4 Hurricane Sea Surface Temp http://podaac.jpl.nasa.gov/hurricanes Daily Sea Surface Temperatures from GHRSST for Hurricanes GHRSST: UKMO's OSTIA, NCDC's AVHRR and AMSR+AVHRR Wed, 17 Oct 2007 22:52:12 GMT Datacasting Feed Publishing Tools http://datacasting.jpl.nasa.gov/datacasting.html NARI at 2007-09-17 12:00 GMT http://podaac.jpl.nasa.gov/cgi-bin/hurr/storm.pl?id=12W&date=20070917&inst=GI Mon, 17 Sep 2007 00:00:00 GMT Mon, 17 Sep 2007 23:59:00 GMT 15.000000 120.000000 40.000000 145.000000 <enclosure url="http://podaac.jpl.nasa.gov/hurricanes/dataGI/OSTIA/2007/20070917-UKMO-L4HRfnd-GLOB-v01-fv02-OSTIA- 12W.nc.gz" length="299728" type="application/x-gzip”/ http://podaac.jpl.nasa.gov/hurricanes/OSTIA/2007/20070908-UKMO-L4HRfnd-GLOB-v01-fv02- OSTIA12W.jpg Wed, 19 Sep 2007 00:55:00 GMT PO.DAAC Hurricanes Dataset info & definition of custom metadata Granule info (including custom metadata values)
9
System Configuration Data provider creates XML feeds and provides access to files using the Datacasting Feed Publishing software Users subscribe and download relevant files using the Datacasting Feed Reader software Data provider creates XML feeds and provides access to files using the Datacasting Feed Publishing software Users subscribe and download relevant files using the Datacasting Feed Reader software
10
Datacasting Tools Publishing Software A set of easy to use, portable, Python based tools for publishing your data Text based configuration file specifies location of repository, data format, and other information for the feed Publishing Software A set of easy to use, portable, Python based tools for publishing your data Text based configuration file specifies location of repository, data format, and other information for the feed Feed Reader Uses an email reader type interface Subscribes to feeds Filters available data files and downloads only what the user needs Stores and manages files for later use Previews data for interactive selection Feed Reader Uses an email reader type interface Subscribes to feeds Filters available data files and downloads only what the user needs Stores and manages files for later use Previews data for interactive selection
11
Rich Metadata Filtering is a function of the metadata Some metadata are part of the core Datacasting XML specification, e.g. Location or Extent Acquisition or Start time Other metadata can be defined by the data provider and included in the XML feed, for example: % of cloud free pixels Min, mean & max of data parameter Event information capture in the data (algae blooms, hurricanes, fires……) Quality control parameters Filtering is a function of the metadata Some metadata are part of the core Datacasting XML specification, e.g. Location or Extent Acquisition or Start time Other metadata can be defined by the data provider and included in the XML feed, for example: % of cloud free pixels Min, mean & max of data parameter Event information capture in the data (algae blooms, hurricanes, fires……) Quality control parameters
12
Demo
13
Feed Reader start up
14
Acquire Datcasting Feed URL
15
Add Datacasting Feed
16
Interrogate feed
18
Add RSS feed
19
Create a filter
20
Reduced list based on need – configure to download file
21
Combine other Datacasting Feeds
22
Mashup - Combining data and information from more than one source into a single integrated tool - Creating a new and distinct web service that was not originally envisaged by either source
23
Now What? Need data providers to create Datacasting Feeds http://datacasting.jpl.nasa.gov Need suggestions on what metadata should be included in the GDAC feeds Current: Time, space extents % of valid pixels Max & Min SST, AOD, SSI, Wind Speed, Deviation from previous day SST Anything else? Quality parameters? E.g. GMPE or Pixie metrics Statistical parameters that summarize the granule content Need data providers to create Datacasting Feeds http://datacasting.jpl.nasa.gov Need suggestions on what metadata should be included in the GDAC feeds Current: Time, space extents % of valid pixels Max & Min SST, AOD, SSI, Wind Speed, Deviation from previous day SST Anything else? Quality parameters? E.g. GMPE or Pixie metrics Statistical parameters that summarize the granule content
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.