NASA ROSES ACCESS Project Review Science Data Systems for Satellite and Airborne LiDAR Data 23 February 2011 C. Meertens and J. McWhirter, UNAVCO S-J.

Slides:



Advertisements
Similar presentations
Supersites infrastructure: data access services and tools Chuck Meertens, Jeff McWhirter, Fran Boler, Stuart Wier, Susanna Gross, Scott Baker, UNAVCO Supersites.
Advertisements

SAN DIEGO SUPERCOMPUTER CENTER Choonhan Youn Viswanath Nandigam, Nancy Wilkins-Diehr, Chaitan Baru San Diego Supercomputer Center, University of California,
The ICESat-2 Mission: Laser altimetry of ice, clouds and land elevation T. Markus, T. Neumann NASA Goddard Space Flight Center W. Abdalati Earth Science.
ReCover for REDD and sustainable forest management EU ReCover project: Remote sensing services to support REDD and sustainable forest management in Fiji.
Interoperable Data Systems for Satellite, Airborne and Terrestrial LiDAR Data C. Meertens and J. McWhirter, UNAVCO S-J. S.Khalsa and T. Haran, NSIDC/CU.
The GEON LiDAR Workflow: An Internet-Based Tool for the Distribution and Processing of LiDAR Point Cloud Data Christopher J. Crosby, J Ramón Arrowsmith,
2. Point Cloud x, y, z, … Complete LiDAR Workflow 1. Survey 4. Analyze / “Do Science” 3. Interpolate / Grid USGS Coastal & Marine.
Planned Title: Review of Evaluation of Geospatial Search Allan Doyle.
Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.
Update on ASU GEON Activities J Ramon Arrowsmith GEON PI Meeting Reston, VA May 13, 2006.
1 CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Global Earth Observation Grid Workshop, Bangkok, Thailand, March Integration Platform.
Data Sources & Using VIVO Data Visualizing Scholarship VIVO provides network analysis and visualization tools to maximize the benefits afforded by the.
November 2011 At A Glance GREAT is a flexible & highly portable set of mission operations analysis tools that increases the operational value of ground.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
The GEON Integrated Data Viewer (IDV) and IRIS DMC Services: CyberInfrastructure Support for Seismic Data Visualization and Interpretation Charles Meertens.
Managing IceBridge Airborne Mission Data at the National Snow and Ice Data Center M. Kaminski, M. Brodzik, J. Deems, T. Scambos National Snow and Ice Data.
© 2013 National Ecological Observatory Network, Inc. ALL RIGHTS RESERVED. THE NEON APPROACH TO DATA INGEST, CURATION, AND SHARING Christine Laney (Data.
OCLC Online Computer Library Center CONTENTdm ® Digital Collection Management Software Ron Gardner, OCLC Digital Services Consultant ICOLC Meeting April.
MAHI Research Database Data Validation System Software Prototype Demonstration September 18, 2001
AERONET Web Data Access and Relational Database David Giles Science Systems and Applications, Inc. NASA Goddard Space Flight Center.
, Increasing Discoverability and Accessibility of NASA Atmospheric Science Data Center (ASDC) Data Products with GIS Technology ASDC Introduction The Atmospheric.
, Implementing GIS for Expanded Data Accessibility and Discoverability ASDC Introduction The Atmospheric Science Data Center (ASDC) at NASA Langley Research.
ISpheres Project. Project Overview iSpheresCore iSpheresImage Demonstration References.
Updates from EOSDIS -- as they relate to LANCE Kevin Murphy LANCE UWG, 23rd September
IUScholarWorks is a set of services to make the work of IU scholars freely available. Allows IU departments, institutes, centers and research units to.
Page 1 CSISS Center for Spatial Information Science and Systems Design and Implementation of CWIC Metrics Weiguo Han, Liping Di, Yuanzheng Shao, Lingjun.
ICESat/GLAS Tools at NSIDC Melinda Marquis NSIDC Product Team Lead, AMSR-E and GLAS Feb. 16, 2005.
Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.
Enabling Access to High-Resolution LiDAR Topography through Cyberinfrastructure-Based Data Distribution and Processing Christopher J. Crosby, J Ramón Arrowsmith.
Efrat Frank, Ashraf Memon, Vishu Nandigam, Chaitan Baru
1 Ilkay ALTINTAS - July 24th, 2007 Ilkay ALTINTAS Director, Scientific Workflow Automation Technologies Laboratory San Diego Supercomputer Center, UCSD.
SIF Status to ADC Co-Chairs Siri Jodha S. Khalsa Steve Browdy.
Where to find LiDAR: Online Data Resources.
Researcher requires geographical subset due to disk space restriction. Goes directly to NSIDC site hoping for “one stop shopping” at
1 NASA CEOP Status & Demo CEOS WGISS-24 Oberpfaffenhofen, Germany October 15, 2007 Yonsook Enloe.
ESIP Federation 2004 : L.B.Pham S. Berrick, L. Pham, G. Leptoukh, Z. Liu, H. Rui, S. Shen, W. Teng, T. Zhu NASA Goddard Earth Sciences (GES) Data & Information.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Page 1 CSISS Center for Spatial Information Science and Systems CWIC Metrics: Current and Future Weiguo Han, Liping Di, Yuanzheng Shao, Lingjun Kang Center.
User Working Group 2013 Data Access Mechanisms – Status 12 March 2013
GEON2 and OpenEarth Framework (OEF) Bradley Wallet School of Geology and Geophysics, University of Oklahoma
GBIF Data Access and Database Interoperability 2003 Work Programme Overview Donald Hobern, GBIF Programme Officer for Data Access and Database Interoperability.
The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.
Science Data Systems for Satellite and Airborne LiDAR Data AKA – NASA LiDAR Access System (NLAS) NASA ROSES Access project Chuck Meertens Chris Crosby.
The goal of this NASA ACCESS-funded project is to create easy to use, multi-faceted web services for access, browse, online analysis and delivery of data.
29 March 2004 Steven Worley, NSF/NCAR/SCD 1 Research Data Stewardship and Access Steven Worley, CISL/SCD Cyberinfrastructure meeting with Priscilla Nelson.
An Examination of the Relation between Burn Severity and Forest Height Change in the Taylor Complex Fire using LIDAR data from ICESat/GLAS Andrew Maher.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
Mapping Greenland Using NASA’s Full- Waveform, Medium/High-Altitude, LVIS Lidar System: Potential 2009 Coverage and Expected Performance Michelle Hofton.
Feb 24-27, 2004ICDL 2004, New Dehli Improving Federated Service for Non-cooperating Digital Libraries R. Shi, K. Maly, M. Zubair Department of Computer.
Fire Emissions Network Sept. 4, 2002 A white paper for the development of a NSF Digital Government Program proposal Stefan Falke Washington University.
NSDL STEM Exchange: Technical Overview and Implications for Active Dissemination of Federally Funded Resources Across Implementation Systems.
Science Review Panel Meeting Biosphere 2, Tucson, AZ - January 4-5, 2011 Vegetation Phenology and Vegetation Index Products from Multiple Long Term Satellite.
MODIS Data at NSIDC MODIS Science Team Meeting - Nov. 2, 2006.
SPDF Science Advisory Group - September 29-30, 2005 Page 12/24/2016 9:09:48 PM Services of the Space Physics Data Facility (SPDF) / Sun-Earth Connection.
Satellite Aerosol Comparative Analysis using the Multi-Sensor MAPSS and AeroStat, powered by Giovanni Presented at the Goddard Annual Aerosol Update, at.
Distributed Archives Interoperability Cynthia Y. Cheung NASA Goddard Space Flight Center IAU 2000 Commission 5 Manchester, UK August 12, 2000.
March 2004 At A Glance The AutoFDS provides a web- based interface to acquire, generate, and distribute products, using the GMSEC Reference Architecture.
1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)
ECHO Technical Interchange Meeting 2013 Timothy Goff 1 Raytheon EED Program | ECHO Technical Interchange 2013.
Physical Oceanography Distributed Active Archive Center THUANG June 9-13, 20089th GHRSST-PP Science Team Meeting GHRSST GDAC and EOSDIS PO.DAAC.
The Virtual Observatory and Ecological Informatics System (VOEIS): Using RESTful architecture and an extensible data model to provide a unique data management.
U.S. Department of the Interior U.S. Geological Survey July 2014 OPeNDAP Services – Present and Future at LP DAAC Brian Davis 1, Rob Quenzer 1, Jason Werpy.
1 Design and Implementation of EarthScope Data Portal Chaitan Baru, Kai Lin San Diego Supercomputer Center.
ESDRs Distribution and User Support – Miscellaneous Topics
PADMA ALEKHYA V V L, SURAJ REDDY R, RAJASHEKAR G & JHA C S
INTAROS WP5 Data integration and management
DataNet Collaboration
NASA ROSES 2007: Decision Support through Earth Science Research Results Improving an Air Quality Decision Support System through the Integration of Satellite.
WSAQDW planning documents
Presentation transcript:

NASA ROSES ACCESS Project Review Science Data Systems for Satellite and Airborne LiDAR Data 23 February 2011 C. Meertens and J. McWhirter, UNAVCO S-J. S.Khalsa and T. Haran, NSIDC/CU C. Baru and C. Crosby, SDSC/UCSD D. Harding and B. Blair, NASA/GSFC and M. Hofton, U. Maryland ACCESS: Advancing Collaborative Connections for Earth System Science

Objective Approach Co-Is/Partners Key Milestones Advancing Collaborative Connections for Earth System Science ACCESS Science Data Systems for Satellite and Airborne LiDAR Data PI: Charles Meertens, UNAVCO TRL in =5TRL current =5 19 April 2010 Provide an integrated system (NLAS) for web services- based access to distributed NASA airborne (LVIS) and satellite (ICESAT/GLAS) LiDAR data archives using OpenTopography technologies. Define enhanced quality control (QC) metrics and a standard data format to improve evaluation of data quality, data usability, and compatibility with commercial LiDAR software packages. Provide users access to point cloud and waveform LiDAR data, on-demand digital elevation models (DEMs) generation, and derivative products and visualizations. Collaborate to define an industry-based LiDAR data format for NASA LiDAR data; to define enhanced QC metrics; and to design web services. Develop NSIDC ICESAT/GLAS web service. Develop UNAVCO/GSFC LVIS data repository system and web service. Provide integrated user access using OpenTopography Portal. Science partner testing of enhanced data and QC products and portal access. Portal provides user access to LiDAR point cloud, standard and on-demand digital elevation models generation. Complete data format, QC, and web services definitions. 12/10 Implementation of Data Web Services. 04/11 Beta release of NLAS/OpenTopography Portal with connectivity to NSIDC and UNAVCO. 10/11 Production release of NLAS Portal. 03/12 S-J. S.Khalsa, NSIDC/CU; C.Baru, SDSC/UCSD; D.Harding, B.Blair, NASA/GSFC; and M.Hofton, U.MD.

High Altitude Airborne Laser Vegetation Imaging System ( LVIS) ~500 thousand points/day 600 km altitude 70 m footprint Terrestrial LiDAR Scanning million points/day tripod 1 cm footprint Satellite - Icesat Low Altitude Airborne Laser Scanning 500 million-1 billion/day 0.5 km altitude 20 cm footprint LiDAR Data Platforms 5 million points/day 10 km altitude 10 m footprint Distributed and Large Collections from a diverse set of LiDAR Platforms

NLAS Project Overview Provide an integrated system (NLAS) for web services-based access to distributed NASA airborne (LVIS) and satellite (ICESAT/GLAS) LiDAR data archives using OpenTopography technologies. Define enhanced quality control (QC) metrics and a standard data format to improve evaluation of data quality, data usability, and compatibility with commercial LiDAR software packages. Provide users access to point cloud and waveform LiDAR data, on-demand digital elevation models (DEMs) generation, and derivative products and visualizations. Public Website (search for “NLAS Opentopography”) Hosted at OpenTopography Objectives of NLAS Project (NASA Laser Access System) Project Collaborative Wiki (uses Google sites nlaswiki)

Project Partners UNAVCO PI: Charles Meertens, Project management Software Engineer: Jeff McWhirter, Web services, API, and UI development San Diego Supercomputing Center, University of California, San Diego Co-PI: Chaitan Baru, Project management support Science Specialist: Chris Crosby, Algorithm Development, Science Evaluation Software Engineers: Vishu Nandigam, Minh Phan Web services, API, and UI development National Snow and Ice Data Center, NSIDC, CU Co-PI: Siri Jodha Singh Khalsa, Project management support, ESDSWG Software Engineers: Terry Haran, Nitin Khanna Web services, API, UI and Visualization development Goddard Space Flight Center Co-PI: David Harding and Collaborator: Bryan Blair Define data format, QA/QC, generate new products, science evaluation University of Maryland Collaborator: Michelle Hofton Define data format, generate new products, science evaluation

Major Accomplishments AGU NLAS presentation and NASA booth demo with end-to-end proof of concept for LVIS, ICESat GLAS, and TLS data Ingest of select LVIS and ICESAT/GLAS data sets into Open Topography portal Definition of initial new GLAS GLANL format and LVIS Comprehensive format NLAS/RAMADDA supports LVIS, LAS, GLAS and ASCII data formats as well as aggregations of those files Provisional support for new ICEBRIDGE data sets – ATM, MCORDS, PARIS OpenTopography federated web service query to NLAS/RAMADDA for remote data repository collection summary metadata using ATOM caster. Public OpenTopography website at (sample LVIS collections, LVIS point cloud, DEM service)

Major Milestones Complete data format, QC, and web services definitions. 12/10 Draft definitions Final definitions will evolve in stages over next two quarters Develop file conversion software for LVIS, GLAS 12/10 Note major step to use NLAS/RAMADDA to provide common data repository and richer web services for LVIS and GLAS Sample data sets ingested in OT; Presentations at 2010 AGU 04/11 Implementation of Data Web Services. 04/11 Prototype Catalog of services (connectivity between UNAVCO/SDSC OT) Full data service implementation at NSIDC, OT Beta release of NLAS/OpenTopography Portal with connectivity to NSIDC and UNAVCO. 10/11 Production release of NLAS Portal. 03/12

Project Schedule Summary Prototyping of NLAS workflow from data repositories to OT samples for LVIS and GLAS: slightly ahead of schedule Expedited by decision to utilize RAMADDA framework for data repositories and universal, flexible data translation web service Definition of data formats and enhanced QC/QA: on schedule Definition of full data web services API and implementation : behind schedule Processing of complete datasets into comprehensive formats: behind schedule Science partner utilization of NLAS services: slightly behind schedule Publicizing NLAS project: ahead of schedule

NLAS System Overview OpenTopography Currently: Provides users access to ten’s of billions of LiDAR point cloud measurements and in industry standard formats (e.g. ascii, LAS) Provides on-demand processing services to enable generation of digital elevation models (DEMs), derivative products and visualizations. NLAS will provide: Web Services access to distributed archives using a downloadable data repository package (NLAS/RAMADDA) Integrated access to NLAS data via OpenTopography An extensible Java package for reading and writing various LiDAR formats Add full waveform access and enhanced quality control (QC) metrics for ICESat/GLAS and LVIS USERS OpenTopography LVIS ICESat/GLAS NLAS RAMADDA

Open Topography OpenTopography Search Map Interface Metadata about Dataset Access to download and processing interface Public website OpenTopography is funded through the National Science Foundation

Open Topography – Data Access Data Downloading and Processing User specifies Download/Processing Parameters 1.) Select area from Map Interface 2.) Point Cloud - Select format (ASCII or LAS) 3.) DEM Generation -Select Gridding Method, parameters, and format

Open Topography - Results Page OpenTopography On-demand Processing Products: Subset of point cloud data Digital Elevation (DEM) Map Visualization product and.kmz file if desired DEM on browser embedded GoogleEarth LVIS example in Sierra Nevada

Open Topography - GLAS LiDAR Examples ICESat/GLAS example Subset area of interest (below left) Download ASCII Lat, Long, Elev (X,Y,Z) Example of subsequent import into Excel and plot of shot number vs elevation (below right) Beta Server at: OT query of NLAS/RAMADDA product catalog using web services with ATOM casting just completed (see beta server)

Open Topography - Other LiDAR Examples Terrestrial Laser Scanning example (NSF OT Development) Subset area of interest Download options include: ASCII Point Cloud, DEM, Hillshade & Slope Products, kmz, and image Note that UNAVCO will implement TLS into RAMADDA for NSF

NLAS Interoperability NLAS/RAMADDA Open TopographyVisualization/Analysis Clients NLAS Web Service HTML Interface NLAS Web Service Interoperability with Data Repository Web Service

NLAS – ICESat/GLAS GLANL/QCQA ICESat Consolidated and Enhanced Altimetry Data Structure Consolidation of parameters from 8 GLA product files Eliminating numerous redundant, invalid and unused parameters Producing a single data record for each laser shot Selection of subsets from full ICESat GLA product files Currently ~160 unique parameters in products NLAS/GLANL 100 parameters in “long” subset for advanced users NLAS/GLANL 18 parameters in “short” subset for casual users Definition of 25 additional derived parameters for improved ease of use e.g., Conversion of waveform-derived relative range offsets to elevations Definition of 6 QA/QC parameters to guide selection of appropriate data Source of return: Earth surface, cloud, noise, indeterminate Cause when no geolocation result reported: no return detected, instrument anomaly Non-nadir data acquired during spacecraft maneuver Possible presence of forward scattering range delay Qualitative range precision assessment from S/N and signal fitting residual Data reliability: composite yes/no flag for 8 parameters effecting quality

NLAS/RAMADDA RAMADDA Overview Data repository framework developed at Unidata/UCAR –Freely available Java based open source package –Extensible plugin architecture Basic content management functionality NLAS/RAMADDA plugin to support LiDAR data services LiDAR data ingest, metadata harvesting and publication Services include: –Map overview, time series and waveform displays –Spatial subset, decimation –Conversion to multiple data format: original binary, LAS, ASCII CSV, … –Product generation - Image, hill shade image, KMZ

NLAS/RAMADDA (LVIS or GLAS – same package) Open Topography Portal Web services Spatial queries Subsetting Data Conversion Product generation Ingest Metadata Database LVIS Product Generation New Comprehensive NLAS/LVIS Format (50 parameter) New Comprehensive NLAS/LVIS Format (50 parameter) Other Clients Collection Query Metadata harvesting LVIS Data System File store LiDAR Tools Data Request ATOM Response LAS, ASCII CSV, etc. NLAS Data Repository – LVIS Example

NLAS/RAMMADA Open Topography Portal Web services Spatial queries Subsetting Data Conversion Product generation Ingest Metadata Database GLAS Data Pool GLAS Data Pool New GLANL “long” 100 para “short” 18 para +25 additional para +6 QA/QC New GLANL “long” 100 para “short” 18 para +25 additional para +6 QA/QC Other Clients Collection Query Metadata harvesting ICESat/GLAS Data System File store LiDAR Tools Data Request ATOM Response LAS, ASCII CSV, etc. NLAS Data Repository – ICESat/GLAS Example

NLAS/LiDAR Tools Stand-alone Extensible Java package for reading and writing various LiDAR formats (made available via SourceForge) –LAS –LVIS (including waveform) –ICESAT/GLAS products –Other Icebridge formats – ATM, MCORDS, PARIS –Terrestrial Laser Scanning formats – text, PTX –Transforms global XYZ to Lat, Long, Elevation –…

LiDAR Subset Form Data Access direct from NLAS/RAMADDA (may run demo instead if time allows)

LiDAR Waveform Data Transmit Pulse Alternate Signal Start Outer Canopy Relief Canopy Inflection Standard Gaussian Fit Within Canopy Light Extinction Ground Relief Alternate Signal End Amplitude (volts) 1064 nm laser pulse footprint ~ 70 m diameter spaced 175 m along profile Full Waveform LiDAR Data

NLAS/RAMADDA Waveform Display NLAS/RAMADDA Waveform Display

Project Metrics Metrics Plan Custom and web log statistics from OpenTopography NLAS/RAMADDA Repository implementation includes web service API logging Web log analysis software will be used to parse and summarize information on service requests (e.g. Domain analysis such as IP address, unique visitors, volume of data transferred, number of points processed, jobs executed) UNAVCO will aggregate repository statistics and will be responsible for upload to NASA metrics system Additional impact and software reuse metrics separate from the web logs are planned OT has guest user (no login) and login access that allows increased processing limits and a custom workspace (mylidarjobs). NLAS will determine login and services access requirements and apply to RAMADDA and OT. NLAS will also need to pass through user id or IP through web service API and will bring this up with broader ESDSWG. Example 2008 LVIS Sierra Nevada, CA: Dec jobs 23,341,301 points Jan jobs 2,186,947 points Feb jobs 14,797,042 points

ESDSWG Participation Siri Jodha Khalsa is the representative to the ESDSWG Highlights: Standards Process Group (SPG) activities Participation in monthly teleconferences Service on two SPG Technical Working Groups evaluating proposed standards and reviewer on third. New Orleans meeting in October, 2010 (McWhirter also attended) Incorporated Atom-based Service Cast output format into the NLAS web service implementation. Feedback planned to ESDSWG: Federated distributed search/processing from NLAS Repositories Web service pass-through of user id/IP Khalsa is tracking developments of a proposed US national lidar data encoding standard and will provide feedback to ESDSWG.

Status and Next Steps Accomplishments LiDAR data repository framework NLAS/RAMADDA Data Repository LiDAR Data translation software tool End-to-end proof of concept –LVIS, ICESat GLAS, TLS New GLAS product –GLANL containing enhanced QC Some Next Steps Define NLAS web service API –Data and metadata discovery –Data service access Implement NLAS production servers for GLAS and LVIS data sets Implement full wave form service into OpenTopography Explore potential application of NLAS services to other data sets – ATM, TLS, etc. GSFC LVIS Antarctic Glacier Scan