1. Multi-tiered Peer-to-Peer Data Distribution for Spatially Enabled Data Nazila Merati OAR/PMEL Ann Matarese NMFS/AFSC Jason Marshall NOS/CSC

2. November 2, 2005NOAATech 2006 Silver Spring, MD Overview Data transfers are integral to NOAA’s mission In some cases, data are still transferred using CDs/DVDs and via “Sneakerweb” due to firewalls and security concerns Inter-disciplinary projects, which rely on data from other organizations for developing models and scientific research, can be delayed by slow data distribution

3. November 2, 2005NOAATech 2006 Silver Spring, MD How do we get data to collaborators faster? Peer to peer data distribution has been used by the physical science community to distribute model results and large scale datasets to collaborators. Works well in a academic setting with large servers and different security requirements Overnet and Gnutella as examples of file sharing systems

4. November 2, 2005NOAATech 2006 Silver Spring, MD Challenges we face Security concerns – getting data and keeping data secure Bandwidth – file sharing of large files can slow down other traffic Trusting peers and data sources – whose data are most correct and up to date? Data integrity – how can transfers keep data uncorrupted?

5. November 2, 2005NOAATech 2006 Silver Spring, MD Solutions Using COTS software that will allow us to transfer data in its native format – ArcPublisher/ArcReader Using a online collaborative tool to view maps/review and whiteboard solutions – Toucan and GrooveSite Testing a secure peer to peer network – BestPeer for GIS data transfers/JXTA Building a central repository of data sources and creating a trusted peer network within NOAA for data distribution using Webservices

6. November 2, 2005NOAATech 2006 Silver Spring, MD Test Case – Alaska Fisheries Science Center AFSC Recruitment Processes group has developed an Atlas of Fisheries Abundance for major icthyoplankton groups in the North Pacific. Data are stored in a GIS, but currently only a hard copy Atlas is available.

7. November 2, 2005NOAATech 2006 Silver Spring, MD Physical Data AFSC needs access to PMEL/FOCI data Data currently stored in netCDF, but conversion tools exist to convert data to shapefiles Cruises of data can be delivered via ArcPublisher

8. November 2, 2005NOAATech 2006 Silver Spring, MD Data delivery – who are peers? Data (occurrence and size distribution) need to be distributed to collaborators within AFSC working on ecosystem assessments as well as to collaborators at PMEL in the FOCI program and vice versa. Data are also needed by outside collaborators at the UW Fish Collection.

9. November 2, 2005NOAATech 2006 Silver Spring, MD Constraints Data are to be distributed with a time lag and to very specific groups Physical data (CTD, MBT and XBT) taken at the same time are also needed for recruitment studies All groups need base layers – bathymetry, shoreline, graticules, isobaths

10. November 2, 2005NOAATech 2006 Silver Spring, MD First Technique tried ArcPublisher for ArcGIS 9.1 was used to create maps of icthyoplankton abundance and distribution for peers within and outside the Recruitment Processes group network and sent via email Maps can be customized with a corporate look and feel (NOAA logo, disclaimers, information and links) Maps can have specific access-control attributes

11. November 2, 2005NOAATech 2006 Silver Spring, MD ArcPublisher/ArcReader for data transfer Maps are viewed using ArcReader, a stand-alone free client available from ESRI or can be added to existing ArcGIS application Data can be extracted and edited or just viewed depending on the level of trust within the network

12. November 2, 2005NOAATech 2006 Silver Spring, MD ArcPublisher Map authored in ArcGIS including layout Publisher will allow map creator to determine levels of access control in Publisher options Identical look and feel to ArcGIS, so integrates seamlessly with existing projects Easy to author and package

13. November 2, 2005NOAATech 2006 Silver Spring, MD ArcPublisher Security and Control Password Protection to open map in ArcGIS or an IMS application Data Packaging and map/data timeouts are options – good for data that need to be refreshed Raster, vector data can be distributed in map document

14. November 2, 2005NOAATech 2006 Silver Spring, MD More enhancements in ArcPublisher 9.2 Publish Geodatabases of data Connections to ArcWeb Services More service oriented architecture, integrating with ArcGIS 9.2

15. November 2, 2005NOAATech 2006 Silver Spring, MD GeoCollaboration on the Web Many on line project management sites are used by cross-line office projects (Grovesite, Groovesite, Basecamp) all work to help organize large teams with geographically separated collaborators to work more efficiently Many tools can be extended by third party extensions to work with mapping applications

16. November 2, 2005NOAATech 2006 Silver Spring, MD Geocollaboration can be used in a secure networking environment Data and documents are placed in secure areas for teams to review and comment on using messaging areas or white boards. In some cases, versioning of data is handled. Can be crucial with regularly updated maps and figures. Can link to ArcIMS or Webmapping services to provide static updates to site specific projects Downside – some PM tools may still be platform dependent May be built, may not be used by PIs and managers

17. November 2, 2005NOAATech 2006 Silver Spring, MD Groovesite and Toucan Navigate Groovesite is used by groups large and small for project collaboration. Developers have added GIS functionality to Groovesite to allow real time on line collaboration in a map room. Uses Toucan systems’ own GIS, but can edit and view shapefiles and MapInfo Format files

18. November 2, 2005NOAATech 2006 Silver Spring, MD True Peer to peer solutions – build your own and they will use it Programmer code exists to build your own peer to peer network that has reliable security and will work with large scale data sets JXTA (Sun Microsystems) has large code repository and deals with many of the security issues faced by NOAA Interoperable and platform independent Successful in distributing large volume data sets – used in many universities for parallel and grid computing Requires programmer time and strong support from NOAA IT security officers

19. November 2, 2005NOAATech 2006 Silver Spring, MD BestPeer Architecture BestPeer application is written specifically for GIS file types, uses UDDI and LIGLO for data discovery Secure security model and global identities to keep peers uniquely identified Maybe tested at AFSC, used in University communities AFSC is more one to many case

20. November 2, 2005NOAATech 2006 Silver Spring, MD Long term options for data distribution Building a distributed data management system within NOAA that would allow data sharing of base data layers so that we could –Reduce duplication of effort and data storage of most used data sets –Create a practice and protocol for peer recognition (using webservices and service discovery using UDDI)

21. November 2, 2005NOAATech 2006 Silver Spring, MD Potential Applications Project Management on a NOAA wide level using geocollaboration framework Disseminating data and maps to first responders – via the web and hard copies with the best available data – coastal storms, oil spills, tsunami response in field conditions Distributed data and sharing of geospatial data within NOAA – avoiding duplication of effort by using a trusted peer network Integrated Decisions Support Systems – collaboration tools in emergency planning and mitigation

22. November 2, 2005NOAATech 2006 Silver Spring, MD Next Steps – Peer to peer collaboration Investigate the use of Groovesite and Toucan for collaborative map making Test larger data sets in ArcPublisher to see how file transfer works, install new release and test extended functionality Test a open-source peer to peer data network using JXTA framework to test security models and data delivery with PMEL, CSC and AFSC