1. 1 HYCOM Data Service www.hycom.org/dataserver HYCOM Data Service www.hycom.org/dataserver An overview Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI)

2. 2  Data Service was launched in 2002 in Miami and served HYCOM Outputs via FTP, LAS and OPeNDAP servers  Started Serving Near Real Time Atlantic data in 2003  Hosted about 3 TB of Data until Dec, 2006 in Miami.  Jan -2007 – service moved to FSU and hosted on a 100 TB SAN  Currently serving approximately 60 TB of data HYCOM Data Service: History and Current Status

3. 3 New Hardware Software Software Red Hat Linux Red Hat Linux Red Hat cluster suite and Global File System Red Hat cluster suite and Global File System Apache Web Server Apache Web Server THREDDS THREDDS LAS Server LAS Server Vsftpd server Vsftpd server  100 TB Fibre Channel SAN  Three 8 CPU machines  32 GB RAM/machine

4. 4 Operational Enhancements All components (LAS,OPENDAP,FTP) co-located in the same physical machine. All components (LAS,OPENDAP,FTP) co-located in the same physical machine. Subset of files are uncompressed for faster access Subset of files are uncompressed for faster access In memory, decompression by using RAM scratch disks In memory, decompression by using RAM scratch disks High-availability architecture – minimal downtime during updates High-availability architecture – minimal downtime during updates

5. 5 Currently Available Datasets Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 – current) Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 – current) Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available shortly) Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available shortly) Four years of Global Non-Assimilative Simulation (2003-2007) Four years of Global Non-Assimilative Simulation (2003-2007) Near real-time 1/12 Atlantic Ocean prediction system output (June 2003 – Present) Near real-time 1/12 Atlantic Ocean prediction system output (June 2003 – Present) Monthly mean 1/12 Pacific Ocean simulation output (1978-2003) Monthly mean 1/12 Pacific Ocean simulation output (1978-2003)

6. 6 Global HYCOM Data Serving Challenges Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts) Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts) Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization on large files resource intensive and time consuming Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization on large files resource intensive and time consuming Need to handle model products in non- standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO formats) Need to handle model products in non- standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO formats)

7. 7 HYCOM Ocean Prediction System Data Flow 50 G pulled to FSU ~ 2-3 hrs Conversion to NetCDF and re-gridding to Z levels THREDDS and LAS Update Global run at NAVO daily – 5 day hindcast and 5 day forecast in curvilinear and hybrid coordinates *Approximately 4-5 hrs of data processing daily

8. 8 Means of Data Access THREDDS THREDDS Provides a dataset catalog and OPeNDAP access to all datasets. Provides a dataset catalog and OPeNDAP access to all datasets. Live Access Server Live Access Server V isualization and downloads in different formats, batch access etc. V isualization and downloads in different formats, batch access etc. OPeNDAP enabled clients OPeNDAP enabled clients Choice of common applications or user written programs to access data Choice of common applications or user written programs to access data FTP FTP NetCDF files for each variable NetCDF files for each variable

9. 9 Provides NetCDF data in both native HYCOM (staggered) and regular (non-staggered) grids Provides NetCDF data in both native HYCOM (staggered) and regular (non-staggered) grids On-the-fly conversion to fixed Z levels (but limited to pre-configured Z levels) On-the-fly conversion to fixed Z levels (but limited to pre-configured Z levels) tested HYCOM format IO as a product –but needs better integration tested HYCOM format IO as a product –but needs better integration Full resolution global outputs not available currently via LAS Full resolution global outputs not available currently via LAS HYCOM LAS : http://hycom.coaps.fsu.edu/las/

10. 10 Access to native coordinates and regridded fields

11. 11 Hybrid-Z regridding native-Z fixed-Z

12. 12 HYDAE model intercomparison

13. 13

14. 14 Synthetic Data Used in the Experiments

15. 15 Truth and Forecasts on Day 50

16. 16 RMS Error in 1-Day Forecasts

17. 17 SSH 1-Day Forecast Error Distribution

18. 18  The Matlab GUI developed for the 1/12 o North Atlantic Model Runs has been significantly modified.  Access the GUI from: http://opendap.org/download/ml-toolbox.html  The OPeNDAP-IDL client is now robust and can be used to access HYCOM data (no GUI yet).  Many modifications to the netCDF-OPeNDAP libraries.  Server4 - supporting GridFTP is now available. HYCOM specific Matlab tools from the OPeNDAP Developers

19. 19 The HYCOM Matlab GUI

20. 20 Access the OPeNDAP Matlab GUIs from the OPeNDAP download webpage http://opendap.org/download Or go directly to the GUI web page http://opendap.org/download/m l-toolbox.html http://opendap.org/download/m l-toolbox.html

21. 21 FTP – Access FTP://www.hycom.org/datasets

22. 22 User Community HYCOM Partners HYCOM Partners U. Miami Collaborators U. Miami Collaborators Breck Owens, Woods Hole Breck Owens, Woods Hole Joao Ferreira, Portugal Joao Ferreira, Portugal N. Idrissi, UVI N. Idrissi, UVI Erik van Sebille, S. Africa Erik van Sebille, S. Africa TNC TNC WRI WRI Meteorem, France Meteorem, France Global Marine Global Marine 1.5 TB data accessed since Feb 2007

23. 23 Access Details for June 2007

24. 24 Community Collaborations Working with John Caron at Unidata on testing the Forecast Aggregation server with large operational datasets Working with John Caron at Unidata on testing the Forecast Aggregation server with large operational datasets Building a HYCOM.[ab] format IO service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 library Building a HYCOM.[ab] format IO service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 library Partnership with IOOS data management Caucus Partnership with IOOS data management Caucus

25. 25 Other Addition/Improvements Forcing data Forcing data Reference observational data sets Reference observational data sets A completely redesigned website is being planned along the lines of GODAE server A completely redesigned website is being planned along the lines of GODAE server Searchable mailing list, FAQ’s Searchable mailing list, FAQ’s Links to other data repositories Links to other data repositories