1. GOES-R Data Distribution Timothy J. Schmit NOAA/NESDIS – Office of Research and Applications (ORA) Advanced Satellite Products Team (ASPT), Madison, WI James Gurka and Roger Heymann (NOAA/NESDIS/OSD) and the whole GRB working group, etc May 2004 GOES User Conference Introduction GOES Re-Broadcast (GRB) Data compression Other options Summary

2. Introduction GOES Re-Broadcast (GRB) Data compression Other options Summary

3. What we know Data distribution is very important There are a large number of current GVAR reception sites, more sites are being added These are a combination of fixed and mobile sites. There is a wide range of current users needs GOES-R instrument data rates increase by approximately two orders of magnitude over current instruments Data compression can help to reduce data rates (and hence distribute more data) while preserving information

4. What we think we know Data (raw) downlink most likely will be in the X-band Data re-broadcast will most likely be in the L-band GOES Re-Broadcast (GRB) format will be different than todays GVAR There will be some form of satellite re-broadcast A tunable range of data compression options, depending on data, is ideal. Data compression techniques will continue to improve

5. What we do not know Data formats Data rate of re-broadcast data Contents of re-broadcast data Type or Amount of data compression Relationship between amount of data “pushed” versus “pulled” Etc.

6. Working Group (WG) assembled to examine the nature of GOES-R high rate data to be distributed Led by James Gurka (OSD) & Tim Schmit (ORA) Includes senior scientists, senior engineers, and policy officials James Gurka - WG Co-LeadNESDIS OSD Tim Schmit - WG Co-LeadNESDIS ORA Bill BrockmanShort & Associates Ken CareyMitretek Dennis ChestersNASA Monica CoakleyMIT Lincoln Lab Roger HeymannNESDIS OSD Eric MadsenNESDIS International Affairs John PereiraNESDIS OSD Thomas RenkevensNESDIS OSDPD Dick ReynoldsShort & Associates Steve ShortShort & Associates GOES Re-Broadcast (GRB)

7. While no final decisions have been made, several data distribution options are being investigated for the GOES-R era. - Various amounts of bandwidth rebroadcast via GOES - Various amounts of data rebroadcast via other than GOES (commercial) - Other methods (Internet, push/pull etc) - Combination, etc.

8. GRB Summary Pursue a robust GOES rebroadcast capability that balances national and international customer needs with the policy, technical and budgetary issues Consider appropriate data compression options to maximize GOES-R information availability and dissemination to NOAA's diverse customer base Consider options that allow maximum flexibility for users depending on their capability to acquire the data Recommend all GOES-R information be made available for all users, and work to try to accommodate customers with limited capabilities consistent with NOAA policy NOAA/NESDIS/OSD direction is to limit GOES-R high rate data (via GOES) to no more than 24 Mbps and look to smaller values Options for full data set distribution greatly exceed 24 Mbps (~110 Mbps)

9. GVAR SYSTEM The GVAR is a combination of (mostly) imager and sounder (scaled) radiances (no products). No data compression is used. “Downlink” (2.6 Mbps) “GVAR” (2.11 Mbps) Users Satellite Wallops CDAS Any GVAR site

10. GVAR GVAR database developed within OSD to support: Making decisions to whom how much data to broadcast data Support NESDIS RF spectrum coordination work with ITU to defend NOAA's keeping L-band to support GOES broadcast Direct GVAR users are located throughout the western hemisphere There are presently 155 known fixed land-based GVAR sites DoD also plans to have some 161 mobile GVAR sites Efforts are ongoing to identify more sites Current GOES 8-P high rate Data Broadcast is 2.1 Mbps More information on the GVAR system: http://www.osd.noaa.gov/gvar/gvardownload.htm

11. There are well over 100 known land-based GVAR sites. More receiving sites are expected to be identified. GVAR Sites

12. Future GOES Rebroadcast GRB system is extension of current GVAR system for GOES-R era –Serve users in Near RealTime throughout western hemisphere –Current GVAR mainly only rebroadcasts radiance data; GRB may/could rebroadcast both radiances and some products GRB system is payload service, separate from any direct service (LRIT, DCPI/R, EMWIN, and SAR) –GRB is needed to make a large amount of data available to a wide range of users (both geographically and in terms of data use) in a cost efficient manner Main difference between GVAR and GRB : –Due to new, large data rates for the instruments currently planned for GOES-R, the GRB system will not realistically be able to transmit all level 1b data without data compression

13. Possible GOES ReBroadcast System “Data Pool” Data compression can have applications in a number of areas: - downlink –rebroadcast - distribution - archive “Downlink” 2:1? “Re-broadcast” 6:1? Users Satellite Wallops CDAS/ Backup Any GRB site “Archive” 2:1 10+:1 “Data pool” of full GOES (the compete level 1b data) would not be compressed, nor rebroadcast. Full GOES would be available for ground-based transfers and archive.

14. General GRB Assumptions User needs –All users and applications not known –Forecasting all future users and applications uncertain –If the data is available, user will work to gain access to it Users (DoD, WMO nations, academia, etc) will expect a similar (or higher) level of service Communications capabilities will continue to evolve – bringing improved capability, technologies, and lower cost Future data compression techniques will continue to improve (both lossless and lossy) Send out as much information (as opposed to just data) to as many users as possible while balancing cost of dissemination with the goal of maximizing the usefulness of the information

15. How might we move from 100 to 24 Mbps for a rebroadcast ? Data compression is the key Maintain a "data pool" concept for land access and archive of uncompressed sensor data  Users could then "test what is not being sent“ for their own application(s) One method going from ~100 Mbps to 24 Mbps:  10X for the 0.5 km visible band  6X for the "1km” bands  2X for the IR bands of the ABI  6X for the HES-IR Means approximately half of the band width would be used by the imager and half by the sounder; each image could be sent out

16. Total (uncompressed) 43.5 Mbps Assuming 13-bit data and a 5-minute full disk scan mode 0.5 km visible data dominates the data rate of ABI

17. Total (uncompressed) 43.5 Mbps Total (compressed) 10.4 Mbps IR dominates the data rate of ABI after compression 10:16:12:1

18. Original MODIS 0.64 μm (0.5 km on ABI) Compression Ratio ~9:1

19. Original MODIS 0.86 μm (1.0 km on ABI) Compression Ratio ~6:1

20. Approximate comparisons for high spectral IR data Principal Component Wavelet Transform Processing SpeedSlowFast MemoryLargerSmaller LosslessYes, if residual are saved Possible Lossy20-50X5-6+X Method Reduces the redundancy by creating a new coordinate systems that points in the direction of decreasing variance A transformation that uses functions to exploit the structure of the data-newer field of math Other methods are also being explored.

21. Simulation study shows that reconstructed brightness temperature is nearly noise-free AIRS instrument noise Reconstructed minus noise-free Poster…Goldberg

22. Lossless Compression Comparison of Methodologies Without pre-processing, JPEG-LS gives the best compression ratio Poster…B. Huang

23. Very Fast Lossy Compression JPEG-LS RMS error A compression ratio of 5 is less than the instrument noise. A pre-processing step (right side graph) improves the performance. Poster…B. Huang

24. NOAA NESDIS has investigated Alternative Dissemination Methods (ADM) for distribution of weather/environmental data by means of Internet, Commercial Space Communications, or Dedicated Landline. The ADM methods of communication are separate from communication methods utilized in Direct Readout (DR), which is a broadcast from government satellites. From Marlin Perkins

25. A Broad Agency Announcement (BAA) for future geostationary satellite architecture study i s currently under-way. Task Area 3: Production Generation and Distribution, Archive and Access, and User Interface Segments http://www.osd.noaa.gov/goesr_arch_study/index.htm

26. Breakout Session Quesiton Question for each group during the break- out session on Thursday: “Please discuss your needs for [realtime] data and product distribution, archiving and access. For example, timeliness, registration, etc. ”

27. end