Science Processing Status of the Earth Observing System Microwave Limb Sounder (EOS MLS) Elmain Martinez September 27, 2010 Aura Science Meeting - Data System Working Group Boulder, Colorado Jet Propulsion Laboratory California Institute of Technology Copyright 2010 California Institute of Technology. Government sponsorship acknowledged.

Topics 27 September 2010EOS MLS2 Recap MLS Interfaces & Dataflow Data Processing Status Software Status Processing Environment Improvements Summary Aura MLS

System Interfaces and Dataflow 27 September 2010EOS MLS3 US MLS Science Computing Facility (SCF) Data Management Layer UK MLS Science Computing Facility (SCF) Scientific Algorithm Development Scientific Analysis Science Processing SW Development Quality Control & Assessment All SIPS Inputs All Standard Products Diagnostics Logs PGEs Calibration Files Production Policies Science Investigator-led Processing System (SIPS) Scheduling/Planning Layer Data Management Layer Fetch Input Data Store Output Data Product Generation Executables (PGEs) EOSDIS Core System (ECS) Goddard Space Flight Center (GSFC) Earth Science (GES) Data and Information Services Center (DISC) SDP Toolkit Meteorological Data Delivered Algorithm Package Data Quality Assessment MLS L1B/2 PDRs DN, PAN, PDRD MLS L0 – PDS Aura SC Engr. Aura SC Definitive Orbit Aura SC Attitude Operational Meteorological Data Leap Second & UTC Pole Data

DATA PROCESSING STATUS 27 September 2010EOS MLS4

MLS Near-Real-Time Status SIPS NRT configuration is continuing to deliver Ozone and Temperature L2 products using V2.23 – Ingests pre-processed NRT MLS PDS data from the GES DISC 15 minute granules – L1 and L2 PGS software was modified in 2008 to improve product delivery timeliness – L2 products are delivered to SCF and GES-DAAC; available ~3 hours after observation Customers include GMAO, NOAA/NCEP, NOAA/NESDIS/STAR, Environment Canada, ECMWF, and Met Office 27 September 20105EOS MLS

Forward Processing Status SIPS is processing all incoming data with V2.23 – L1/2 products delivered to SCF and GES-DAAC V2.23 in operation since March 2008 – This is the 4th minor versions of V2 v02.20 – initially released Nov 2006 v02.21 – de-convolve DACS radiances in Level 1 v02.22 – increased performance without change of quality v02.23 – dealt with data variances produced by Aura anomaly – (data since Dec 4, 2008) We will transition from V2.23 to V3.3 once the Data Quality Document is released 27 September 20106EOS MLS

Reprocessing Status V3.10 reprocessing halted in Nov 2009 due to data quality concerns – Products delivered to GES DISC, but never publicly released; now removed SIPS is currently reprocessing all science data with V3.30 – Reprocessing began in April 2010 More than halfway through the 72 months (2273 days) to be reprocessed – V3.30 products being delivered to GES DISC; to be publicly released in Oct. 27 September 2010EOS MLS7 Days Processed

SOFTWARE STATUS 27 September 2010EOS MLS8

Major Improvements in V3.3 Software changes: – Processing with GEOS5 meteorology for dates after Sept 1, 2009 MERRA data for dates prior – Using HDF and SDP Toolkit – Products are augmented to be HDF5-NetCDF compliant – Software built with 64-bit compiler – Additional error handling added to scheduler – Products to be released to public once the Quality Document is finalized – See the poster “Version 3.30 Products from EOS MLS” for further details Science improvements – CH3CL (Methyl Chloride) is a new standard product See the Validation and Climatology poster on this species – Revised spectroscopy to reflect best-fit to unknown continuum – Special handling of Core+R5 phase if THz is turned off 27 September 2010EOS MLS9

Version 4 Plans Product Generation Software – Target release date is Spring 2012 – Improvements currently planned: Read various parameters using HDF5, e.g. temperature Retrieve species in clouds Flag radiances to not be used when clouds are detected Generating a new product, CH3OH – Intended to account for radiances seen in other species – Integrate latest available version of HDF5 and SDP Toolkit NRT improvements: – XML-formatted metadata – Browse product for choosing granules to download – Capability for Data Casting (RSS) – Better forward model such as 2 nd derivative 27 September 2010EOS MLS10

PROCESSING ENVIRONMENT IMPROVEMENTS 27 September 2010EOS MLS11

SIPS Processing Environment Improvements Replaced old IBM 32 bit production cluster with a Dell 64 bit cluster in June, 2010 – Specs 48 nodes 8 CPUs per node 146 GB storage per node 2.93 GHz 24 GB memory (vs 16GB on older clusters) Benchmark tests for a single day completed 3 hours faster compared to an older cluster; ~27% improvement in speed – About 6.34 additional days per week with the 27% performance improvement – With the additional cluster and its higher relative performance, weekly output has increased by apx. 22 additional days per week 27 September 2010EOS MLS12

SIPS HW/Network Architecture 27 September 2010EOS MLS13 sips-switch (LAN) OperationsDev/TestNRT DB Server sftp Server SIPS Server L1B (V2.23) L2 (V2.23) L2 (V3.30) sftp Server NRT Server (2) JPL router JPL net L2 (V3.30) Changes since 9-09: Operations: - Replaced aging L2 Ops cluster - Replaced aging L1 server Development - Replaced aging NFS/L1 server L1B (V2.23) DB Server File Server/ L1B (V3.30) Dev/Test (2) L2 (V3.30) L1B (3, V30) Replaced

SIPS Future Plans No upgrades planned for next FY FY12 and beyond we will be looking at: – Additional storage for V4 production – Replace aging servers – Replacing aging cluster 27 September 2010EOS MLS14

SCF Processing Environment Improvements Improvements in the last year: – Added 169 TB raw NetApp storage – Retired standalone NAS devices due to age. Moved data to the 60 TB NetApp – Retired aging test cluster – MLS webpage upgraded Future plans: – No upgrades planned for next FY but there never seems to be enough storage… – FY12 and beyond we will be looking at: Purchase additional storage for V4 processing Replace 32 bit cluster with 64 bit Replace aging NetApp filer 27 September 2010EOS MLS15

SCF HW/Network Architecture 27 September 2010EOS MLS16 JPL Network User Processing (4) Web Server NIS, (+backup) DB Server (+ backup) Collaboration Server SCF Internal Network Test Cluster Dev Cluster Network Storage 224 TB Touchscree n Lab & Flight Ops (5) Workstations (34) Data Retrieval IF Analysis Cluster Expanded

Science Community Support Mobile Phone data tracking of MLS daily plots – Accessible from MLS webpage Hurricane portal ( – Tropical Cyclone Information System (TCIS) brings together satellite and in situ data sets from various sources (e.g. MLS, OMI, QuikSCAT, and others) so users can find information for a particular tropical cyclone. Site up since April data loaded, other years to follow Site has an analysis tool that allows users to plot histograms, maps, and profiles for many different data sets and products. Uses MLS data products for temp, ozone, CO2, water vapor MLS webpage improvements, e.g. NRT Viewer – See the poster “The Microwave Limb Sounder Website: Improved Access to Data and Information” for more information 27 September 2010EOS MLS17

SUMMARY 27 September 2010EOS MLS18

Recap of Future Plans V4 is about 19 months away – Algorithm cycle is usually 18 months or more, but it could be sooner if Level 2 algorithms can mature SIPS hardware is stable and should be for a couple years SCF to make another leap to keep up…to purchase a 64 bit development machine and additional storage in the future 27 September EOS MLS