Page 1 Ocean PEATE Fred Patt. Page 2 Ocean PEATE Agenda Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule.

Ocean PEATE Fred Patt

Ocean PEATE Agenda Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule Documentation Issues Land SD3E PSOE Ocean Atmosphere Ozone Sounder I&TSE NICSE

NPP/VIIRS Ocean Team Chuck McClain – Ocean Color PI Peter Minnett (U. Miami) – SST PI Gene Feldman – Ocean PEATE Manager VIIRS Ocean Science Team –Wayne Esaias –Barney Balch (Bigelow Lab) –Mike Behrenfeld (Oregon State U.) –Janet Campbell (U. New Hampshire) –Bob Evans (U. Miami) –Stephane Maritorena (UCSB) –Norm Nelson (UCSB) –Dave Siegel (UCSB) –Ken Voss (U. Miami) –Menghua Wang (NOAA/NESDIS)

Ocean Team (cont.) VIIRS Instrument and Calibration Support –Kevin Turpie –Bob Barnes –Gene Eplee –Gerhard Meister Validation Support –Sean Bailey –Jeremy Werdell Software Support –Bryan Franz –Joel Gales Data System –John Wilding Systems Management –Paul Smith

NASA/GSFC Ocean Biology Processing Group (OBPG) Projects Sea-viewing Wide Field-of-view Sensor (SeaWiFS) / Orbview-2 – active Moderate-resolution Imaging Spectroradiometer (MODIS) / Terra and Aqua – active Coastal Zone Color Scanner (CZCS) / Nimbus-7 – heritage Ocean Color and Temperature Scanner (OCTS) / ADEOS-I – heritage Glory data system prototype – 2009 launch Aquarius / SAC-D – May 2010 launch VIIRS / NPP – September 2009 launch

Ocean PEATE Agenda Land SD3E PSOE Ocean Atmosphere Ozone Sounder I&TSE NICSE Science Team Introductions Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule Documentation Issues

Ocean PEATE Design ● The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS) ● This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.

Element Overview Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements Provide suggested algorithm improvements to the IDPS via the Project Science Office Element (PSOE) Process selected data subsets in support of Evaluation and Validation activities

Changes since PDR Established interface with RSMAS (U. Miami) for SST Validation – using MODIS validation for proof of concept. Expanded and elaborated on xDR evaluation methodologies (as presented at July Peer Review).

Ocean PEATE Interface Diagram PSOESD3EI&TSENICSE VOSTCLASS (ADS) Ancillary Data Providers Ocean Science Community Casa- NOSA xDRs, IPs, Ancillary Data Alternate Ancillary Data Management Direction Calibration Updates and Evaluations Interaction xDR Eval. Results, Algorithm Updates Pre-flight Algorithms, Data, Info Software, Data In Situ Data Ocean PEATE Algorithm Updates, Test Requests & Results xDRs, IPs, Ancillary Data (if unavailable from SD3E) Analysis Results, Proposed Algorithm Updates RSMASSeaBASS In Situ Data Matchups

Ocean PEATE External Interfaces (1 of 2) SDS Science Data Distribution and Depository Element (SD3E) –Provides NRT access to raw data –Primary source of RDRs –Provides selected SDRs and EDRs SDS Integration and Test System Element (I&TSE) –Build and test updates to operational code in mini-IDPS –Run tests on selected data per request of PEATE Archive Distribution Segment (ADS) –Primary source for archived data xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products Ancillary Data Providers (ADP) –Provides alternate ancillary data sets (e.g., ozone, meteorological data sets) CasaNOSA –Serves as the NPP pre-flight repository of Government held data for distribution to Government user teams –Place to acquire pre-launch NPP algorithms and supported data files

Ocean PEATE External Interfaces (2 of 2) NASA VIIRS Ocean Science Team (VOST) –Coordinate activities with PEATEs and PSOE on xDR and recommended algorithm improvements. Supports Independent Calibration Validation Activities NPP Instrument Calibration Support Element (NICSE) –Provides alternative calibration LUTs and recommended improvements to calibration algorithms –PEATE provides results of LUT and algorithm tests Project Science Office Element (PSOE) –Provides management direction –Accepts algorithm update recommendations SeaBASS/ODPS –Provides Ocean Color in situ data RSMAS/U. Miami –Provides SST in situ locations –PEATE provides SST EDR matchups Ocean Science Community –Relies on Ocean PEATE to provide evaluation products and results

Assumptions – Role of I&TSE The Ocean PEATE will rely upon the SDS-provided I&TSE (mini- IDPS) to serve as the testbed for algorithm evaluation and potential improvements and will not attempt to duplicate efforts by running the operational code within the PEATE environment. The staff of the I&TSE will have the ability to modify any operational code as per Ocean PEATE specification and run the code within the operational environment against a set of PEATE-specified test data sets. The I&TSE will receive and install the most current version of the code and executable programs for all operational IDPS software required to produce NASA-evaluated EDRs, starting from RDRs. The I&TSE will maintain all processing code under configuration control. The PEATEs will have access to the configuration controlled code.

Ocean PEATE Support for IDPS Software The Ocean PEATE will maintain a working knowledge of the VIIRS SDR code and the EDR code for the Ocean Color and SST EDRs. The Ocean PEATE will maintain an understanding of the relevant IP characteristics, but will (initially) assume that the IPs are properly generated. The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution. The Ocean PEATE will develop appropriate test cases and request runs to verify and evaluate the changes. The Ocean PEATE will provide recommended changes to the PSOE, including a description of the proposed change, effect on the EDR performance, and the evaluation of the test runs.

VIIRS RDR Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files DEM NDT Ancillary Files Previous VIIRS Gridded Products Processing Module VIIRS Product Dynamic Ancillary Data Static Ancillary Data IDPS VIIRS Ocean EDR Data Flow

IDPS VIIRS Ocean EDR Data Flow VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.IM 375m SDR VIIRS_SDR.MOD 750m SDR VIIRS RDR Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files DEM NDT Ancillary Files Previous VIIRS Gridded Products VIIRS SDR VIIRS SDR VIIRS Geolocation Processing Module VIIRS Product Dynamic Ancillary Data Static Ancillary Data

IDPS VIIRS Ocean EDR Data Flow VIIRS_GD_28 Surface Pressure Adjustment VIIRS_GD_08 750m Granulation VIIRS_GD_25 NAAPS Granulation VIIRS_GD_27 L/W Mask Granulation VIIRS_GD_09 GFS Granulation VIIRS_GD_12 Bathymetry Granulation VIIRS_GD_13 Temperature Granulation ALL_GD_01 Time Interpolation VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.IM 375m SDR VIIRS_SDR.MOD 750m SDR VIIRS RDR Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files DEM NDT Ancillary Files Previous VIIRS Gridded Products VIIRS_GD_11 Ancillary Profile VIIRS SDR VIIRS SDR VIIRS Geolocation Processing Module VIIRS Product Dynamic Ancillary Data Static Ancillary Data

IDPS VIIRS Ocean EDR Data Flow VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask VIIRS_GD_11 Ancillary Profile VIIRS_GD_28 Surface Pressure Adjustment VIIRS_GD_08 750m Granulation VIIRS_GD_25 NAAPS Granulation VIIRS_GD_27 L/W Mask Granulation VIIRS_GD_09 GFS Granulation VIIRS_GD_12 Bathymetry Granulation VIIRS_GD_13 Temperature Granulation ALL_GD_01 Time Interpolation VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.IM 375m SDR VIIRS_SDR.MOD 750m SDR VIIRS RDR Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files DEM NDT Ancillary Files Previous VIIRS Gridded Products VIIRS SDR VIIRS SDR VIIRS Geolocation Processing Module VIIRS Product Dynamic Ancillary Data Static Ancillary Data

VIIRS_OC_01 Ocean Color / Chlorophyll VIIRS_ST_02 Surface Temp VIIRS_SN_03 Ice Concentration VIIRS_ST_01 Sea Surface Temperature VIIRS_AR_01 Aerosol Type VIIRS_SN_02 Ice Quality VIIRS_LN_06 Active Fires VIIRS_CL_01 Cloud Optical Properties VIIRS_CM_01 Cloud Mask VIIRS_GD_11 Ancillary Profile VIIRS_GD_28 Surface Pressure Adjustment VIIRS_GD_08 750m Granulation VIIRS_GD_25 NAAPS Granulation VIIRS_GD_27 L/W Mask Granulation VIIRS_GD_09 GFS Granulation VIIRS_GD_12 Bathymetry Granulation VIIRS_GD_13 Temperature Granulation ALL_GD_01 Time Interpolation VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.IM 375m SDR VIIRS_SDR.MOD 750m SDR SST EDR OCC EDR VIIRS RDR Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files DEM NDT Ancillary Files Previous VIIRS Gridded Products Processing Module VIIRS Product Dynamic Ancillary Data Static Ancillary Data VIIRS SDR VIIRS SDR VIIRS Geolocation IDPS VIIRS Ocean EDR Data Flow

ODPS MODIS Ocean Product Data Flow Processing Module MODIS Product Dynamic Ancillary Data Static Ancillary Data MOD_PR01 Level-0 to 1A MOD_PR03 Geolocation MSl12 Level-1B to 2 MOD_PR02 Level-1A to 1B MODIS Level-0 Land/Water Mask Platform ATTEPH Data Ozone Ancillary Files MET Ancillary Data MODIS 1 km Level-1B MODIS Geolocation MODIS Level-1A MODIS SST MODIS Ocean Color

Evaluation vs. Product Level Level-1 (SDR) Evaluations –Onboard calibration analyses –Vicarious calibration Level-2 (EDR) Evaluations –Matchup analyses –Residual detector (striping) and scan (RVS) dependence Level-3 Product Evaluations –Sensor cross-comparisons –Algorithm comparisons –Temporal anomaly evaluations

SDR Example – Vicarious Calibration

EDR Example – Scan and Detector Dependence

Level-3 Example – Zonal Cross-Comparisons

ODPS Design Overview Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data Highly scalable Easily adaptable to support multiple concurrent missions Graphical user interfaces for controlling and monitoring system functions and activity Non-platform specific

ODPS Design Philosophy Building-Block approach Programs are usually small and do one thing well Programs are less complex and subsequently easy to maintain Promotes reuse Programs loosely coupled so testing and production can be done in the same environment Adopt basic standards ANSI, POSIX, C9x Use existing technology when possible Exit statuses indicate successful or failure conditions

Components and Subsystems Device Manager RDBMS VDC/Scheduler Data Acquisition and Ingest Level 3 Scheduler Data Distribution File Migration and Management

Components and Subsystems (1 of 2) RDBMS is the primary element that manages all system activity Generic core databases support system infrastructure and non-mission-specific functions Mission databases catalogue products and house mission- specific data and procedures High level of reuse possible for similar missions; e.g., MODIS Aqua/Terra, SeaWiFS, CZCS, and OCTS are all ocean-color missions and have similar product suites and requirements

Components and Subsystems (2 of 2) Relational Database Management System (RDBMS) supports all of the system components (subsystems) VDC/Scheduler is the primary controlling module within the system Other subsystems are independent modules, yet rely on the VDC/Scheduler for some their functions Archive Device Manager (ADM) Data acquisition and ingest Level-3 Scheduler File migration and management Data distribution

ODPS COTS and Freeware Linux OS (CentOS 4.x) Solaris OS Sybase RDBMS Subversion (source code management) Pro-active DBA Interactive Data Language (IDL) Generic Mapping Tool (GMT) Netpbm (graphic image toolkit) HDF5 Library Languages: C, PERL, SQL

ODPS Architecture: Hardware Processing Servers Intel-based dual Xeon / AMD-based dual Opteron 8 GB RAM Five 72 GB SCSI drives Storage Servers Intel-based P4 / AMD-based single Opteron 1 GB / 2 GB RAM 1.5 TB IDE RAID 5 (3ware) / 9.6 TB SATA RAID 6 (Areca) 2 hot spare drives per RAID5 Database Server Sun V880 8-16 GB RAM 6-12 70 GB SCSI HDD

ODPS Current Components Processing Cluster 34 processing nodes 1.5 TB Ingest Servers 2 SeaSpace ground stations 5 storage nodes 11.2 TB Distribution Servers (FTP) 1 processing node 6 storage nodes 25.5 TB Distribution Servers (web) 1 large 3 processing nodes 63 storage nodes 605 TB Testing Cluster 13 test nodes 2 TB Network Support Systems Database Server 1 large server 876 GB Backup Servers 1 large server 876 GB 2 storage nodes 11 TB Extreme Networks Black Diamond 6816 Gigabit Ethernet switch Development Servers 1 processing node 2 storage nodes 2.4 TB User Desktops Cal/Val & QC Systems Mission Operations Systems

Building 28 Room W220 Computing Facility

Reliability and Redundancy Critical components (database server, network systems) have full maintenance contracts to ensure rapid response to problems Multiple-server components (ingest, processing, storage, distribution) have substantial redundancy to maintain full capability; spares maintained for rapid replacement. Testing nodes are separated from mainstream production components.

Technology Refresh Hardware technology advances (CPU, storage) are continuously monitored to select new components for evaluation. –Typical upgrade threshold is a doubling in capacity (~18 months). –Two generations of hardware are generally in use. Candidate components are procured, installed in testing cluster and rigorously evaluated in a production-like environment. Following successful evaluation, multiple copies are procured, installed, tested and swapped in for older components. –ODPS design allows new components to be rapidly added to resource tables without interrupting system operations. Performance of new components is closely evaluated following installation in operations environment. Critical components are run in parallel with existing system to ensure reliability under production loading.

Ocean PEATE Gap Analysis (1 of 2) Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest, Device Manager and File Migration and Management). –Status: Development of acquisition and ingest scripts underway based on sample products in SD3E. Process selected Ocean EDRs (SST and OCC) to Level-3 to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner). –Status: Prototype Level-3 processing has been demonstrated using sample IDPS Build 1.4 OCC and SST EDRs. Perform VIIRS OCC EDR matchups with SeaBASS Ocean Color in situ data (extract code). –Status: Pending completion of acquisition and ingest capabilities.

Ocean PEATE Gap Analysis (2 of 2) Incorporate VIIRS SDR processing for vicarious calibration analysis. (*) –Status: Pending availability of sample SDRs in Build 1.5 format Produce VIIRS proxy data using VOST-developed software (VDC/Scheduler). –Status: Prototype geolocation and EDR simulation developed to test Level-3 binning of EDRs. Acquire SST in situ data from RSMAS and perform matchups with SST EDRs (*) –Status: Pending final MOU Support browse and distribution of data products for team members (Data Distribution). –Status: Pending completion of acquisition and ingest capabilities. (*) New PEATE capability identified since PDR

Prototype Level-3 Processing EDR to Level-3 processing prototype completed using IDPS Build 1.4 EDRs.

Existing Software Reuse ODPS Components –Database –VDC/Scheduler –Data Acquisition and Ingest –Level-3 Scheduler –File migration and management –Archive Device Manager –Data distribution Level-2 multi-mission software (vicarious calibration) Level-3 multi-mission software (long-term trends and comparisons Level-2 to Level-3 comparison software (residual sensor errors) SeaBASS (in situ data management) Matchup/extraction software

Basis of Estimate (1 of 2) Acquire, ingest and catalog NPP VIIRS data products: –200 LOC (combined UNIX shell, SQL, C) per new product, for each of the four VIIRS products Process Ocean EDRs (SST and OCC) to Level-3: –300 C LOC to add VIIRS Ocean EDR input to existing binning software (L2bin) –300 SQL LOC per temporal range (10 ranges total) –200 shell LOC for all ranges Perform VIIRS EDR matchups: –100 C LOC + 10 PERL LOC to add VIIRS Ocean EDR input to existing extraction software

Basis of Estimate (2 of 2) Process SDRs for vicarious calibration analysis: –1000 C LOC to add VIIRS SDR input to existing Level-2 processing software (MSl12) Produce VIIRS proxy data: –100 shell LOC for each processing stage Support browse and distribution of data products : –3000 C LOC to implement browse image generation –100 PERL LOC to add VIIRS products to existing browse and order web site

Ocean PEATE Data Storage Estimate Data TypeDaily 1 Year5 Years RDR150 GB53.5 TB268 TB SDR (M-band)242 GB (2) 8.6 TB (1,2) 43 TB (1,2) OCC EDR84 GB3 TB (1) 15 TB (1) SST EDR19 GB0.7 TB (1) 3.4 TB (1) Inter. Products~70 GBN/A Ancillary Data0.1 GB.04 TB.2 TB Total565 GB66 TB330 TB Assumptions: (1)Long-term storage is sized for 100% of RDRs and 10% of SDRs and EDRs (2)SDR volume includes geolocation

New Hardware for the Ocean PEATE 7 Storage Servers @ 9.6 TB – first-year VIIRS data storage Additional servers acquired post-launch to handle years 2 – 5 No new processing or network capacity required; technology refresh cycle to be continued within the ODPS as described.

Ocean PEATE Build Schedule Build 1 (L-18 months) All interfaces fully implemented and tested Verify initial versions of operational code ported and running in I&TSE L-3 product code developed and tested Prelaunch VIIRS test data storage and SDS interface testing support with existing ODPS storage capacity Initial test products generated for review by VIIRS Ocean Science Team Build 2 (L-12 months) Routine exercise of interfaces to acquire proxy, surrogate (Aqua?) and/or simulated data Verify pre-launch version of operational code running in I&TSE Browse and distribution capability developed and tested Test products routinely acquired as available and posted for access by VIIRS Ocean Science Team Data storage for one year

Ocean PEATE Development Schedule

External Schedule Dates C3S –SDS Integration –October 2008 IDPS – SDS Integration –October 2008 NSIPS – SDS Integration –October 2008 Functional Thread Test Dates –FTT8 A –Ingesting xDRs – October 2008 –FTT8 B –Validating xDRs – October 2008 NPP Compatibility Tests –NCT2C - December 2007 –NCT3 - September 2008

Ocean PEATE Testing ODPS Testing Philosophy : –Test as you run. –Test early, test often. Phased approach to testing: get one thing working and move on to the next step until entire end-to-end stream is operational. Testing VIIRS support to be done within operational ODPS environment, with products clearly identified for separation from operational product stream. Testing stages: –Interfaces –Internal functionality (processing, evaluation) –End-to-end Test data sources: –VIIRS proxy data –VOST simulation –MODIS products (e.g., initial SST validation) Availability of “official” test data is an issue.

Ocean PEATE Requirements Implementation 95% of requirements (19) implemented by Build 2; add additional storage capacity before launch

Sustaining Operations and Maintenance ODPS runs 24x7 with on-site support 8x5. Support is shared across all projects. Staffing needs are covered by existing OBPG support personnel.

Ocean PEATE Staffing Levels All staffing needs will be met with existing OBPG staff Increases will be accommodated as other projects (e.g., MODIS Aqua) wind down.

Documentation Ocean PEATE Level 4 Requirements and Operations Concept ICDs –SD3D to PEATEs –CLASS? MOU with RSMAS (TBS) ODPS Project Data Management Plan (draft) OCDPS Risk Management Plan Network IT Security Plan

Historical Milestones PEATE selected January 2004 PEATE Peer Review November 30, 2006 SDS PDR September 19, 2006 EDR Evaluation Peer Review July 18, 2007

Issues/Challenges/Concerns Limits of available test data –Ability to adequately test and flow data across interfaces –Verification of EDR evaluation processes Extent of prelaunch end-to-end (observatory & ground system) data flows ADS to PEATE interface Complexity of IDPS internal processes and data flows Version synchronization of operational products with IDPS software in the I&TSE Ability to diagnose software and algorithm problems in the I&TSE Bandwidth of NSIPS/SD3E interface –Sole source of IPs including OBC products Separation of spacecraft diary from science RDRs –Requires tracking >4000 small granules/day

Conclusion Ocean PEATE will meet all requirements Next Up: Land PEATE – Ed Masuoka NPP_SDS_Land.ppt

Backup Slides

Ocean PEATE Level 4 Requirements Allocation (1 of 2) SDS Rqmt Number SDS Rqmt TitleSubsystem 3Ocean PEATEN/A 3.1Acquire RDRs, SDRs, and Ocean EDRsN/A 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.1.6 3.1.7 3.1.8 Acquire and Ingest VIIRS RDRs from SD3E Submit Requests to SD3E for Subsets of SDRs and EDRs Acquire and Ingest SDRs and EDRs from SD3E Submit Requests to ADS/Class for SDRs and EDRs Acquire and Ingest SDRs and EDRs from ADS/CLASS Provide Storage for RDRs, SDRs and EDRs Support Cataloging, Searching, Ordering and Distribution VDC/Scheduler Data Acquisition & Ingest File Migration and Management Device Manager Data Distribution PEATE Personnel 3.2Assess the Quality of the NPP ProductsN/A 3.2.1Assess the Quality of the Ocean EDRsN/A 3.2.1.1 3.2.1.2 3.2.1.3 3.2.1.4 3.2.1.5 3.2.1.6 3.2.1.7 Ground Truth Validation with In Situ Data Cross-comparisons with Concurrent Satellite Data Sets Comparisons with Climatological Data Sets Assessments of Internal Consistency Flagging and Masking Assessment Prelaunch Algorithm Assessment VDC/Scheduler Level 3 Scheduler Level 3 Binner SeaBASS Matchup/Extraction Software

Ocean PEATE Requirement Allocation (2 of 2) SDS Rqmt Number SDS Rqmt TitleSubsystem 3.2.2Support VOST in Assessing Quality of the VIIRS SDRsN/A 3.2.2.1 3.2.2.2 3.2.2.3 Assessment of long-term radiometric stability Assessment of instrumental corrections Analysis of prelaunch test results VDC/Scheduler Level 2 Processing Software PEATE Personnel 3.3Provide Suggested Algorithm Improvements 3.4Process Selected Data SubsetsN/A 3.4.1Interface with I&TSE for Processing DataPEATE Personnel 3.4.2Acquire Processing Code from I&TSEPEATE Personnel ODPS CM 3.4.3Acquire SDRs and EDRs from I&TSEVDC/Scheduler Data Acquisition & Ingest File Migration and Management Device Manager Data Distribution

ODPS Design Details

RDBMS Vendor Client LibraryVendor Library Module Database Services Layer C Interface FunctionsPerl DBI Module Perl Scripts C Programs Goal: Isolate RDBMS from system software To use a different RDBMS vendor, swap in a new Database Services Layer Components and Subsystems: RDBMS RDBMS

VDC/Scheduler C program with supporting database procedures Runs in a daemon-like state on every processing server Primary system element responsible for coordinating most of the system activity Monitors task records in a to-do list database table Runs tasks according to defined task attributes Standard job-shell interface allows new programs to be quickly adapted as tasks for Scheduler control Subsystems: VDC/Scheduler

VDC/Scheduler Highly scalable, distributed infrastructure for concurrent processing of serial streams (e.g., L0  L1A  L1B  L2) Suite of C programs with supporting database procedures Uses recipes to encapsulate data-specific processing schemes, parameters, and pre-processing rules Virtual Processing Units (VPUs) serve as distinct distributed processing resources VPUs dynamically allocated based on available time and current OS load Comprehensive, user-defined processing priorities Subsystems: VDC

VDC: Ancillary Data Stager Runs in a daemon-like state Monitors entries in the processing queue and runs the rule procedures that are associated with the rule scheme for each recipe Updates queue-entry status when all rule procedures return successfully Governed by currently configured processing priorities VDC: Pre-processing Rule Manager

VDC: MakeVDC Selects processing-queue entries that have passed pre-processing-rule requirements Generates custom VDC job files from templates according to configured priorities Runs as a Scheduler task, so it can easily be configured to run as often as needed to keep the VDC queue full VDC: MakeVDC

VDC: Engine Function performed by the VDC/Scheduler module Each instance actively competes for jobs that it is allowed to run, based on priority, length of time in the queue, and user “nudges” Monitors and manages processing resources Initializes processing streams Invokes recipe steps and monitors step-execution time Handles operator-requested stream actions VDC: Engine

Archive Device Manager User defines logical pools of storage devices Processes request a device in a specific pool DM returns information for a storage device in the requested pool If auto-cycling is enabled, the DM time-stamps the record for the selected device, so a different device within the pool will be selected for the next request When a device in a pool exceeds a user-configured usage threshold, the device is no longer eligible to be selected Disk-monitor process polls all devices periodically to record usage statistics and invoke threshold handlers Subsystems: Device Manager (DM)

Data Acquisition and Ingest (1 of 2) Data types and sources are described in the database Active, passive, and periodic notification methods Active method scans remote systems for new files and populates the ingest queue Passive method waits for arrival of e-mail messages describing type and location of new file and populates the ingest queue Periodic method schedules transfers of files at user-specified intervals Subsystems: Data Acquisition and Ingest

Data Acquisition and Ingest (2 of 2) File transfers handled by ingest daemons and Scheduler tasks FTP, RCP, SCP, WGET transfer protocols supported Generic script handles the file transfer and then hands off to data- specific post-ingest scripts Subsystems: Data Acquisition and Ingest

Level-3 Scheduler Responsible for scheduling processing for level-3 composite products Runs as a Scheduler task Configuration is database driven Mission-specific stored procedures are invoked to identify input files for a composite product Subsystems: Level-3 Scheduler

File Migration and Management Responsible for compressing files and migrating them to their various destinations Event- or time-based actions –Ex1: compress files after they are QC’d –Ex2: remove files that are more than N days old –Ex3: mirror files upon creation Queries associated with each action are run periodically by a Scheduler task to select files that are eligible for some type of migratory action and populate a migration queue Command-line queuing for file removal and delayed copies Migration daemons query the migration queue, perform registered actions on the files, and update catalog tables Subsystems: File Migration and Management

Data Distribution Interactive, web-based Data Ordering System, currently supporting SeaWiFS and MODIS Aqua Data Subscription System, currently supporting MODIS Aqua, allows users to define region and products of interest Order and Subscription Manager Daemons monitor the order and subscription queues and stage files on FTP servers (stage rate ~12 GB/hr) Near-real-time data extraction and image support Web-CGI applications that allow users to view and update their orders and subscriptions Subsystems: Data Distribution

New Data/Source Acquisition and Ingest Insert DB records for new data-source servers and data types Compose data-specific post-ingest scripts Configure ingest daemons if that method is going to be used for any of the new data types Define archive-device pools for product storage

New Data Product Cataloging Insert record into the core tables (catalog DB) that describe the mission and products Create mission specific database and objects, reusing objects from existing mission databases where applicable Compose a program to provide geographical L1 meta-data information including granule start and stop times, day- night flag, and geographic coordinates, e.g., MSl1info Compose functions for DB-metaload program, so meta- data files can be read and product tables can be populated Configure file migration and management actions for new mission data

New Data Processing Streams Define a recipe for each distinct serial processing stream Insert records for each recipe and each recipe step Compose a job template for each recipe Compose a ancillary-selection procedure for each recipe that requires ancillary data Compose scripts for each science program associated with the new mission's data Insert record in recipe-constraints table for each processing host allowed to run a recipe Compose AP-load procedure for each base data type that can be processed with a recipe Update the reproc program to support each base data type that has an AP-load procedure Configure L3-Scheduler for desired composite processing

VIIRS Ocean Level-3 Processing Develop data input routines to support processing of VIIRS Ocean EDRs using existing multi- mission Level-3 binning software

New Data Product Distribution Update Subscription CGI to support new mission data Compose match-subscription procedure If data extraction and mapping is to be supported: Compose extraction and mapping programs Compose match-XM-requests procedure Modify XM CGI to support new mission products Compose browser capabilities for new mission products Provide FTP access to new mission products

Page 1 Ocean PEATE Fred Patt. Page 2 Ocean PEATE Agenda Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule.

Similar presentations

Presentation on theme: "Page 1 Ocean PEATE Fred Patt. Page 2 Ocean PEATE Agenda Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Page 1 Ocean PEATE Fred Patt. Page 2 Ocean PEATE Agenda Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule.

Similar presentations

Presentation on theme: "Page 1 Ocean PEATE Fred Patt. Page 2 Ocean PEATE Agenda Science Team Introduction Ocean PEATE Design ODPS Design Overview Implementation Plan and Schedule."— Presentation transcript:

Similar presentations

About project

Feedback