Requirements for EO Data Processing Farms ESA Workshop “Models for Scientific Exploitation of EO Data” ESA ESRIN, Oct 11-12, 2012 Stephan Kiemle German Remote Sensing Data Center DFD German Aerospace Center DLR

Evolution of EO PGS Processing Facilities Dedicated Facility single mission dedicated hardware tight coupling, static scheduling predictable performance expensive investment, housing, operating no flexibility e.g. 1 st generation ENVISAT PAF Chart 2> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data > Shared Facility multi-mission shared hardware static deployment, dynamic scheduling controlled performance usage rate reduces costs growth and renewal still difficult e.g. ESA MMFI Virtualized Facility

Virtualized Processing Facilities multi-purpose independent hardware dynamic deployment and scheduling dynamic performance initial + continuous renewal investment, pay per use scaling with low impact on applications Sounds good! But … Chart 3> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data > Infrastructure as a Service Control VM Processor = #CPU + #Mbit/s Control Processor = #request Platform as a Service Control Processor = #VCPU + #MB/day Software as a Service Processor Control Host

Scientific Exploitation Use Case: Reprocess Tbyte of Archived EO Data Chart 4> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data > Processor in out in processing out in processing out in processing out in processing out in processing out in processing out in processing out p in processing out b in processing out

Example – Large Scale Reprocessing  joint analysis of processing and data management required execute processing algorithms where the data is cross-distribute data archiving Chart 5> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data > Sentinel-5 Precursor L1b-L2 reproc. 1 year n14.3 * 365 = 5220 products s50 GB per L1b product r2 % in-to-out ratio t2400 s processing time per product (21MB/s) - moderate Local FacilityLAN CloudWAN Cloud b1 Gbit/sb500 Mbit/sb100 Mbit/s p6.9 nodesp3.9 nodesp1.6 nodes 24 days50 days248 days

Example – Small Scale Analysis Chart 6> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data > Hypothetic Data Analysis Scenario n1000 products s1 GB per input product r10 % in-to-out ratio t300 s processing time per product (3.4 MB/s) - complex Local FacilityLAN CloudWAN Cloud b1 Gbit/sb500 Mbit/sb100 Mbit/s p(38) 10 nodesp19.4 nodesp4.8 nodes 8.7 hours4.6 hours22.7 hours  processing complexity versus data volume determines distribution

Requirements for EO Data Processing Farms Processing performance versus i/o rate Dynamically balance distributed processing taking into account number of CPUs, RAM, disk cache allocated other local resources (e.g. embedded DBs, log files) actual transfer rates for inputs, auxiliary data, outputs Coordination Define procedures and guidelines for use Reconcile conflicts between projects Accounting Monitoring and control Privacy/security/availability Clear separation of production environment and other “scientific” environments Chart 7> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data >

Consequences for Processing and Data Management Individual analysis for best system approach (local, farm, private cloud, …) data rates, processing level/complexity project characteristics, processing strategies Algorithms encapsulated in deployable processors/processing systems Data processors shall dynamically use CPUs, RAM, disk cache as allocated Establish/extend standards for algorithm integration and processor deployment Bulk product transfer capabilities, pipelining/streaming for input data set provision and output data set repatriation Evolve archives to data lifecycle centers layered data sets for tailored access performance defined consolidation/migration capacities (LTDP context) new primary data access interfaces: geodata, time series Chart 8> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data >

“GeoFarm” for Scientific EO Data Exploitation at DLR Oberpfaffenhofen 2 Blade Centers (Dell), total 672 cores Opteron, 3.3 TB RAM, interconnected with 10Gb/s Ethernet 288 TB SAN storage, connected with 4 GB/s Fiber-Channel Virtualized using Citrix XenServer 6 (advanced edition) Separated pools for production network/normal infrastructure Usage examples: Project scope: ENVISAT/MERIS data reprocessing for CCI Fire using CATENA Continuous operational: O3M-SAF NRT, offline and re-processing Ongoing definitions: use scenario and application procedure monitoring accounting, cost calculation and sharing Chart 9> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data >

Conclusions Evolution of processing facilities towards virtualization Different scientific EO exploitation use cases require different distributed computation models, depending on input data size, processing complexity/strategy and network bandwidth Requirements in context of EO data processing farms: processors need to become deployable in standard environment and dynamically use allocated resources bulk input data provision using elaborated data management principles and technologies DLR operates a virtualized EO processing infrastructure “GeoFarm” Chart 10> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data >

Thank you! Questions? Stephan Kiemle Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) German Aerospace Center Earth Observation Center | German Remote Sensing Data Center Oberpfaffenhofen Wessling | Germany Chart 11> Req. for EO Data Processing Farms > Stephan Kiemle Models for Scientific Exploitation of EO Data >