1 Cloud paradigm, standards and middleware for PGS * ESRIN *

Calvalus Full mission EO cal/val processing and exploitation services 2 Cloud paradigm, standards and middleware for PGS * ESRIN *

Deployment and processing scenario 3 Cloud paradigm, standards and middleware for PGS * ESRIN * UserCalvalus PortalHadoop Cluster installation package output data request v4.10 node 1 local disk node 2 local disk node n local disk... master feeder external data source or destination test server test 1 test 1 test 1 test 1 test 1 test 1 vm1vm1 node 3 local disk node 4 local disk

Selecting processors in the Calvalus portal 4 Cloud paradigm, standards and middleware for PGS * ESRIN *

Processor bundles 5 Cloud paradigm, standards and middleware for PGS * ESRIN * v4.10 A bundle is a versioned package of software and configuration and optionally auxiliary data. Bundles are deployed. Processing jobs refer to a bundle and a processor to be executed Examples: BEAM Case2R 1.5.3, SEADAS 6.3 Processor parameters installation package

Processor bundles for the BEAM framework 6 Cloud paradigm, standards and middleware for PGS * ESRIN * ls -1 /mnt/hdfs/calvalus/software/1.0/case2-regional beam-meris-case2-regional jar beam-collocation-1.3.jar beam-meris-brr-2.3.jar beam-meris-glint jar beam-meris-l2auxdata-1.2.jar beam-meris-radiometry jar beam-meris-sdr-2.3.jar flint-processor-1.2.jar bundle-descriptor.xml Java Service Provider Interface (SPI) declarations in META- INF/services SPI implementation “Case2IOPOperator.Spi” as factory of the BEAM GPF Operator BEAM GPF Operator implementation “Case2IOPOperator” with processor name “Meris.Case2Regional” annotation Processor descriptors with name “Meris.Case2Regional”, formal parameters, output variables

Requests and processor name resolution 7 Cloud paradigm, standards and middleware for PGS * ESRIN * L2 productionName case2r-${year}-${month} processorBundleName case2-regional processorBundleVersion processorName Meris.Case2Regional processorParameters true false true RADIANCE_REFLECTANCES...

Bundle descriptor for BEAM Operators 8 Cloud paradigm, standards and middleware for PGS * ESRIN * case2-regional BEAM Meris.Case2Regional MERIS Case2 Regional doAtmosphericCorrection boolean Whether or not to perform atmospheric correction. true... BEAM-DIMAP,NetCDF,GeoTIFF... !case2_flags.INVALID a_total_443 AVG

Deployment process for BEAM operators  Stand-alone test in VISAT (optional)  copying to HDFS, automated replication  bulk request submission (portal, batch client)  deployment via distributed cache, automatic unpacking, all JARs on class path  processor adapter  calls operator  provides input and output as product object  BEAM handles stream interface from/to HDFS via tile cache 9 Cloud paradigm, standards and middleware for PGS * ESRIN *

Processor bundles for UNIX executables 10 Cloud paradigm, standards and middleware for PGS * ESRIN * ls -1 /mnt/hdfs/calvalus/software/1.0/seadas-6.3 bundle-descriptor.xml l2gen-process.vm l2gen-prepare.vm l2gen-parameters.vm nasa-seadas-6.3.tar.gz

Converting a request to a UNIX call  Transformation of scripts with Velocity (.vm) or XSLT (.xsl) providing parameters, inputs and outputs in the context/document  execution of an optional “prepare” script to avoid unnecessary input retrieval and processing (in case output exists)  execution of a “process” script, the processor wrapper 11 Cloud paradigm, standards and middleware for PGS * ESRIN *

Converting a request to a UNIX call 12 Cloud paradigm, standards and middleware for PGS * ESRIN * #!/bin/bash inputPath="$inputFile" #[[ input=$(basename $inputPath) output=${input:0:8}2${input:9:$((${#input} - 12))}.hdf SEADAS=./seadas-6.3/nasa-seadas-6.3 L2GEN_BIN=${SEADAS}/run/bin/linux_64/l2gen L2GEN_ENV=${SEADAS}/config/seadas.env. $L2GEN_ENV # determine and get AUX data files... function handle_progress() { line=$1 if [[ ${line} =~ Processing\ scan\ #\ +[0-9]+\ +\(([0-9]+)\ +of\ +([0-9]+)\)\ +after ]]; then a1=${BASH_REMATCH[1]} a2=${BASH_REMATCH[2]} progress=$(echo "scale=3; ${a1} / ${a2}" | bc) printf "CALVALUS_PROGRESS %.3f\n" $progress fi } ${L2GEN_BIN} ifile=${inputPath} ofile=${output} par=parameters | \ while read x ; do handle_progress "$x" ; done echo CALVALUS_OUTPUT_PRODUCT ${output} ]]# l2gen-process.vm

Converting a request to a UNIX call 13 Cloud paradigm, standards and middleware for PGS * ESRIN * #!/bin/bash inputUrl=$1 outputDir=$2 inputFile=$(basename $inputUrl) outputFile=${inputFile:0:8}2${inputFile:9:$((${#inputFile} - 12))}.hdf outputPath=$outputDir/$outputFile if hadoop fs -ls $outputPath 2>&1 > /dev/null then echo "skipping $inputFile, $outputFile exists" echo “CALVALUS_OUTPUT_PRODUCT $outputPath” echo "CALVALUS_SKIP_PROCESSING yes“ fi l2gen-prepare

Converting a request to a UNIX call 14 Cloud paradigm, standards and middleware for PGS * ESRIN * = l2gen-parameters.xsl

Deployment process for UNIX executables 15 Cloud paradigm, standards and middleware for PGS * ESRIN *  Stand-alone test (optional)  packing as versioned.tar.gz file(s) + wrapper scripts  copying to HDFS, automated replication  bulk request submission (portal, batch client)  deployment via distributed cache, automatic unpacking, available as symlink in working dir  processor adapter  transforms scripts  checks availability of output  retrieves input into working dir  calls wrapper scripts (prepare, process, finalize)  archives output

Summary  Convention for processor software packages  Requests with processor (version) identification  Integration of existing processors by wrappers  Hadoop distributed cache for deployment  Input provision, working dir, output archiving  Progress monitoring, exception handling  Acknowledgement: The initial Calvalus idea was developed and its realisation was funded by the European Space Agency under the SME-LET programme. 16 Cloud paradigm, standards and middleware for PGS * ESRIN *