1. 1 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013

2. Calvalus Full mission EO cal/val processing and exploitation services 2 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013

3. Deployment and processing scenario 3 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 UserCalvalus PortalHadoop Cluster installation package output data request v4.10 node 1 local disk node 2 local disk node n local disk... master feeder external data source or destination test server test 1 test 1 test 1 test 1 test 1 test 1 vm1vm1 node 3 local disk node 4 local disk

4. Selecting processors in the Calvalus portal 4 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013

5. Processor bundles 5 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 v4.10 A bundle is a versioned package of software and configuration and optionally auxiliary data. Bundles are deployed. Processing jobs refer to a bundle and a processor to be executed Examples: BEAM Case2R 1.5.3, SEADAS 6.3 Processor parameters installation package

6. Processor bundles for the BEAM framework 6 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 hadoop@master00:~$ ls -1 /mnt/hdfs/calvalus/software/1.0/case2-regional-1.5.3 beam-meris-case2-regional-1.5.3.jar beam-collocation-1.3.jar beam-meris-brr-2.3.jar beam-meris-glint-1.2.1.jar beam-meris-l2auxdata-1.2.jar beam-meris-radiometry-1.0.1.jar beam-meris-sdr-2.3.jar flint-processor-1.2.jar bundle-descriptor.xml Java Service Provider Interface (SPI) declarations in META- INF/services SPI implementation “Case2IOPOperator.Spi” as factory of the BEAM GPF Operator BEAM GPF Operator implementation “Case2IOPOperator” with processor name “Meris.Case2Regional” in @OperatorMetadata annotation Processor descriptors with name “Meris.Case2Regional”, formal parameters, output variables

7. Requests and processor name resolution 7 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 L2 productionName case2r-${year}-${month} processorBundleName case2-regional processorBundleVersion 1.5.3 processorName Meris.Case2Regional processorParameters true false true RADIANCE_REFLECTANCES...

8. Bundle descriptor for BEAM Operators 8 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 case2-regional 1.5.3 BEAM Meris.Case2Regional MERIS Case2 Regional doAtmosphericCorrection boolean Whether or not to perform atmospheric correction. true... BEAM-DIMAP,NetCDF,GeoTIFF... !case2_flags.INVALID a_total_443 AVG 1.0...

9. Deployment process for BEAM operators  Stand-alone test in VISAT (optional)  copying to HDFS, automated replication  bulk request submission (portal, batch client)  deployment via distributed cache, automatic unpacking, all JARs on class path  processor adapter  calls operator  provides input and output as product object  BEAM handles stream interface from/to HDFS via tile cache 9 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013

10. Processor bundles for UNIX executables 10 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 hadoop@master00:~$ ls -1 /mnt/hdfs/calvalus/software/1.0/seadas-6.3 bundle-descriptor.xml l2gen-process.vm l2gen-prepare.vm l2gen-parameters.vm nasa-seadas-6.3.tar.gz

11. Converting a request to a UNIX call  Transformation of scripts with Velocity (.vm) or XSLT (.xsl) providing parameters, inputs and outputs in the context/document  execution of an optional “prepare” script to avoid unnecessary input retrieval and processing (in case output exists)  execution of a “process” script, the processor wrapper 11 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013

12. Converting a request to a UNIX call 12 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 #!/bin/bash inputPath="$inputFile" #[[ input=$(basename $inputPath) output=${input:0:8}2${input:9:$((${#input} - 12))}.hdf SEADAS=./seadas-6.3/nasa-seadas-6.3 L2GEN_BIN=${SEADAS}/run/bin/linux_64/l2gen L2GEN_ENV=${SEADAS}/config/seadas.env. $L2GEN_ENV # determine and get AUX data files... function handle_progress() { line=$1 if [[ ${line} =~ Processing\ scan\ #\ +[0-9]+\ +\(([0-9]+)\ +of\ +([0-9]+)\)\ +after ]]; then a1=${BASH_REMATCH[1]} a2=${BASH_REMATCH[2]} progress=$(echo "scale=3; ${a1} / ${a2}" | bc) printf "CALVALUS_PROGRESS %.3f\n" $progress fi } ${L2GEN_BIN} ifile=${inputPath} ofile=${output} par=parameters | \ while read x ; do handle_progress "$x" ; done echo CALVALUS_OUTPUT_PRODUCT ${output} ]]# l2gen-process.vm

13. Converting a request to a UNIX call 13 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 #!/bin/bash inputUrl=$1 outputDir=$2 inputFile=$(basename $inputUrl) outputFile=${inputFile:0:8}2${inputFile:9:$((${#inputFile} - 12))}.hdf outputPath=$outputDir/$outputFile if hadoop fs -ls $outputPath 2>&1 > /dev/null then echo "skipping $inputFile, $outputFile exists" echo “CALVALUS_OUTPUT_PRODUCT $outputPath” echo "CALVALUS_SKIP_PROCESSING yes“ fi l2gen-prepare

14. Converting a request to a UNIX call 14 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013 = l2gen-parameters.xsl

15. Deployment process for UNIX executables 15 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013  Stand-alone test (optional)  packing as versioned.tar.gz file(s) + wrapper scripts  copying to HDFS, automated replication  bulk request submission (portal, batch client)  deployment via distributed cache, automatic unpacking, available as symlink in working dir  processor adapter  transforms scripts  checks availability of output  retrieves input into working dir  calls wrapper scripts (prepare, process, finalize)  archives output

16. Summary  Convention for processor software packages  Requests with processor (version) identification  Integration of existing processors by wrappers  Hadoop distributed cache for deployment  Input provision, working dir, output archiving  Progress monitoring, exception handling  Acknowledgement: The initial Calvalus idea was developed and its realisation was funded by the European Space Agency under the SME-LET programme. 16 Cloud paradigm, standards and middleware for PGS * ESRIN * 23.01.2013