Cluster Computing for SWAP Crop Model Parameter Identification using RS HONDA Kiyoshi Md. Shamim Akhter Yann Chemin Putchong Uthayopas

Importance of Crop model parameters identification  Agriculture Activity Monitoring Sowing date, Cropping intensity, Growth, Water stress, Yield, etc. Production for Food Security Water Management in Irrigation Activity Remote Sensing is quite useful data, however modeling has not been done. – empirical model  Crop growth model ( SWAP ) Continuous monitoring in various aspects Prediction Spatial Parameter estimation & Calibration ->RS  Data Assimilation Technique To estimate parameters which cannot be observed by RS High-Resolution Remote Sensing High-Resolution Remote Sensing Temporal Info -> Low Resolution Remote Sensing; Mixed Pixel Temporal Info -> Low Resolution Remote Sensing; Mixed Pixel

SWAP Model Diagram Adopted from Van Dam et al. (1997) Drawn by Teerayut Horanont (AIT)

SWAP Model Parameter identification - Data Assimilation using RS and GA - - Data Assimilation using RS and GA Day Of Year Evapotranspiration LAI RS Observation SWAP Crop Growth Model SWAP Input Parameters sowing date, soil property, Water management, and etc. LAI, Evapotranspiration Day Of Year Eavpotranspiration LAI Fitting LAI, Evapotranspiration Assimilation by finding Optimized parameters By GA RS Model

Field photos  Longitude:  Latitude:

 Estimated parameters  DOYCrop1= 19  DOYCrop2= 188  Crop.Int.Crop2= 0.32  Fitness=  Generation found= 31 (popsize=5)  Calculation timeapproximate 15 minutes

A Practical Problem Arises  A practical issue arises with the overall calculations time load for assimilating data with remote sensing data.  The calculation time for identify SWAP parameters only for 1 pixel (1 sq.km) takes several minutes to 30 minutes.  Thus, a RS image of 1000 x 1000 sq.km of 1000x1000 pixels will take more than 50 years (30min x 1000 x 1000) is not acceptable.  Solutions LUT Method Segmentation Parallel Computing

Objectives  Implement SWAP-GA model in cluster computers  Propose and evaluate job distribution models Population by Population Pixel by Pixel Hybrid  Develop a model to estimate calculation time

 3 Clusters (AIT and Kasetsart universities Clusters)  Optima 9 nodes: Front-end: Athlon XP MB RAM 80 GB IDE Disk Compute Node: Athlon Xp MB RAM 40 GB IDE Disk Disk Interconnection: Fast Ethernet  Magi 4 nodes: Athlon XP MB RAM 80 GB Hard Disk Disk Interconnection: Gigabit Ethernet Interface Clusters Technical Specifications

Maeka 32 nodes: Front-end: Dual AMD Opteron 1400 MHz RAM PC2100 : 6GBs Disk SCSI Ultra 320: 72GBs 15 machines Dual AMD Opteron 1400 MHz RAM PC2100 : 3GBs Hard Disk SCSI Ultra 320: 72GBs 16 machines Dual AMD Opteron 1800 MHz RAM PC2700 : 3GBs Hard Disk SCSI Ultra 320: 72GBs Gigabit Ethernet Interface2 Technical Spec Continue..

Implementation Schemes  3 Implementation Schemes are proposed  Population by population (Distributing population)  Pixel by pixel  Hybrid (combination of Population by population and pixel by pixel)

Distributing Population by Population  Each Generation have n populations to be evaluated  1 population ( 1 set of parameters ) will be distributed to 1 slave. If n is more than slave, more than 1 population will be distributed to 1 slave.  Slaves do the evaluation, generate fitness and send back the population (with fitness) to Master. SLAVE n n=popsize Population n./SWAP MASTE R SWAP-GA 1PIXEL SLAVE 1 Population 1./SWAP Master Slave Distribute Population Without Fitness  Read Input Files  Run SWAP  Analyze Result  Population With Fitness Next Generation

 All pixels will be distributed among the available slaves.  Each slave will evaluate total serial SWAP-GA procedure inside itself for one pixel at a time and produce a total assimilation result for that pixel in a file in their local Hard Disk Distributing Pixel by Pixel SLAVE n n=10 1 PIXEL SWAP-GA MASTER SLAVE 1 1 PIXEL SWAP-GA Nodes Master Slave Distribute Pixel  All Populations Without Fitness  Read Input Files  Run SWAP  Analyze Result  All Populations With Fitness Assimilation result Next Generation

Slave 1 1 PIXEL SWAP- GA Slave n 1 PIXEL SWAP- GA... Slave n n=popsize Population n./SWAP Local Master 1 PIXEL SWAP-GA Slave 1 n=popsize Population n./SWAP Available Nodes 8 Available Nodes Hybrid (Population by Population and Pixel by Pixel Together) Here Node 0,3 are Master Node 1,2,4,5 are Slaves Here Node 0,4 are Master Node 1,2,3,5,6,7 are Slaves Pixel1 Pixel2 pop1 pop2 pop1 pop2 Master

Hardware Architecture Influence on Speed up : ( Pop & Pop: 1 SWAP on 1 Slave )  Distribution Population Method  Copy all necessary (inputted) files (CF)=336KB, Read files (RF)=.0067KB, Run SWAP executable file (RS)=32KB, Write files (WF)=1.46KB  The parameters inside parallel SWAP-GA are fixed in all Clusters (No of population=2, No of Generation=1, PXOVER=0.8, PMUTATION=0.25) The Bandwidth Times for Copying Files and Executing SWAP According to above Table it is clear that Hardware Architecture Influence on Speed Up. Copy FileRun 1 SWAP

Software Architecture Influence on Speed up: ( Total Throughput / n-SWAP : sec/SWAP) Average SWAP Execution Time (in second) on Different Clusters  Here, a comparison between the running time (in second) of a single SWAP execution (Total Model running time / Total No. of SWAP execution) on these Clusters (by guessing that the I/O factors and GA running factors are taking constant time in all cases). Parameters in Optima, Maeka and Magi: -No of pixel (9), No of Population (10), No of generation (20). -9 processors are used in Optima and Maeka but due to reason that Magi is a 4 nodded Cluster so, 4 processors are used in Magi. Hybrid model is given best performance (time domain) among the three predefined models

Compare Serial SWAP_GA with Clustering SWAP-GA Figure: Timing Diagram Serial SWAP-GA and Parallel SWAP-GA by Distributed Pixel in Optima Figure: Timing Diagram Serial SWAP-GA and Parallel SWAP-GA by Distributing Population in Optima Pop by Pop model n Slaves = n Pop. Serial Cluster Pix by Pix model n pixel = n slave

Scalability for Pop by Pop model ( Optima: n Slave = n Pop, 1 pix ) The symbols, P i is the population number, T is the time in population pi and generation gj. Agj and Cgj are find out with respect to the value of gj and used as constants to find out the value of T. For i belongs to population, j belongs to generation T =Ag j P i + Cg j (sec) (Serial SWAP-GA) Ag j = g j – 6.94 Cgj = g j (Parallel SWAP-GA) Ag j = 0.865gj Cg j =5.57 gj – 1.93  However, within this experiment highest 25 generations and 8 populations are used but all the above equations have given a generic model.  For any value of generation and population number the running time in both parallel SWAP-GA and serial SWAP-GA can be measured by using these equations.  The above equations can solve the complication of running parallel SWAP-GA within Clusters those hold limited number of slaves (PC). gjgj Ag j Cg j PiPi T(sec) Serial Parallel

Conclusion & Future Work  3 Implementation Models were proposed and evaluated on 3 clusters  Hybrid model always gives the fastest result  A Model to estimate calculation time on Optima was established. Models in different model will be developed.  Improve Time Measurement  Implementation for bigger clusters ( new cluster in Kasetsrat, Titech )  A monitoring System ( CPU load )  Finding idling CPU to send jobs : scheduler  Implementation to Thai GRID.  A fully SWAP-GA software module Package, documentation, scripting.  We believe that this research will include one new part not only in the Grid-Cluster computing arena but also in the Remote Sensing image analyzing field.