High Performance Cluster Computing with PHOTOMOD HPC Edition Mikhail Drakin Head of Software Development Department Racurs September 2011, Tossa de Mar, Spain 11 th International Scientific and Technical Conference From Imagery to Map: Digital Photogrammetric Technologies

In Brief  Distributed processing of remote sensing data – the key to performance and efficiency  Parallelizing trends in modern hardware  Distributed processing in “conventional” PHOTOMOD: pros and cons  PHOTOMOD HPC Edition for computing clusters  PHOTOMOD Conveyor – the solution for on-line satellite monitoring

RSD – Ideal for Parallel Processing Most operations on remote sensing data satisfy the conditions for effective parallelization – can be easily split into numerous tasks, which:  Are similar in nature and/or  Take source data bits independent from other tasks and/or  Produce data bits independent from output of other tasks and/or  Are performed automatically

RSD – Ideal for Parallel Processing Common operations, implemented in automatic distributed mode in PHOTOMOD:  Tie points measurement  DTM calculation (pickets and “dense” DEM)  Orthorectification  Mosaics including sequence of distributed task sets: Cutlines creation Tie points searching Final mosaic building + a bunch of service operations (image format conversion, pan-sharpening, DEM reprojection, etc). Parallel processing helps saving precious time, reducing days of work to hours

Parallel Hardware Trends The same trend at all scales: performance gained by increasing number of identical computing units. This allows building applications scalable over a wide range  Increasing number of cores in off-the-shelf CPUs  Larger local networks  Supercomputers as clusters of tens to thousands similar nodes

“Conventional” PHOTOMOD Distributed processing first implemented in version 4.4. Each new version supports several new operations in distributed mode. Suited for medium-scale parallelization:  Distribution of tasks within single workstation to effectively utilize all processor cores  Distribution of tasks between several workstations to employ LAN resources of a department  Ideology: “one server – several workstations”

“Conventional” PHOTOMOD Drawbacks of distributed processing in standard PHOTOMOD:  Targeted at interactive work (flexible, but multiple GUI options complicate automation)  Not optimized for centralized administering  Implies storing configuration data, logs, etc. locally on the workstation Standard PHOTOMOD does not rely on consistent LAN structure and fixed hardware configuration, thus sacrificing some performance for flexibility.

Hardware Side Massive photogrammetric processing involves: Sophisticated computing algorithms Intensive data exchange Huge stored data volume CPU x K units? Net x M Gbps?x N TB? Storage Hardware configuration must be carefully planned and balanced to avoid bottlenecks

PHOTOMOD HPC Edition: Key Features PHOTOMOD HPC Edition – next level of distributed processing in PHOTOMOD  Ideology: “one workstation – multiple servers”  Oriented at coherent high-end hardware and software complex  Centralized administering, “less message boxes”  No modifications on computing nodes – all installation, configuration, logs storage, etc organized on a dedicated server Customization for particular customer needs and computing environment is implied.

Applications Fully automatic or semi-automatic solutions based on PHOTOMOD HPC Edition can pay off in processing of any kind of digital imagery, including:  Conventional aerial survey and rapidly emerging UAV imaging projects, with constantly increasing productivity and positioning quality, which allows fully automatic processing, especially with properly marked ground control points.  Rapid creation of quality orthomaps from HR and VHR satellite images for online monitoring (emergency management in situation centers, reconnaissance, etc).

Counterparts Examples of counterpart products:  VisionMap A3 Ground Processing System  PCI GeoImaging Accelerator (GXL)  Spot Image Pixel Factory  Photogrammetric systems with distributed processing (LPS, Inpho, etc) “Counterparts” here is not exactly equal to “Competitors”: each high-end product has specific niche.

PHOTOMOD HPC Edition: Distinctions  Distributed processing subsystem is native for the platform and constantly improved for specific applied needs  The system can be easily adopted to any hardware meeting basic requirements, not tied to some specific set of devices. The simplest cluster, which can bring significant productivity benefits with PHOTOMOD HPC, is probably 4 computing nodes (Windows HPC Server) with carefully designed storage system (e.g. FreeBSD-based), starting from $30-40 thousand  A computing cluster with HPC Edition can be tightly integrated with “conventional” interactive PHOTOMOD workstations, taking all advantages and eliminating drawbacks mentioned earlier  The system is initially positioned as a platform for building customized solutions

Testing in Kazakhstan Image courtesy of SmartCom December 2010: PHOTOMOD HPC was successfully tested on KazNTU computing cluster 10.9 TFLOPS 128 computing nodes NetApp storage system Windows HPC Server OS The experiment: Orthomosaic creation Output GSD 0.16 m Area 1289 sq. km DEM cell size 1.6 m Breaklines858 Total output files 156 GB Processing time: 3 hours

PHOTOMOD Conveyor Example of HPC Edition customization for specific application Collaborative project by Racurs and CTT Group (Russia) for heavy-duty mosaic production from VHR satellite data: hardware + software complex solution +

PHOTOMOD Conveyor Fully automatic workflow, including: Creation of the processing project from source satellite products Adjustment of the project (by RPC) Orthorectification by existing DEM Exclusion of images with undue cloud coverage Building cutlines Searching for tiepoints to eliminate residual discrepancies along seams Radiometric balancing Final stitching into seamless mosaic, saved into standard sheets set sq. km per day of 1: mosaic from GeoEye, DG or Alos

Future  Distributed GPU computing in cluster environment  Extensive set of automatic functions for data processing and enhancement  Library of fully automatic workflows for UAV processing, DTM creation and filtering, etc.  Advanced control over scheduling for convenient sharing of a single cluster between several workgroups  Integration with GIS systems, databases and geoportals  Any modifications and customizations requested by customers

Thank you for attention!