Numerical Grid Computations with the OPeNDAP Back End Server (BES)

Two part problem; 1.Compute the so called Stokes vector for a given vector parameter space P using a model F(x,y,λ; P) 2.Correct P to improve the fit of the theoretical signal to the observed signal using non-linear optimization The purpose is to determine a vector parameter space for which the model produces the best fit in the Least Square sense. The complete problem is solved with a Numerical Library Called MERLIN Very computational Intensive, one complete dataset can take hours on a single node. The Problem

High Performance Solution Create multiple versions of the computing node waiting for data to crunch Create a central place (controller) to browse spectra- polarimetry data inventories and launch inversions upon user’s request. Let the controller decide how to distribute the problem. Requires multiple identical processes waiting to be called for duty.

Born to Grid Hyrax BES was designed from the beginning with grid services in mind Suited for problems that DO NOT REQUIRE coordination of the points after each iteration (see next slide) Each point is solved individually regardless of the solution to all other points.

What this problem is not Do not think of a problem like the Navier-Stokes equation using a cluster and finite element methods with mesh refinement. Think Scatter-Gather or Embarrassing Parallel. The Spectral Inversion Problem does not require boundary information to be exchanged in the solution.

The Dataset: Spectral Images Single element from data cube Y X λ

A slice is nice! Each node gets a Y time λ slice

 Each computational node, that is, each BES gets from the controller an OPeNDAP URL specifying the dataset on which to operate and the specific slice using OPeNDAP constraints.  Each computational node returns the solution to the controller as an OPeNDAP data object.  Controller collects all solutions and assembles a complete result as a netCDF data file.  Time to compute solution scales LINEARLY with number of nodes  Grid can be homogeneous or heterogeneous. We do not call it scatter-gather for nothing

BES Server 1 BES Server BES Server n GRID Inversion Engine Data local or remote using OPeNDAP over HTTP Client Each server computes the inversion over spectral images (  y), all n servers compute the complete set (x,  y), HOW SLICING WORKS IN THE INVERSION ENGINE APACHE + MODULE

OUTPUT INVERSION RESULTS Inversion Engine MERLIN ANCILLARY DATA: Instrument Profile Dispersion Spectral Bandpass(es) Atomic Data for Spectral Lines Genetic Algorithm or Neural Network provides Initial Guess Data Source Process HYRAX BES RUNNING MERLIN AS A MODULE CALIBRATED INPUT DATA (e.g. ASP, DLSP, HINODE,...) Process with Custom Data Filters to Standard Data Engine Format Polarization Reference Frame: +Q along Solar E-W Remote OPeNDAP server

Conclusions Hyrax BES can dynamically load modules that not only do data serving and manipulation, but server side functionality as well. The use of multiple servers running Hyrax can easily be implemented to linearly improve the performance. This applies to both server side processing as well as load balancing for serving data in demanding systems like the Earth System Grid. Hyrax Modules offer a simple way to extend the functionality of your server, as well as seamlessly integrate multiple facets of data serving and processing.

QUESTIONS?