Hungarian GRID Projects and Cluster Grid Initiative P. Kacsuk MTA SZTAKI

Clusters Hungarian Grid projects VISSZKI Globus, Condor service SuperGrid Resource scheduling AccountingP-GRADEMPICH-G Supercomputers DemoGrid SecurityGrid monitoring Data storage subsystem Applications

–Testing various tools and methods for creating a Virtual supercomputer (metacomputer) –Testing and evaluating Globus and Condor –Elaborating a national Grid infrastructure service based on Globus and Condor by connecting various clusters Ojectives of the VISSZKI project

Results of the VISSZKI project PVMMW Condor-G Globus Grid middleware Grid fabric MPI Local job management Clusters Condor Clusters Condor Clusters Condor Clusters Condor Low-level parallel development Grid level job management

GRID Generic architecture Structure of the DemoGrid project  GRID subsystems study and development -Data Storage subsystem: Relational database, OO database, geometric database, distributed file system -Monitoring subsystem -Security subsystem Demo applications –astro-physics –human brain simulation –particle physics –car engine design Data Storage Subsystem Monitoring subsystem Security subsystem Applications Hardver RDB OODB GDB DFS Data Tightly Loosely Decomp StorageCPU Network

Structure of the Hungarian Supercomputing Grid 2.5 Gb/s Internet NIIFI 2*64 proc. Sun E10000 BME 16 proc. Compaq AlphaServer ELTE 16 proc. Compaq AlphaServer SZTAKI 58 proc. cluster University (ELTE, BME) clusters

P-GRADE PVMMW Condor-G Globus SUN HPC Compaq Alpha Server Compaq Alpha Server Klaszterek Grid middleware Grid fabric High-level parallel development layer Grid level job management Condor, LSF, Sun Grid Engine Condor, PBS, LSF Condor MPI GRID portal Web based GRID access Condor, PBS, LSF GRID application Low-level parallel development The Hungarian Supercomputing GRID project

Distributed supercomputing: P-GRADE P-GRADE (Parallel GRid Application Development Environment) The first highly integrated parallel Grid application development system in the world Provides: –Parallel, supercomputing programming for the Grid –Fast and efficient development of Grid programs –Observation and visualization of Grid programs –Fault and performance analysis of Grid programs

Condor/P-GRADE on the whole range of parallel and distributed systems Super- computers 2100 Clusters Grid GFlops Mainframes P-GRADE Condor P-GRADE Condor P-GRADE Condor flocking

Berlin CCGrid Grid Demo workshop: Flocking of P-GRADE programs by Condor m0m1 P-GRADE Budapest n0n1 Madison p0p1 Westminster P-GRADE program runs at the Madison cluster P-GRADE program runs at the Budapest cluster P-GRADE program runs at the Westminster cluster

Next step: Check-pointing and migration of P-GRADE programs m0m1 P-GRADE GUI Wisconsin Budapest n0n1 London 1 P-GRADE program downloaded to London as a Condor job 2 P-GRADE program runs at the London cluster London cluster overloaded => check-pointing 3 P-GRADE program migrates to Budapest as a Condor job 4 P-GRADE program runs at the Budapest cluster

Further develoment: TotalGrid TotalGrid is a total Grid solution that integrates the different software layers of a Grid (see next slide) and provides for companies and universities –exploitation of free cycles of desktop machines in a Grid environment after the working/labor hours –achieving supercomputer capacity using the actual desktops of the institution without further investments –Development and test of Grid programs

Layers of TotalGrid Internet Ethernet PVM v. MPI Condor v. SGE PERL-GRID P-GRADE

Hungarian Cluster Grid Initiative Goal: To connect the new clusters of the Hungarian higher education institutions into a Grid By autumn 42 new clusters will be established at various universities of Hungary. Each cluster contains 20 PCs and a network server PC. –Day-time: the components of the clusters are used for education –At night: all the clusters are connected to the Hungarian Grid by the Hungarian Academic network (2.5 Gbit/sec) –Total Grid capacity in 2002: 882 PCs In 2003 further 57 similar clusters will join the Hungarian Grid –Total Grid capacity in 2003: 2079 PCs Open Grid: other clusters can join at any time

Structure of the Hungarian Cluster Grid 2.5 Gb/s Internet 2003: 99*21 PC Linux clusters, total 2079 PCs 2002: 42*21 PC Linux clusters, total 882 PCs TotalGrid

Live demonstration of TotalGrid MEANDER Nowcast Program Package: –Goal: Ultra-short forecasting (30 mins) of dangerous weather situations (storms, fog, etc.) –Method: Analysis of all the available meteorology information for producing parameters on a regular mesh (10km->1km) Collaborative partners: –OMSZ (Hungarian Meteorology Service) –MTA SZTAKI

Structure of MEANDER First guess data ALADIN SYNOP data SateliteRadar CANARI Delta analysis Basic fields: pressure, temperature, humidity, wind. Derived fields: Type of clouds, visibility, etc. GRID Rada r to grid Satelit e to grid Current time Type of clouds Overcast Visibilit y Rainfall state Visualization For meteorologist s:HAWK For users: GIF Lightning decode

P-GRADE version of MEANDER

Implementation of the Delta method in P-GRADE

netCDF output 34 Mbit Shared PERL-GRID CONDOR-PVM job 11/5 Mbit Dedicated P-GRADE PERL-GRID job HAWK netCDF output Live demo of MEANDER based on TotalGrid ftp.met.hu netCDF 512 kbit Shared netCDF input Parallel execution

GRM TRACE 34 Mbit Shared PERL-GRID CONDOR-PVM job 11/5 Mbit Dedicated P-GRADE PERL-GRID job On-line Performance Visualization in TotalGrid ftp.met.hu netCDF 512 kbit Shared netCDF input Parallel execution and GRM GRM TRACE Task of SZTAKI in the DataGrid project

PROVE visualization of the delta method

Conclusions Already several important results that can be used both for –the academic world (Cluster Grid, SuperGrid) –commercial companies (TotalGrid) Further efforts and projects are needed –to make these Grids more robust more user-friendly –having more new functionalities

Thanks for your attention ? Further information: