S.Cozzini www.egrid.it Den Haag, 25th November 2004 The EGRID project S.Cozzini www.egrid.it Den Haag, 25th November 2004

Agenda The EGRID project (short overview) The applications for EGEE The added value for EGRID The added value for EGEE Our commitments

The EGRID project A FISR project funded by MIUR (2003) Original Aim: Use of grid paradigm for research on complex system in economics and finance MIUR asked to similar FISR projects to coordinate together ICTP new task: Provide a National Italian Facility for economical and financial data using grid technology

EGRID and its friends Area project (M.Fanni) “Applicazioni di tecniche di soft computing (neural network/genetic algorithm) per analisi di sistemi complessi applicati alla finanza” Financial Databases INFM project (R. Mantegna) “Dinamica di altissima frequenza nei mercati finanziari” High Frequency Financial Data User Requirements ICTP EGRID project (S.Baroni) “Utilizzo di tecnologie Grid in modellazione sistemi complessi in ambito economico e finanziario” Provide the infrastructure using Grid computing

EGRID project: three lines .. Research in Grid Computing (G.Pau UCLA in collaboration with C.Kesselman) Research in finance/economics on the Grid (M. Gerla UCLA, S.D’addona Columbia) Implementation of the EGRID infrastructure (ICTP Trieste)

EGRID national infrastructure Version 1.0 released: 8-9 october 2004 (meeting in Trieste) Guide lines: Use available technology Develop and integrate only missing tools Grid provider: Grid.it Built after clearly identifying user requirements

User requirements in a nutshell High Frequency Financial Data: raw data Different Stocks Exchanges (NYSE/LSE/Paris/BorsaMilano) . Restricted access to raw data acquired under legally binding disclosure policies ( privacy and security) Raw Data should be: Safe kept/backed-up/shared on the GRID Actions on the raw data: transformations Filtering procedures (size reduced) Building procedures (size increased) Authorized and authenticated access to financial databases

The EGRID community 10 Different research groups in Italy (distributed community) 2 different groups in USA ~ 40 researchers European groups very much interested in joining our infrastructure..

EGRID 1.0: basic architecture Version 1.0 released: 8-9 october 2004 (meeting in Trieste) Globus based EDG middleware. Specific VO within Grid.it. ( our Grid provider..) A star shaped topology for data management: Central SE (large and backed up storage ) hosted in grid.it Local SE hosted in research centres acting as a temporary storage space. Data management based on EDG Replica manager

A graphical view AN Roma PADOVA CE WN WN Main Storage TS WN WN SuperNode: CE+SE+WN FI PA AL

EDG & Egrid EDGLimits Other problems EGRID solutions Data privacy not guaranteed EDG software installations: complex/invasive… Other problems Limited Bandwidth among different sites. EGRID solutions Temporary Work-around on privacy/security Temporary cache system based on star-topology Ad hoc procedures for installing (Live CD for UI and SuperNode)

EGRID solution: UI (2) Egrid UI now also on Live CD: Based on Knoppix 3.6 http://www.egrid.it/download/software/ui/

EGRID applications in EGEE Filtering/Building procedures Now inserted in a software layer built on the top of EDG commands and based on the EGRID architecture Insert them in web portal (GENIUS portal ?) Pricing financial derivatives by means of MCS Already parallel (almost embarrassingly) Under porting on GTK3 using mpich-g2 (USA partners) Under porting on the EGRID infrastructure

EGRID added values in joining EGEE EGEE middleware is addressing some of our requirements Egrid needs to de aligned with major european grid research avoiding duplications Egrid partners want to include European research group: infrastructure needs to be at European scale EGRID is now addressing Banks and Financial Institutions interested in using a continental grid instead of a national one.

EGEE added values in accepting EGRID very important security/privacy issues addressed A well trained and motivated group Scientific & academic community with a large technological transfer Financial & economical entities can play a determinant role in spreading grid technologies outside academic/scientific world

EGRID working plan within EGEE Applications domain Development of a portal for Filtering procedures Porting of parallel applications: pricing financial derivatives (USA partners) Other financial applications Data management domain: SRM ad hoc implementation to be used as a secure and general purpose caching system EGRID is now participating actively in the STORM server deployment together with INFN/CNAF

Other partnership/collaborations ITC-IRST Trento Collaboration on porting similar applications on the GRID INAF-Trieste Some contacts to work together on similar needs A local grid Testbed INFN/CNAF Storm server

EGRID solution: gridmap file pool account only on CE e WN security holes persist but less pronounced Personal accounts on SE: ldap server maintained by Egrid team Scripts/automatic procedures for creating/managing users

EGRID solution: UI (1) UI is independent from Linux distribution (mandrake/debian/fedora/Suse etc...) No need to use LCFG server UI includes only needed software and EGRID add-ons No root privileges needed: user can install it in home dir Freely Available http://www.egrid.it/sw/ui/egrid-ready-ui ~ 500MB

EGRID solution: the SN Egrid SuperNode Live CD Based on Knoppix 3.6 Standard tool for installing local nodes SN SE CE WN

Examples of EGRID procedures Loading files from UI and storing them on local and central SEs: EGRID command: egrid-upload <file> <lfn> Example: egrid-upload ./nyse-2002-01.tar.gz lfn:/00010/cd/nyse-2002-01.tar.gz Transforming data on the grid: egrid-transform <transformation> <lfn> <lfnprefix> egrid-transform taq-trades lfn:/brownlees/nyse-test.tar.gz lfn:/brownlees/test-data/ Transformation: application ported to the grid lfn: input logical filename lfnprefix: prefix for the output filename

EGRID caching system Avoid multiple download of the same file Coherence with access control procedure on data Minimize latencies for data trasfer overcome star topology adopted so far

What next: Full Caching system for EGRID Collaboration with Grid.it on common requirements: SRM on parallel filesystems Parallel Runs I/O performances on filtering procedures.. Accounting system: Important requirement in near future (it must present in final release)