1. e-infrastructures in Portugal
Portugal - Lisboa
19th April 2010
e-infrastructures in Portugal
Hepix 2010 Spring Conference
G. Barreira, G.Borges, M.David, N.Dias, H. Gomes, J.Gomes, J. Martins,

2. Outline Introduction Networking Data Repositories
High Performance Computing
Grid Computing

3. Introduction

4. LIP Portuguese Scientific Research Laboratory Main Activities
Private non-profit association
Created in 1986 when Portugal joined CERN
Funded by the Portuguese funding agencies
Main Activities
ATLAS, CMS, COMPASS, AUGER, AMS, SNO, ZEPLIN, HADES, ...
Instrumentation, Building Detectors and DAQ systems, Detectors R&D, Electronics, Precision Mechanics, ...
Medical Physics
Grid Computing

5. LIP Computing LIP Computing Group 55 Computing support for LIP
10 persons (Backgrounds from computer science and physics)
Operation of LIP Lisbon / LIP Coimbra infrastructures
Remote operation of other infrastructures
Main node for Grid Computing (PT-NGI)
Computing support for LIP
Support for local users
Grid computing research
DataGrid, CrossGrid, EELA, LCG, EGEE, EGEE-II, EGEE-III, int.eu.grid, EGI-DS, IBERGRID, INGRID/+
Starting some HPC activity

6. LIP Computing Team 66

7. Networking

8. Networking / RCTS 5 Gpbs 5 Gpbs RCTS – The Portuguese academic network
Operated by FCCN, the Portuguese NREN
Advanced optical network using fibres owned by FCCN
Serves the academic and research community
Provides IPV4 and IPV6
RCTS & Rediris connected by optical fibre
Better international connectivity to Spain and to the Geant backbone
Better Redundancy with the Iberic Ring
5 Gpbs

9. Networking / RCTS RCTS provides network services for e-infrastructures
Layer 2 connectivity to interconnect remote clusters (a L2 cloud has been deployed to connect the PT federated Tier-2 sites)
Bandwidth for cluster interconnection within the country is not accounted
You can have the maximum bandwidth that the technologies allow without paying more for that
Up to 10 Gigabit using VLANs
Lambdas also possible
FCCN together with the Portuguese NGI also have housing capabilities for clusters with the network connectivity included

10. Data Repositories

11. Scientific Data Repositories
1111
There is an initiative for scientific data repositories
Objective: provide data storage services for important scientific data
FCCN operates a storage system for data repositories housed at the main node for grid computing
The facility was installed just a few months ago and is becoming operational
Tape library and online storage
Several methods for data access
Grid enabled

12. High Performance Computing

13. HPC 1313 Very low activity in the HPC domain HPC resources
No policy for HPC at the government level
There isn’t any real large HPC facility in Portugal
HPC resources
Few small clusters more focused on parallel computing
CFC, UMinho, UPorto, IST, LNEC
The largest HPC cluster is CFC and has 520 cores AMD
All the others have less than 300 CPU cores
The most powerful is IST with 1.6Tflop/s IBM power5
Low latency interconnects
UMinho in Braga about 160 CPU cores and Myrinet
NCG in Lisbon some blades with Infiniband
User community is rather small
Large investments not foreseen
IST cluster

14. HPC
1414
Portugal in PRACE
University of Coimbra through CFC; is the Portuguese partner
CFC is the only organization that is actually opening regular calls for the user community to share its cluster resources
There is some increase of HPC activity
Mainly since last year
Following the agreement signed the government with the TACC / Univ of Austin
Several HPC training events and workshops took place
NCG (PT-NGI) HPC resources are starting to be used by the community
CFC cluster

15. Grid Computing

16. LIP Certification Authority
1616
Grid infrastructures at the global scale require a global trust domain for authentication
The IGTF International Grid trust Federation operates a global trust domain for grid computing
The IGTF is composed of three Policy Management Authorities (Americas, Europe, Asia Pacific)
The LIP CA was created in 2001 to issue authentication credentials for grid computing in Portugal
The LIP CA is accredited by the EUgridPMA and is a component of the IGTF trust domain
The LIP CA has a network of registration authorities for certificate issuance and provides X.509 grid certificates to the Portuguese scientific and academic community

17. Grid Projects with Portuguese contributions
Ingrid
Ibergrid

18. Interactive European Grid Project (I2G)
R-GMA for development
Portuguese partner
LIP-Lisbon
Core and Local services
Operations management and coordination
Helpdesk
Security
coordination
Backup Production
Core Services
CrossBroker
RAS
BDII
VOMS
LFC
MyProxy
APEL accounting
GridICE
R-GMA
Development Core
Services
CrossBroker
RAS
BDII
VOMS
LFC
MyProxy
Pure gLite WMS
Autobuild
Repository
SAM
Network
monitoring
Production Core
Services
CrossBroker
RAS
BDII
VOMS
LFC
MyProxy

19. I2G / Migrating Desktop
Migrating Desktop (MD): User Friendly Grid Access
Java based GUI; Hides the details of the grid
Provides interactivity and visualization features
GVid enables interactivity for OpenGL and X applications
Allows to log-in in the GRID independently from
where you are (laptop, desktop, everywhere ...)
what kind of Computer/OS you are using
Roaming Access Server (RAS):
Gateway for Grid Access
Performs actions on the grid on behalf of the MD UI
RAS MD
GUI to access Grid Services
Data Management
Interactivity and Visualization
Submission of Parallel Jobs
Submission of Interactive Jobs

20. I2G / CrossBroker CrossBroker: Int.EU.Grid meta-scheduler
Offers the same functionalities as the EGEE WMS
Support for Interactive Applications via GVid/Glogin
Full support for Parallel Applications
PACX-MPI and OpenMPI
Scheduling for intra-cluster and inter-cluster jobs
Flexible MPI job startup based on MPI-START
PACX-MPI and OpenMPI support
MPI job startup based on MPI-START
Support for interactivity via GVid/Glogin
WMS
CrossBroker

21. I2G / MPI-START MPI-START: Abstraction layer for MPI jobs WN
Sits between CrossBroker, LRMS schedulers and MPI implementations
Hides MPI job start details
Provides a uniform method to start jobs independently of:
LRMS (PBS/Torque, SGE, …)
MPI implementation (OpenMPI, PACX-MPI, MPICH, …)
Hides local infrastructure details
Shared/not shared home directories
Location of MPI libraries and other local specificities WN
Hooks
Scheduler Plugin: PBS, SGE, LSF, ...
MPI Plugin: OpenMPI, PACX-MPI, ...

22. Portuguese WLCG contributions
2222
WLCG as been the driving force in grid computing in Portugal
Federated PT-Tier 2: LIP-Lisbon, LIP-Coimbra, NCG-INGRID-PT
Grid Core LIP-Lisbon & NCG-INGRID-PT
For resource integration; Critical services;
For WLCG but also for many other projects: EGEE, PT-NGI, IBERGRID...
gLite services
WMSes, TOP-BDII, PX server, VOMS server, LFC, VOBOXes
gLite services are very poor in terms of redundancy and resiliency to failures
Virtualisation approach to grid services

23. ( Virtualisation / Xen paravirtualisation )
2323
Why
Encapsulation
Provide multiple environments for multiple applications and services
Dynamic resource allocation profiting from existing resources
Where and how
Computing farm (mixture of real and virtual resources)‏
To host grid (or other) persistent services
For testing purposes
Things to explore
Dynamic approach to virtual resources for classic farm computing (enable multiple environments tailored to the VOs)‏
Flexible framework for persistent VMs enabling resilient service provisioning
Cloud computing services on top of the existing resources (VMs on demand)‏
KVM (Kernel-based Virtual Machine) native virtualization

24. Central 10 Gigabit Ethernet switch
( Virtualisation / Redundant services )
Redundant services distributed between 2 blade centres
Solution based on Xen Virtual machines
Xen images available Storage accessible via Internet SCSI (iSCSI)
Controlled by OCFS2 shared cluster file system...
ISCSI ARRAY 1
12 x 1TB SATA disks
Raids 10, 6 LUNS
Central 10 Gigabit Ethernet switch
Blade centre
...
ISCSI ARRAY 6
12 x 1TB SATA disks
Raids 10, 6 LUNS
2 x 512 MB iSCSI controllers
Blade centre 24
2 x 512 MB iSCSI controllers
OCFS2 Filesystem
Xen Images Repository 24

25. Tier-2 sites topology and resources
2525
gLite
Computing
Element
SRM +
GSIFTP doors
Monitoring
Box
Site
BDII
LRMS
Farm
Force10
core switch
SRM storage
LUSTRE + STORM

26. Farm Systems 2626 Resource Management Systems
One single FARM in each site
Better optimization and management
Complex resource management
Heterogeneous (blades, servers, ...)
Hardware (AMD, Intel, several generations)‏
Administrative (different groups and purposes, grid and non-grid)‏
Resource Management Systems
Tried and used several resource management systems
Torque+MAUI (LIP-Coimbra); SGE (LIP-Lisbon, NCG-INGRID-PT)
There was no official gLite support for SGE
LIP joined a group of other organizations to develop gLite support
Flexible, Robust, Good documentation (behaves accordingly to the docs)‏
Configuration not too complex; The capabilities fits better to heterogeneous resources

27. 2727
Storage Systems
StoRM; SRM implementation that enables to gridify POSIX filesystems
Lustre is an High Performance filesystem mostly used in HPC
Creation of POSIX filesystems across multiple volumes in multiple servers.
A file can be stored entirely either in one server or stripped across multiple...
Lustre volumes are mounted in the clients and can be accessed locally (POSIX)
The filesystem is the data catalogue
Works reliably and performs well
Storage
gsiftp server
StoRM SRM
Local filesystem

28. Storage Systems

29. WLCG PT Network Activity

30. WLCG PT Tier-2 Performance

31. 3131
EGEE in Portugal
Portugal and Spain compose the EGEE Southwest federation
LIP coordinates EGEE activities in the country since EGEE-I
Infrastructure operations coordination
User and site support
Infrastructure services
Training
Security and Authentication
The Portuguese sites are:
LIP (Lisbon)‏; LIP (Coimbra)
NCG-INGRID-PT‏ (Lisbon)
UP (Porto)‏ (3 clusters)
DI-Uminho; UMinho-CP (Braga)‏
IEETA (Aveiro)‏
CFP-IST (Lisbon)‏
2500 Job slots
Hundreds of TB of Storage

32. EELA in Portugal 3232 EELA (January 2006 – December 2007)
E-Infraestructure shared between Europe and Latin America
EELA (January 2006 – December 2007)
Extending EGEE to Latin America
Pilot infrastructure; Training and dissemination
25 partners Europe and Latin America, Mexico, Brazil, Cuba, Chile, Venezuela, Argentina, Portugal (LIP), Italy, Spain, CERN, CLARA
EELA-II (April March 2010)
Aims at building a high capacity, production-quality, scalable Grid Facility, providing round-the-clock, worldwide access to distributed computing, storage and network resources needed by the wide spectrum of Applications from European - Latin American Scientific Collaborations
Univ of Porto, Univ of Minho, Univ of Aveiro
Sites: UMinho-CP, UPorto

33. Portuguese National Grid Iniciative
In Portugal the e-infrastructures area is under the responsibility of the Ministry of Science and Technology
The National Grid Initiative fits under the context of a larger initiative named “Ligar Portugal” which aims at foster the information society
LIP is responsible for the technical coordination of the initiative
Objectives
Promote modern citizenship; Ensure competitiveness in the communications market
Promote the information technologies uptake by private companies
Promote the scientific and technological development
Reinforce the national competence and capacity in the grid computing domain
Enable the use of grid computing for complex problem solving
Integrate Portugal in international grid computing infrastructures
Reinforce the multidisciplinary collaboration among research communities

34. Infrastructures and projects
3434
PT-NGI Infrastructure baseline
Creation of an autonomous grid infrastructure based on new and existing resources based on middleware widely in use gLite, Int.Eu.Grid
Deploy a main centre to provide grid core services and dedicated resources
The main node for grid computing (NCG) was built by a consortium of research organizations under the Portuguese NGI context:LIP, FCCN, LNEC
Excellent network connectivity:
FCCN national backbone
Géant PoP
Infrastructures and projects
INGRID
EGI
IBERGRID
LCG
...
Core resources
Main node etc
Existing resources
(EGEE, int.eu.grid, EELA,
INGRID projects...)‏
Other resources
Users

35. NCG Computing and Storage Resources
3535
NCG Computing and Storage Resources
Setup
Tape library LTO-4
Grid accessible data repositories
Hierarchical storage
Core grid services
Two blade centers, 192 CPU cores
Grid cluster
HTC and HPC blades
~ 1250 CPU cores for processing
Online grid storage
Server direct attached storage
~ 620TB raw + 70TB raw SAN
Local network
Core 10gigabit Ethernet
Non-blocking, wire-speed, low latency

36. PT-NGI Seed Infrastructure
3636
PT-NGI Seed Infrastructure
L2 connectivity between the 3 resource centres
Geant connectivity 3 Gbps for the whole cloud
LIP
Coimbra
Main
node
Lisboa
Layer 2
10 Gbps
3 Gbps
FCCN
Layer 3
~1250 CPU cores
~ 400 CPU cores
~184 CPU cores
Expand the infrastructure
Initially focus on existing resource centres and than expand to other sites at a later stage
Launch funding line for cluster integration:

37. BING - Brain Image Network Grid
3737
BING - Brain Image Network Grid
Functional magnetic resonance imaging (fMRI) is a medical imaging technique used to characterize brain physiological activity
Brain activity produces an hemodynamic response of the activation areas
Detects oxygen variations attached to blood where its being more used
fMRI can produce enormous quantity of data and demand large CPU usage
16 x 64 x 64 volumes each 3 seconds;
5min. = 100 volumes
Multi-voxel measures:
(16x64x64)x(16x64x64)=~4.3 million
Network
Data provider: 3T scanners in Coimbra
Data processing: Univ Aveiro; Univ Porto
Neuroscience clients; Engineering clients

38. European Grid Initiative
3838
European Grid Initiative
Portugal is member through UMIC
Participates in the EGI Policy Board
First national fee payed
EGI InSPIRE
Integrated Sustainable Pan-European
Infrastructure for Researchers in EuropeUnder EU negotiations
Main project for infrastructure coordination and operation
EGI InSPIRE international tasks
International bid for global tasks
Portugal and Spain in the middleware rollout coordination

39. European Grid Initiative
3939
European Grid Initiative
Establishing the criteria for the Relationships with software providers
Establishing the Roll-out process for the UMD middleware releases

40. 4040
IBERGRID
Is a common Portuguese/Spanish Iberian infrastructure
Grid computing
HPC, applications
Network
Applications
Volunteer computing
IBERGRID will provide an umbrella for an Iberian regional grid
Integrating Portuguese and Spanish NGI resources
Fully interoperable with EGI
Focus is now in the IBERGRID development as a requirement for a successful common participation in EGI
Towards a sustainable model but without loosing synergies and advantages
Portuguese grid initiative
Spanish grid initiative

41. IBERGRID Transition Plan
4141
IBERGRID Transition Plan
Portugal Tasks
Common VO management and coordination
Operations portal
Catalogues and services for the VOs
Certification Authority for Portugal (LIPCA)‏
Spain Tasks
Helpdesk (Request Tracker)‏
Monitoring and accounting
Infrastructure database (GOCDB/HGSM)‏
Certification Authority (PKIrisGrid)‏
Middleware security
Common Tasks
Core services and redundancy
Regional information system; Support groups
Training infrastructure; Operations coordination
Infrastructure security; Seed resources for new users

42. IBERGRID Infrastructure Today
4242
IBERGRID Infrastructure Today
Others
VOMS
INGRID / IBERGRID
ncg.ingrid.pt
LFC
MyProxy UI
PT Sites
ES Sites
TOP II PT
TOP II ES
XB, WMS, Gridway
XB, WMS, Gridway UI UI WN
lip.pt
ncg.ingrid.pt
ifca.es
cesga.es

43. http://wiki.ncg.ingrid.pt/ http://www.ibergrid.eu/2010/ 4343
Hepix Spring 2010
4343