1. Computing at LIP Jorge Gomes LIP Laboratório de Instrumentação e Física Experimental de Partículas LIP Workshop 2008, Luso, 12 January 2008

2. Summary Computing infrastructure –Infrastructure –Network –Services –Future GRID computing –Projects –Tier-2 –National grid initiative –Future More details in the following presentations: Lisbon farm Gonçalo Borges Data storage Mario David Coimbra infrastructure Miguel Oliveira

3. Computing

4. Processing and storage capacity Lisbon –Farm 88 CPU COREs AMD Opteron and Intel Xeon 2 to 4 CPU COREs per machine 1 to 2GB RAM per CPU CORE Gigabit Ethernet LRMS: Sun Grid Engine (SGE) –Storage dcache Coimbra –Farm 84 Workstations Pentium IV CPUs 2.2 to 3.0GHz 1 CPU per machine 1 to 2GB RAM per CPU Fast Ethernet LRMS: Torque/MAUI –Storage dpm

5. Infrastructure upgrades in 2007 Lisbon –New AVAC unit with 22KW capacity (total installed now 66KW close control) –Reorganization of the farm racks –Tape robot for backups LTO-3 (28TB) Coimbra –Cluster entered into operation (hosted at the CFC datacenter) –Installation of storage capacity –New 40KVA UPS

6. Internet connectivity Improvement of network connectivity in 2007 Lisbon: –New fibre link LIP FCCN Fibre rented from PT Max bandwidth 10 Gbps –Agreed capacity with FCCN 1 Gigabit/s full duplex GBICs Gigabit Ethernet LX 995Mbps academic (Geant tagging) + 5Mbps commercial –Router upgrade (supervisor and interfaces) Coimbra: –The LIP farm shares the CFC link where the cluster is hosted CFC connects directly to CIUC and shares the UC link

7. Internet usage at LIP-Lisbon Statistics collected by FCCN The usage is mainly grid file transfer Over last year Last month FCCN atinge os 2Gbps de tráfego na sua ligação internacional, dos quais mais de 800Mbps são de tráfego de investigação - Outubro, 2007 News section in www.fccn.ptwww.fccn.pt

8. FCCN international link: usage Statistics from FCCN Current capacity: 2.5Gbps Impact of LIP ~ 10 a 20% Over last year Last month

9. Other services Other services: –Mail, web, DNS, NTP, etc... –Printing –NFS home dirs and interactive services –Group servers –Centralized management for workstations and servers Repositories Network installations and upgrades (quattor) Linux (SL, Fedora,...), FreeBSD, Windows, Tru64 –Installation support for portables –LAN and WAN network management –Backups –Authentication, security and firewalls –Management of purchases and RMAs

10. Old services... There is a considerable number of old systems and services being maintained –Alphas digital UNIX / Tru64 –Several old storage servers based on classic PCs –ISDN connectivity The closing of these services is desirable... –Management costs –Power and cooling consumption –Space –Limited capacity, performance and reliability The room must be optimized

11. Future: computing and storage Lisbon –20 servers (~152 CPU COREs) two x quad core CPUs 16GB – 24GB RAM 750GB Dual gigabit + IPMI mgmt –80TB storage (RAID SATA disks) Coimbra –12 servers (~96 CPU COREs) two x quad core CPUs 16GB – 24GB RAM 750GB Dual gigabit + IPMI mgmt –40TB storage (RAID SATA disks)

12. Future: infrastructures Lisbon –Floor reinforcement –Cooling optimization Optimize cold air delivery Close the hot corridor –Replace some racks –Optimize the room Coimbra –Reinforce AVAC + 40KW –Several civil engineering works/CFC –Fire detection and extinguishing –New racks

13. Future: network Upgrade the cluster LANs in Lisbon and Coimbra –New core switches –Likely: modular switches wire speed non-blocking c300 from “Force10 networks” –Lisbon Switch Layer3 4x 48 interfaces Gigabit + 4x 4 interfaces 10gigabit Second switch to interconnect the floors –Coimbra Switch Layer3 3x 48 interfaces Gigabit Second switch to –Implement direct connectivity to the LIP Coimbra LAN –Connect the farm to the FCCN network bypassing CFC FCCN will provide network connectivity for grid computing: –Possibly enabling direct connectivity between Lisbon and Coimbra clusters –Possibly enabling more bandwidth at both sites

14. GRID computing

15. LIP in international grid computing projects DataGridCrossGridLCGEGEE-IEELAEGEE-IIInt.Eu.Grid 2001 2002 2003 2004 2005 2006 2007 2008

16. LHC Computing Grid The LHC Computing Grid develops and operates the data storage and processing infrastructure for the LHC community The Portuguese government signed the LCG MoU In this context a Portuguese Tier-2 for LCG is being deployed –Resource centre providing processing and storage capacity integrated into the LHC Computing Grid The Portuguese Tier-2 will have 3 centers: –LIP-Lisbon Hosted at the LIP datacenter in Lisbon –LIP-Coimbra Hosted in partnership with CFC CFC operates the largest HPC cluster in Portugal – “Milipeia” –Main node for grid computing (national grid initiative) Being built at LNEC

17. The LCG depends on two major grid computing infrastructures... The biggest grid computing infrastructure worlwide LCG Infrastructure

18. 240 sites 45 countries 41,000 CPUs 5 PetaBytes >10,000 users >150 VOs >100,000 jobs/day Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences … 91 partners in 32 countries 25 collaborating projects

19. EGEE in Portugal Portugal and Spain constitute the EGEE Southwest federation LIP coordinates EGEE in Portugal Centres in Portugal: –LIP Lisboa (core services, production and pre-prod) Coimbra –Univ Lusiada Famalicão –Univ Porto Porto (3 clusters) –Univ Minho Braga –IEETA Aveiro (pre-prod)

20. The Southwest federation Total CPU COREs in the federation > 1400 Total CPU COREs in Portugal ~ 240

21. CPU usage in the federation

22. The SWE federation usage SWE federation usage: –1.863.000 jobs between Nov-2006 and Out-2007 –301.400 days normalized CPU time –257.000 days absolute CPU time –Equivalent to > 700 machines working 24h/day

23. Portugal in EGEE Production infrastructure usage

24. LIP federated Tier-2 LCG Tier-2 reliability report

25. Int.EU.Grid infrastructure Grid infrastructure focused on: –Parallel processing –Interactivity –Extending the glite MW 12 centers 7 countries ~ 750 COREs Virtual Organizations: –ifusion –ienvmod –iusct –ibrain –ihep –iplanck –iwien2k –icompchem Grid Operations Management

26. EELA E-Infrastructure shared between Europe and Latin America EU funded project coordinated by CIEMAT January 2006 – December 2007 25 partners in Europe and Latin America Mexico, Brazil, Cuba, Chile, Venezuela, Argentina, Portugal, Italy e Spain, CERN, CLARA EGEE extension to Latin America Pilot grid infrastructure Dissemination and training LIP responsible for the authentication and VO management task Coordinate the deployment of grid certification authorities Brazil, Argentina, Chile, Mexico, catch-all Virtual organizations management and authorization File catalogue core services

27. LIP grid core services LIP operates grid core services for several infrastructures –EGEE –Int.Eu.Grid –EELA These services include: –Resource Brokers / CrossBrokers –Information Indexes –RAS servers –LFC servers –VOMS servers –Myproxy server Hosted at: –FCCN datacenter –LIP-Lisbon datacenter

28. LIP Certification Authority The LIP CA is the grid certification authority for Portugal –Accredited by the International Grid Trust Federation (IGTF) –Registered at the TERENA TACAR trust anchor The LIP CA is member of the EUgridPMA Registration authorities at: –Centro de Física de Plasmas (IST) –Centro de Sistemas Inteligentes (UALG) –FCT/UNL –Instituto de Engenharia Electrónica e Telemática de Aveiro (UA) –LIP Lisboa –LIP Coimbra (UC) –Universidade Lusíada (Famalicão) –Universidade Autónoma de Lisboa –Universidade do Minho –Universidade do Porto http://ca.lip.pt

29. National grid initiative Officially launched in April of 2006 by the Portuguese Ministry of Science –Support the development of grid infrastructures for complex problem solving –Development of competences –Integrate Portugal in major international grid infrastructures LIP participates in the coordination of the initiative Activities of the initiative –Funding: 15 pilot projects (1.500.000 €) –Networks for grid computing –Infrastructures: INGRID+ (national grid infrastructure based on EGEE) Main node for grid computing

30. International network connectivity International connectivity being deeply changed with new links being deployed –North: Minho - Galicia –South: through Spanish Estremadura Objectives –Better geant connectivity (higher bandwidth) –Redundancy between both countries –Grid computing support –Have dedicated fibres Grid computing connectivity –Dedicated bandwidth for grid computing –Separate commodity and grid network traffic

31. Main node for grid computing Deployment of the Portuguese main node for grid computing –First step towards INGRID+ –The project began in the summer of 2007 –Expected to become operational in September of 2008 –Consortium: LIP, FCCN and LNEC Datacenter dedicated to grid computing –Host the core grid services for Portugal –Host major computing and storage resources –Host resources from other organizations Users –National grid initiative projects –National Tier-2 for the LCG –Projects in the context of IBERGRID –Portuguese researchers with high computing requirements

32. Main node for grid computing GRID datacenter –Being built at LNEC Civil construction adjudicated last week Cooling and power project in the last phase –Located near the FCCN NOC Direct connectivity to the FCCN backbone and Geant point of presence –Adequate power and cooling infrastructure for grid computing –Budget 3.200.000€ Host: –Core GRID services INGRID+ and other projects –GRID computing cluster > 500 CPU COREs and ~ 200TB in the end of 2008 Up to ~ 2000 CPU COREs and ~ 500TB of storage Resources managed by LIP –FCCN nearline storage robot for data repositories ~ 1PB –LNEC GRID cluster –Host of other grid resources

33. IBERGRID –A common Iberian grid infrastructure is being prepared: In the context of agreements between the Portuguese and Spanish governments Sharing of resources between Portugal and Spain –Main areas: Networking, grid, supercomputing and applications –Based on and profiting from the current collaboration in the framework of European projects: EGEE, int.eu.grid, EELA –LIP is deeply involved: Coordination of the initiative Infrastructure coordination for Portugal –Conferences: 1st conference took place in Santiago de Compostela in May of 2007 2nd conference will take place at the University of Porto

34. EGEE-III EGEE-III project approved by the EU in the end of 2007 –Expand the EGEE infrastructure More resources More users and communities –Prepare the migration towards a future European Infrastructure (EGI) based on the national grid initiatives (NGIs) Will start after EGEE-II (two years duration) Total budget 68.900.000 € –EU contribution 36.250.000 € –EU contribution for LIP 300.000 €

35. EGI Design Study proposal approved to the European Commission (started 1 st September’07) Supported by 31 National Grid Initiatives (NGIs) 2 year project to prepare the setup and operation of a new organizational model for a sustainable pan- European grid infrastructure Federated model bringing together NGIs to build a European organisation Well defined, complimentary responsibilities between NGIs and EGI http://www.eu-egi.org EGI workshop held during EGEE’07 conference last week in Budapest Prof. Gaspar Barreira appointed spokesman of the advisory committee

36. Thank you...