Fermilab Site Report NLIT 2008 Dr. Mark O. Kaletka Computing Division Associate Head for Laboratory and Scientific Core Services

Scientific Computing IT Infrastructure Initiatives MIS Business Computing Cyber Security Facility Operations ITIL Service Desk Initiative

Scientific Computing Success of our current and future scientific programs relies on an adequate amount of computation, storage, networking, trigger, DAQ, control systems, databases and reliable IT services (for the program and the lab) AND on a continuous and evolving strategy for Advanced Computing because of – Scale of computing required for experiment data and theoretical science – Unprecedented volumes of data we are dealing with (~9 Petabytes of scientific data in Fermilab storage systems) – Global collaborations and global involvement in computing – Requirement to keep up with technology and to find innovative, affordable solutions which are sensitive to Space, power and cooling costs Operational loads on people Performance and reliability which scales 3

Visualizing the data volume 4 If each byte were a grain of sand, 8.79  bytes of data now on tape could build the Great Pyramid of Khufu seven times over.

FermiGrid – a Campus Grid Fermilab has adopted a strategy of placing all of its production computing resources in a Grid "meta-facility" called FermiGrid. Optimizes use of resources and allows Fermilab to: – Insure that the large experiments have first priority usage of dedicated resources purchased on their behalf – Allow opportunistic use of all resources by Fermilab Experiment and Open Science Grid Virtual Organizations (VOs) – Fully support Open Science Grid and the LHC Computing Grid and gain positive benefit from this infrastructure in the US and Europe – Provide shared, high availability, robust grid services for all stakeholders at Fermilab (including CMS) It has been highly successful and almost all Fermilab scientific programs and experiments are now able to benefit from use of the broader Grid (if they need to). 5

IT Infrastructure Our IT infrastructure relies on a combination of open- source, commercial and local development. – Replacing our older and more fragile systems Second phase of a pilot to move & calendar to Exchange Readying a pilot to move collaborative web to Sharepoint HelpDesk ticketing will be upgaded to Remedy 7 Monitoring & alarms evolving – Zabbix? HPOpenView? – Re-evaluating open source workflow applications Currently using open-source Plone & AlphaFlow These are candidate for Sharepoint – Several very successful open source applications DocDB document management database CERN Indico meeting/conference software

IT Infrastructure April, 2007, the number of systems running Fermilab Scientific Linux exceeded 30,000 in less than 3 years. Software inventory was exercised by the Adobe license review – Improved the number of Windows systems reporting to SMS – Deployed the QMX SMS client to 300 Mac’s in two weeks – Open-source OCS Inventory client bundled with Fermilab Scientific Linux – not integrated with SMS Launched a VoIP & unified messaging pilot project Completed re-design & modernization of Computing web pages

MIS Business Computing MIS computing builds on COTS components: – Oracle E-Business Suite (financial & material management) – PeopleSoft (HR management) – Sunflower (asset management) – DelTek Cost & Scheduling (project management) Completing a platform migration of business & administrative systems from HP/Compaq Alphaservers (Tru64 Unix) to HP/Proliant servers (RHEL) Completed the first phase of implementation of Oracle's Time & Labor module for approximately 250 FTEs. – Remaining phases will occur over FY08-9 Investigating Oracle Identity Management Suite for identity and access management

Cyber Security - Authentication Pushing applications to comply with our authentication strategy: – Use any of three central authentication methods Kerberos credentials (MIT or Windows domain) Certificates derived from Kerberos credentials Central LDAP service (Windows domain) – Not quite SSO but reduces the number of passwords to two – Also updating our Kerberos infrastructure

Cyber Security - Scanning Continue deployment & development of aggressive automated scanning & blocking infrastructure Continue deployment & development of self assessment tools for system administrators: – ScanMeNow: scans for vulnerabilities without a standalone Nessus client – PortScanMeNow: quickly port scans your system – SurfMeNow: connects to a FNAL URL from outside to test security controls – CertTestNow: verifies if your browser certificates are loaded properly. – ExploitMeNow: verifies if your system is vulnerable to certain client side exploits – DefendMeNow: monitors ports to detect and block greedy scanners

Facility Operations Operating three data centers with eight computer rooms. – Over 30,000 square feet of raised floor; 3.5mW footprint – 6500 multi-cpu, multi-core computers – more than 1 PetaByte of disk and 9 PetaBytes of tape Commissioned Computer Room B in January – 2500 square feet; 84 10kW racks of HPC computing Computer Room C will come online in July – 2500 square feet; 64 14kW racks of HPC computing Feynman Computer Center – Tier 2 data center with UPS and standby generator power – Supports core services, networking, business, DBs – > 99.95% uptime at the Tier 2 data center with 4.12 calculated hours downtime in 2007

Space, power, cooling

ITIL Service Desk Initiative Completed a HelpDesk assessment in late Strategic decision to move to the ITIL v3 framework & (perhaps) ISO certification Consolidation of desktop & helpdesk groups, with Particle Physics staff moving to Computing, forms the core of the new Service Desk Bringing in a Project Manager for a HelpDesk to ITIL Service Desk transition project – Definition of the staffing and organization of the Service Desk – Implementation of Incident and Problem Management ITIL practices – Implementation of service desk tools – Training and mentoring of Service Desk staff and management