1. 20th October 2003Hepix Vancouver - Oxford Site Report1 Oxford University Particle Physics Site Report Pete Gronbech Systems Manager

2. 20th October 2003Hepix Vancouver - Oxford Site Report2

3. 20th October 2003Hepix Vancouver - Oxford Site Report3 Central Physics Computing Services l E-Mail hubs n In last year 2.7M messages were relayed (78GB), 0.8M from Physics systems. A further, 2.8M rejected as spam. Last month 345K rejected, 208K delivered. n Anti-virus and anti-spam measures increasingly important in email hubs. Some spam inevitably leaks through and clients need to deal with this in a more intelligent way. l Windows Terminal Servers n Use is still increasing. n Retired NT4 based service, Now Win2k and 2003. n Introduced an 8 CPU server (TermservMP). Much more powerful system but still awaiting updated versions of some applications which will run properly on OS. l Web / Database n New web server (Windows 2003) in service. Some initial problems with migrating the permissions from the old service. n New web applications for lecture lists, Computer inventory n Other databases for admissions and finals l Exchange Servers n Running two new servers using Exchange 2003 running on Windows server 2003. Default for new accounts. Much better Web interface, support for mobile devices and for tunnelling through firewalls. Existing mailboxes will be migrated soon.

4. 20th October 2003Hepix Vancouver - Oxford Site Report4

5. 20th October 2003Hepix Vancouver - Oxford Site Report5 Particle Physics Strategy The Server / Desktop Divide Win 2K PC Linux System Desktops Servers General Purpose Unix Server Group DAQ Systems Mail Server Web Server Windows File Server Win 2K PC Win XP PC Approx 200 Windows 2000 Desktop PC’s with Exceed used to access central Linux systems

6. 20th October 2003Hepix Vancouver - Oxford Site Report6 Windows Status l Migration to Windows 2000 domain nearly complete for PP users and their computers. l Windows XP pro is default OS for new desktops and laptops. l We now have to expect routine reboots of desktops to apply security patches. Give notice whenever possible. Grant year Windows Desktops Installed Minimum Spec Maximum Spec 98/9925P2/350P3/450 99/0034P3/450P3/650 00/0122P3/733P3/866 01/0278P3/1000P4/1800 02/0349P4/2.0GHzP4/2.6GHz

7. 20th October 2003Hepix Vancouver - Oxford Site Report7 Migration to Linux l Central Unix systems are Linux based n Red Hat Linux 7.3 is becoming the standard n Treat Linux as just another Unix and hence a server OS to be managed centrally. n Wish to avoid badly managed desktop PC’s running Linux. l Linux based file server (April 2002) l Digital Unix and VMS services were closed in August 2002 l General purpose Linux server installed August 2002 l Small batch farm installed Feb 2003

8. 20th October 2003Hepix Vancouver - Oxford Site Report8 PP Computation - Linux l Strategy of Windows desktop with Linux Computing Servers accessed by X windows. l General purpose interactive login provided by pplxgen and pplx2, file server pplxfs1. l Have been running an 8 cpu general purpose batch farm Since Feb 2003. We are installing a further 8 cpus to deal with increased load. Major users are LHCb-DC, Harp, Licas. Jobs submitted from pplxgen, this provides fast turn round for development. Large numbers of jobs can be submitted to the csf at RAL (in the future these will go to the grid). l Installed several GRID and Data Challenge machines l CDF – JIF Second round of procurement purchased a farm containing 20 cpus and 7.5 TB of disk space. Installed Jan 2003. This compliments the 8 way IBM server used for program development. l DAQ systems for Cresst, Minos, Atlas and Harp. l Security incident in April emphasises need to keep up-to-date with patches and kernel versions. l What version of Linux to run ? Currently almost all 7.3 but Red Hat’s proposal to have limited support & hiving off free releases to fedora project has become a problem.

9. 20th October 2003Hepix Vancouver - Oxford Site Report9 pplx1morpheus pplxfs1pplxgen pplx2 1Gb/s ppcresst1ppcresst2 ppatlas1atlassbc ppminos1ppminos2 gridtbwn01 pptb01 pptb02 Grid Development pplx3 (SNO) ppnt117 (HARP) CDF minos DAQ Atlas DAQ cresst DAQ General Purpose Systems tblcfgsece RH 7.3 Fermi 7.3.1 RH 7.3 RH 6.2 RH 7.1 RH 7.3 RH 6.2 RH 7.3 Fermi 7.3.1 PBS Batch Farm 4*Dual 2.4GHz systems RH 7.3 Autumn 2002 4*Dual 2.4GHz systems RH 7.3 Autumn 2003 matrix 7.3.1

10. General Purpose Linux pplx2 Dual 450MHz Pentium II 1024MB RAM (1999) Early Linux Systems pplx1 P4 Xeon 2.4GHz 2GB RAM (1998) pplx3 Dual 800MHz Pentium II 512MB RAM (2000) CDF group system runs Fermi 7.3.1 SNO group system runs Red Hat 6.2

11. 20th October 2003Hepix Vancouver - Oxford Site Report11 Zero - D X- 3i SCSI -IDE RAID 12 * 160GB Maxtor Drives Supplied by Compusys This proved to be a disaster and was rejected in favour of bare scsi disks which we internally mounted in our rack mounted file server

12. 20th October 2003Hepix Vancouver - Oxford Site Report12 The new (April 2002) Linux File Server: pplxfs1 8*146GB SCSI disks Dual 1GHz PIII, 1GB RAM

13. 20th October 2003Hepix Vancouver - Oxford Site Report13 General Purpose Linux Server : pplxgen pplxgen is a Dual 2.2GHz Pentium 4 Xeon based system with 2GB ram. It is running Red Hat 7.3 It was brought on line at the end of August 2002 to share the load with pplx2 as users migrated off al1 (the Digital Unix Server)

14. 20th October 2003Hepix Vancouver - Oxford Site Report14 PP batch farm running Red Hat 7.3 with Open PBS can be seen below pplxgen This service became fully operational in Feb 2003. Additional 4 worker nodes to be installed this month. (October 2003)

15. 20th October 2003Hepix Vancouver - Oxford Site Report15 Power Cut http://www-pnp.physics.ox.ac.uk/ganglia-webfrontend-2.5.4/

16. 20th October 2003Hepix Vancouver - Oxford Site Report16 CDF Linux Systems Morpheus is an IBM x370 8 way SMP 700MHz Xeon with 8GB RAM and 1TB Fibre Channel disks Installed August 2001 Purchased as part of a JIF grant for the CDF group Runs Fermi Red Hat 7.3.1 Will use CDF software developed at Fermilab and here to process data from the CDF experiment.

17. Second round of CDF JIF tender: Dell Cluster - MATRIX 10 Dual 2.4GHz P4 Xeon servers running Fermi Linux 7.3.1 and SCALI cluster software. Installed December 2002

18. 20th October 2003Hepix Vancouver - Oxford Site Report18 Approx 7.5 TB for SCSI RAID 5 disks are attached to the master node. Each shelf holds 14 * 146GB disks. These are shared via NFS with the worker nodes. OpenPBS batch queuing software is used. CDF Linux Systems - MATRIX

19. 20th October 2003Hepix Vancouver - Oxford Site Report19 Plenty of space in the second rack for expansion of the cluster. Additional Disk Shelf with 14*146GB plus two extra nodes will shortly be ordered. (Autumn 2003)

20. 20th October 2003Hepix Vancouver - Oxford Site Report20 Grid development systems. EDG Test bed setup, currently 2.0.3

21. 20th October 2003Hepix Vancouver - Oxford Site Report21 Tape Backup is provided by a Qualstar TLS4480 tape robot with 80 slots and Dual Sony AIT3 drives. Each tape can hold 100GB of data. Installed Jan 2002. Netvault Software from BakBone is used, running on morpheus, for backup of both cdf and particle physics systems. Main userdisks backed up every weekday night data disks not generally backed up BUT weekly backups to OUCS HFS service provide some security.

22. Network Access Campus Backbone Router Super Janet 4 2.4Gb/s with Super Janet 4 OUCS Firewall depts Physics Firewall Physics Backbone Router 100Mb/s 1Gb/s 100Mb/s 1Gb/s Backbone Edge Router depts 100Mb/s depts 100Mb/s Backbone Edge Router 1Gb/s

23. Physics Backbone Upgrade to Gigabit Autumn 2002 desktop Server switch Physics Firewall Physics Backbone Router 1Gb/s 100Mb/s Particle Physics desktop 100Mb/s 1Gb/s 100Mb/s Clarendon Lab 1Gb/s Linux Server Win 2k Server Astro 1Gb/s Theory 1Gb/s Atmos 1Gb/s

24. 20th October 2003Hepix Vancouver - Oxford Site Report24 Network l Gigabit network installed for the physics backbone. l Most PP servers are now interconnected via gigabit. l Many switches have been upgraded to provide 100 mpbs to almost every port with gigabit uplinks to the core network. l Connection to campus remains at 100 mbps, campus upgrade to 10Gbps core not expected till end of 2004. l Virtual Private Network (VPN) server getting increased usage, overcomes problems getting some protocols through firewalls. Allows authorised users to get into the Physics network from remote sites, but it has its own security risks…..

25. 20th October 2003Hepix Vancouver - Oxford Site Report25 Network Security l Constantly under threat from worms and viruses. Boundary Firewall’s don’t solve the problem entirely as people bring infections in on laptops. l New firewall based on stateful inspection. Policy is now `default closed`. Some teething problems as we learnt what protocols were required but there has been a very significant improvement in security. l Main firewall passes average 5.8GB/hour (link saturates at peak). Rejects 26,000 connection per hour (7 per second). Mischievous connects rejected 1500/hour, one every 2.5 secs. During blaster worm this reached 80/sec. l Additional firewalls installed to protect the Atlas construction area and to protect us from attacks via dialup or VPN. l Need better control over how laptops access our network. Migrating to a new Network Address Translation system so all portables connect through a managed `gateway`. l Have made it easier to keep Anti-Virus software uptodate via simply connecting to a web page. Important that everyone managing their own machines takes advantage of this. Very useful for both laptops and home systems (see http://www.physics.ox.ac.uk/sophos) l Keeping OS’s patched is a major challenge. Easier when machines are all inside one management domain but is still very time consuming. Must compare to perhaps 1-few man months of IT support staff effort to clean out a successful worm from the network.

26. 20th October 2003Hepix Vancouver - Oxford Site Report26 Goals for 2003/4 (Computing) l Continue to improve Network security n Need better tools for OS patch management n Need users to help with their private laptops –Use automatic updates (e.g. Windows Update) –Update Antivirus software regularly n Segment the network by levels of trust n All the above without adding an enormous management overhead ! l Reduce number of OS’s n Remove last NT4 machines and exchange 5.5 n Digital Unix and VMS very nearly gone. n Getting closer to standardising on RH 7.3 especially as the EDG software is now heading that way. l Still finding it very hard to support laptops but now have a standard clone and recommend IBM laptops. l What version of Linux to run ? Currently almost all 7.3 but Red Hat’s proposal to have limited support & hiving off free releases to fedora project will become a problem. l Looking into Single Sign On for PP systems