CHEP 97 Berlin, April 97Frédéric Hemmer1/11 Managing Outsourcing Contracts for System Administration Frédéric Hemmer CERN

CHEP 97 Berlin, April 97Frédéric Hemmer2/11 Capacity explosion Physics Data Processing capacity Benefits Cheaper and more affordable computing cycles Large data stores Services dedicated to experiments Large increase of computer nodes (20 in 1990, 300 now) Over 1000 disks î need for more manpower to perform the system administration (est.) Year CPU Capacity - CERN Units Disk Storage - GigaBytes

CHEP 97 Berlin, April 97Frédéric Hemmer3/11 Manpower evolution Physics Data Processing manpower system administration user support general administration planning etc. Total decreased by 25 % Sys. Admin. decreased by 40 % (est.) Year man-years per year TotalSystems Administration

CHEP 97 Berlin, April 97Frédéric Hemmer4/11 Outsourcing Very wide definition Total outsourcing of a service Total transfer of ownership (equipment, infrastructure and staff) Long term mega-deals Temporary labour contracts Selective sourcing (scope, size and duration) different activities lead to different management attention create constant competition more flexibility to adapt to changes î We try to follow this approach

CHEP 97 Berlin, April 97Frédéric Hemmer5/11 “Best Practices in Information Technology Sourcing” M.C. Lacity, L.P. Wilcocks Executive Research Briefing, Oxford Institute of Information Management, Templeton College, University of Oxford 1996

CHEP 97 Berlin, April 97Frédéric Hemmer6/11 Our approach Short term contracts System administration only CERN retains planning, HW purchase/maintenance, user support, choice of technology and architecture SLA’s defining quality measures At least two contracts (offset in time) î allows for regular retendering (competition)

CHEP 97 Berlin, April 97Frédéric Hemmer7/11 Our experience (1) Initially tried to use temporary labour Difficulty in finding experience candidates CERN policy evolved to service contracts Difficulty to express specifications of sys. admin. Outsourced the task of writing the specifications Long iterative process Vendor and customer have different goals ! Contract covering 28 HP and 27 DEC starting Jan. 96

CHEP 97 Berlin, April 97Frédéric Hemmer8/11 Our experience (2) Documents Service Definition Document (Scope, communication and escalation procedures) Service Level Agreement (responsibilities, tasks and performance metrics) Specific Sys. Admin. documents for each system System Administration tasks installation, configuration changes, reactive and proactive problem management, development of service quality and performance measurements Aggressive timescale 2 months for sys. admin. guides, take over one month later

CHEP 97 Berlin, April 97Frédéric Hemmer9/11 Our experience (3) May review Objectives met No degradation in service Documentation very good Improvements needed Communication needs to be more formal î Mastersoft’s OSM Inventory, SLA, communication, reporting, quality measurement System managers showed reluctance to give away “their” systems (lack of confidence, faster to fix it now). Not solved yet. Decided to expand 24/hours day later expand to all DEC, HP, SUN and SGI equipment

CHEP 97 Berlin, April 97Frédéric Hemmer10/11 Potential Problem areas Call Management System Installed in Autumn, not used before March No PC for our OPS manager Lack of momentum in contract follow-up (increased responsibilities of the team leader, staffing problems) î no quality or performance measures Coverage and end of 96 shutdown Contract excluded this period Period of intense activity as the computers are down î return of control to CERN staff Contractor’s staff stability î unable to test whether procedures and documentation are enough

CHEP 97 Berlin, April 97Frédéric Hemmer11/11 Conclusions Globally positive Procedures well documented Very few problems/mistakes Not too many delays Clear cost attributable to system administration Scaling up no longer limited by CERN staffing level Improvement needed More involvement from CERN needed in contract follow-up The performance metrics need to be gathered and analysed î We hope to assess performance by the summer and be ready for tendering at the end of the year.