Presentation is loading. Please wait.

Presentation is loading. Please wait.

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Hardware failures Wayne Salter on behalf of Olof B ärring.

Published byAdelia Cole Modified over 8 years ago

Similar presentations

Presentation on theme: "Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Hardware failures Wayne Salter on behalf of Olof B ärring."— Presentation transcript:

1 Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Hardware failures Wayne Salter on behalf of Olof B ärring

2 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Outline Failures –What fails? –How often? –When? Repairs –How? –By whom? –How quickly? Conclusions CERN IT facility

3 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF What fails? and how do we know? The only things we know for sure about hardware are: 1.It will fail 2.Some of it fails more often than other… disk drives for instance Monitoring failures –Disks: assume fail-stop but reality more complex –At CERN we base our decision on SMART counters and failed media scans Monitoring ‘repairs’ rather than ‘failures’: –Vendor tickets (~4k 2010-11) –Changes in serial numbers inventory (~10k 2010-11) CERN IT facility

4 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Failure space CERN IT by numbers (14/9/2011) CERN IT facility Number of systems8,792 Number of processors14,972 Memory modules55,729 Number of HDD's62,023 Number of RAID controllers3,607 Number of Fibre channel ports742 Number of 1G ports16,773 Number of 10G ports622

5 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF How often? Monitoring changes in serial numbers gives an idea CERN IT facility Bulk campaigns

6 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF How often? Monitoring changes in serial numbers gives an idea –Excluding campaigns ~170 disks /month (5 /day) CERN IT facility HDD failures/day:5 Hours/day:24  ~1 fail per 5hrs 64,000 drives in the centre  MTTF = 320,000 hrs (Spec: 1.2Mhrs)

7 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF When? Failure rates of hardware products typically follow a “bathtub curve” with high failure rates at the beginning (infant mortality) and the end (wear-out) of the lifecycle 1. CERN IT facility 1 http://www.usenix.org/events/fast07/tech/schroeder/schroeder.pdf

8 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF When? Process and categorize 2010-11 vendor calls according to ‘Warranty age’ when call was opened CERN IT facility 10x disks to CPU servers

9 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF When? Quarterly disk failure rate normalized to number of disks CERN IT facility Early failures (infant mortality)

10 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF When? Other failure types Swappable: RAM, PSU, BBU, BMC, … Complex repairs: cabling, backplane, main board, … no clue… CERN IT facility

11 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Repairs CERN IT facility Alarm Vendor call New sn: WD3342ABC

12 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF By who,? CERN IT facility Vendor

13 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF How quickly? Two contract types ‘Normal’ only used for CPU servers CERN IT facility TypeTime to interveneRepair time Normal24 working hours40 working hours Fast4 working hours12 working hours ~30%

14 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF CERN IT facility Ongoing Improvements Tracking changes to servers –Keep current tools that report HW info Controller 0: Vendor="Intel Corporation" Model="82801JI (ICH10 Family) SATA AHCI Controller" Location="/sys/devices/pci0000:00/0000:00:1f.2" BBU="None" Cache="None" Serial="None" Version="None" Driver="ahci" Type="sata” Controller 0 Port 0: Vendor="WDC" Model="WD1002FBYS-02A6B0" Size="953869" Serial="WD-WMATV4729249" Version="03.00C06" Device="sda” Controller 0 Port 1: Vendor="WDC" Model="WD1002FBYS-02A6B0" Size="953869" Serial="WD-WMATV8136033" Version="03.00C06" Device="sdb” Controller 0 Port 2: Vendor="WDC" Model="WD1002FBYS-02A6B0" Size="953869" Serial="WD-WMATV4713233" Version="03.00C06" Device="sdc” BIOS: Vendor="American Megatrends Inc." Version="080015 (07/20/2009)" smt="enabled” BMC: Vendor="Winbond" Model="IPMI 2.0" IPMI Version="2.0" MAC="00:00:00:00:00:0A" Serial="" Version="1.12” CPU 0: Vendor="GenuineIntel" Model="Intel(R) Xeon(R) CPU L5520 @ 2.27GHz" Cores="4" Speed="2270” CPU 1: Vendor="GenuineIntel" Model="Intel(R) Xeon(R) CPU L5520 @ 2.27GHz" Cores="4" Speed="2270” NIC 0: Vendor="Intel Corporation" Model="82574L Gigabit Network Connection" Location="/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0" MAC="00:00:00:00:00:00" Speed="1024000" Bus="pci" Media="ethernet" Version="1.9-0” NIC 1: Vendor="Intel Corporation" Model="82574L Gigabit Network Connection" Location="/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0" MAC="00:00:00:00:00:0F" Speed="1024000" Bus="pci" Media="ethernet" Version="1.9-0” RAM 0: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P2-DIMM1A" Type="Other" Serial=”00000001” RAM 1: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P2-DIMM1B" Type="Other" Serial="00000002” RAM 2: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P2-DIMM2A" Type="Other" Serial="00000003” RAM 3: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P2-DIMM2B" Type="Other" Serial="00000004” RAM 4: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P2-DIMM3A" Type="Other" Serial="00000005” RAM 5: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P2-DIMM3B" Type="Other" Serial="00000006” RAM 6: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P1-DIMM1A" Type="Other" Serial="00000007” RAM 7: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P1-DIMM1B" Type="Other" Serial="00000008” RAM 8: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P1-DIMM2A" Type="Other" Serial="00000009” RAM 9: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P1-DIMM2B" Type="Other" Serial="00000010” RAM 10: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P1-DIMM3A" Type="Other" Serial="00000011” RAM 11: Vendor="Hyundai" Model="HMT151R7BFR4C-G7" Size="4096" Data Rate="1066" Location="P1-DIMM3B" Type="Other” Serial="00000012” Serial: ”SDFGSDFG34DFGDFG345DFGDFG345" –Will store each server’s HW info as a document (HW inventory) –Key is unique id stored in the BMC when hardware is purchased –Change log, e.g. replaced parts, for each server –Goals: –Better accessibility and usability of data –Provide base for a more comprehensive HW inventory tool –Systematic tracking of parts replacement due to failure –Trending and potential action (e.g. #disk replacements in last month > X

15 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Conclusions Hardware fails –As expected –More often than expected MTTF ~320khours rather than 1.2Mhours –When expected: Effect of early failures (infant mortality) in first year No sign of wear-out at the end of the 3 years warranty Repairs are currently carried out by vendor –Missed repair targets in ~30% of cases –Looking at a different model… CERN IT facility

16 CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Questions? CERN IT facility

Download ppt "Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF Hardware failures Wayne Salter on behalf of Olof B ärring."

Similar presentations

IS 4506 Tuning and Monitoring Internet Information Server.

IS 4506 Tuning and Monitoring Internet Information Server.
PC Deployment, Maintenance, Replacement & Discard VUSD Technological Services Jerry Walkowiak, Director Chuck Boone, Systems Manager.

PC Deployment, Maintenance, Replacement & Discard VUSD Technological Services Jerry Walkowiak, Director Chuck Boone, Systems Manager.
Confidential Prepared by: System Sales PM Version: 1.0 Lean Design with Luxury Performance.

Confidential Prepared by: System Sales PM Version: 1.0 Lean Design with Luxury Performance.
Linux Clustering A way to supercomputing. What is Cluster? A group of individual computers bundled together using hardware and software in order to make.

Linux Clustering A way to supercomputing. What is Cluster? A group of individual computers bundled together using hardware and software in order to make.
Introduction to Information Technology: Your Digital World © 2013 The McGraw-Hill Companies, Inc. All rights reserved.Using Information Technology, 10e©

Introduction to Information Technology: Your Digital World © 2013 The McGraw-Hill Companies, Inc. All rights reserved.Using Information Technology, 10e©
Component 4: Introduction to Information and Computer Science Unit 1: Basic Computing Concepts, Including History Lecture 1 This material was developed.

Component 4: Introduction to Information and Computer Science Unit 1: Basic Computing Concepts, Including History Lecture 1 This material was developed.
Secondary Storage Unit 013: Systems Architecture Workbook: Secondary Storage 1G.

Secondary Storage Unit 013: Systems Architecture Workbook: Secondary Storage 1G.
1 RAL Status and Plans Carmine Cioffi Database Administrator and Developer 3D Workshop, CERN, 26-27 November 2009.

1 RAL Status and Plans Carmine Cioffi Database Administrator and Developer 3D Workshop, CERN, November 2009.
BNL Oracle database services status and future plans Carlos Fernando Gamboa RACF Facility Brookhaven National Laboratory, US Distributed Database Operations.

BNL Oracle database services status and future plans Carlos Fernando Gamboa RACF Facility Brookhaven National Laboratory, US Distributed Database Operations.
THE CPU Cpu brands AMD cpu Intel cpu By Nathan Ferguson.

THE CPU Cpu brands AMD cpu Intel cpu By Nathan Ferguson.
Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t CF CERN Business Continuity Overview Wayne Salter HEPiX April 2012.

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF CERN Business Continuity Overview Wayne Salter HEPiX April 2012.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Integrating Lemon Monitoring and Alarming System with the new CERN Agile Infrastructure.

CERN IT Department CH-1211 Genève 23 Switzerland t Integrating Lemon Monitoring and Alarming System with the new CERN Agile Infrastructure.
HEPiX 21/05/2014 Olof Bärring, Marco Guerri – CERN IT

HEPiX 21/05/2014 Olof Bärring, Marco Guerri – CERN IT
How Computers Work. A computer is a machine f or the storage and processing of information. Computers consist of hardware (what you can touch) and software.

How Computers Work. A computer is a machine f or the storage and processing of information. Computers consist of hardware (what you can touch) and software.
Basic Computer Structure and Knowledge Project Work.

Basic Computer Structure and Knowledge Project Work.
Fundamentals of Networking Discovery 1, Chapter 2 Operating Systems.

Fundamentals of Networking Discovery 1, Chapter 2 Operating Systems.
By: Dwayne Burl.  The Central Processing Unit (CPU) is responsible for interpreting and executing most of the commands from the computer's hardware and.

By: Dwayne Burl.  The Central Processing Unit (CPU) is responsible for interpreting and executing most of the commands from the computer's hardware and.
Chapter 6 Advanced Installation. Objectives  Describe the types and structure of SCSI devices  Explain the different levels of RAID and types of RAID.

Chapter 6 Advanced Installation. Objectives  Describe the types and structure of SCSI devices  Explain the different levels of RAID and types of RAID.
Day 10 Hardware Fault Tolerance RAID. High availability All servers should be on UPSs –2 Types Smart UPS –Serial cable connects from UPS to computer.

Day 10 Hardware Fault Tolerance RAID. High availability All servers should be on UPSs –2 Types Smart UPS –Serial cable connects from UPS to computer.
Module 9 Review Questions 1. The ability for a system to continue when a hardware failure occurs is A. Failure tolerance B. Hardware tolerance C. Fault.

Module 9 Review Questions 1. The ability for a system to continue when a hardware failure occurs is A. Failure tolerance B. Hardware tolerance C. Fault.

Similar presentations

Ads by Google