Intelligent Platform Management Interface (IPMI) Monitoring and Control Ian Collier RAL Tier1 Fabric Team July 2 nd 2009 HEPSYSMAN With apologies/thanks to Massimiliano Masi at CERN

IPMI at RAL Tier1 At RAL Tier1 we are just beginning rolling out significant use of IPMI In our new building were able to implement a separate management network for IPMI, APC PDUs etc

What and Why Started in 1998, IPMI is now at revision 2.0

What and Why Started in 1998, IPMI is now at revision 2.0 Is a standard accepted by DELL, IBM, SUN, INTEL and many others including SuperMicro of course

What and Why Started in 1998, IPMI is now at revision 2.0 Is a standard accepted by DELL, IBM, SUN, INTEL and many others including SuperMicro of course Goal 1: IPMI is a spec for monitoring and controlling the machine via special hardware, the Baseboard Management Controller, BMC

What and Why Started in 1998, IPMI is now at revision 2.0 Is a standard accepted by DELL, IBM, SUN, INTEL and many others including SuperMicro of course Goal 1: IPMI is a spec for monitoring and controlling the machine via special hardware, the Baseboard Management Controller, BMC Goal 2: Serial Over Lan (SOL). This is a method to redirect serial connections over an ethernet cable. Many cards now also provide KVM over LAN – eliminating need for expensive network KVMs!

What and Why? Major IPMI concepts: Sensors (Fans speed, CPU Temperature, voltage) Events (What the BMC should do when the CPU temperature reach 100 degrees? SNMP Traps) SDR (Sensor data repository, where the data are collected) SEL (System Event Log, a log of all critical situation) Session (Between the client and the BMC)

What and Why? SECURITY Can define users Can define privileges Can encrypt communication with BMC The security depends on the version of the specification

What and Why? SECURITY Can define users Can define privileges Can encrypt communication with BMC The security depends on the version of the specification Version 2.0: RMCP/RMCP+: based on RAKP messages (HMAC like protocol) Serial-Over-Lan is encrypted with RMCP+ only

Manufacturers provide GUIs

Open source tools OpenIPMI (ipmitool) Lmsensors Freeipmi (no drivers)

ipmitool sensor local output ~]# ipmitool sensor CPU Temp 1 | | degrees C | ok | na | na | na | | | CPU Temp 2 | | degrees C | ok | na | na | na | | | CPU Temp 3 | na | degrees C | na | na | na | na | | | CPU Temp 4 | na | degrees C | na | na | na | na | | | Sys Temp | | degrees C | ok | na | na | na | | | CPU1 Vcore | | Volts | ok | | | | | | CPU2 Vcore | | Volts | ok | | | | | | V | | Volts | ok | | | | | | V | | Volts | ok | | | | | | V | | Volts | ok | | | | | | V | | Volts | ok | | | | | | VSB | | Volts | ok | | | | | | VBAT | | Volts | ok | | | | | | Fan1 | | RPM | ok | | | | na | na | na Fan2 | | RPM | ok | | | | na | na | na Fan3 | | RPM | ok | | | | na | na | na Fan4 | | RPM | ok | | | | na | na | na Fan5 | | RPM | ok | | | | na | na | na Fan6 | | RPM | ok | | | | na | na | na Fan7 | | RPM | nr | | | | na | na | na Fan8 | | RPM | nr | | | | na | na | na Power Supply | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU0 Internal E | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU1 Internal E | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU Overheat | 0x0 | discrete | 0x0000| na | na | na | na | na | na Thermal Trip0 | 0x0 | discrete | 0x0000| na | na | na | na | na | na Thermal Trip1 | 0x0 | discrete | 0x0000| na | na | na | na | na | na

ipmitool sensor remote output # ipmitool -I lanplus -H U ADMIN \ sensor get'CPU1 Temp Password: Locating sensor record... Sensor ID : CPU1 Temp (0x0)Entity ID : 3.0 Sensor Type (Discrete): OEM reserved #c0

ipmitool sensor remote output # ipmitool -I lanplus -H U ADMIN \ sensor get'CPU1 Temp Password: Locating sensor record... Sensor ID : CPU1 Temp (0x0)Entity ID : 3.0 Sensor Type (Discrete): OEM reserved #c0 Note that the lanplus option encrypts communication including passwords

ipmitool remote power control # ipmitool -I lanplus -H U ADMIN \ power off Password: Chassis Power Control: Down/Off

ipmitool remote power control # ipmitool -I lanplus -H U ADMIN \ power off Password: Chassis Power Control: Down/Off # ipmitool -I lanplus -H U ADMIN \ power status Password: Chassis Power is off

ipmitool remote power control # ipmitool -I lanplus -H U ADMIN \ power off Password: Chassis Power Control: Down/Off # ipmitool -I lanplus -H U ADMIN \ power status Password: Chassis Power is off # ipmitool -I lanplus -H U ADMIN \ power on Password: Chassis Power Control: Up/On

ipmitool remote power control = less visits to the machine room!

ipmitool serial over lan # ipmitool -I lanplus -H –U \ ADMIN sol activate Password: [SOL Session operational. Use ~? for help] Scientific Linux SL release 4.6 (Beryllium) Kernel ELsmp on an i686 gdss328.gridpp.rl.ac.uk login:

ipmitool serial over lan = even less visits to the machine room!

ipmitool sensor local output ~]# ipmitool sensor CPU Temp 1 | | degrees C | ok | na | na | na | | | CPU Temp 2 | | degrees C | ok | na | na | na | | | CPU Temp 3 | na | degrees C | na | na | na | na | | | CPU Temp 4 | na | degrees C | na | na | na | na | | | Sys Temp | | degrees C | ok | na | na | na | | | CPU1 Vcore | | Volts | ok | | | | | | CPU2 Vcore | | Volts | ok | | | | | | V | | Volts | ok | | | | | | V | | Volts | ok | | | | | | V | | Volts | ok | | | | | | V | | Volts | ok | | | | | | VSB | | Volts | ok | | | | | | VBAT | | Volts | ok | | | | | | Fan1 | | RPM | ok | | | | na | na | na Fan2 | | RPM | ok | | | | na | na | na Fan3 | | RPM | ok | | | | na | na | na Fan4 | | RPM | ok | | | | na | na | na Fan5 | | RPM | ok | | | | na | na | na Fan6 | | RPM | ok | | | | na | na | na Fan7 | | RPM | nr | | | | na | na | na Fan8 | | RPM | nr | | | | na | na | na Power Supply | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU0 Internal E | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU1 Internal E | 0x0 | discrete | 0x0000| na | na | na | na | na | na CPU Overheat | 0x0 | discrete | 0x0000| na | na | na | na | na | na Thermal Trip0 | 0x0 | discrete | 0x0000| na | na | na | na | na | na Thermal Trip1 | 0x0 | discrete | 0x0000| na | na | na | na | na | na

Gathering IPMI metrics in Ganglia Perl script runs ipmitool sensor and pulls out non null values Metric labels vary with manufacturer and specific BMC Test deployment at: nding&c=Workers_SL4&h=lcg0954.gridpp.rl.ac.uk

Future Our new hardware has BMCs that support KVM over lan as well – with SuperMicros web interface The data gathered by Ganglia can be mined for very granular information about the conditions in the machine room – indicating airflow problems etc. Useful in diagnosing hardware problems after the event Configure snmp traps for alarms