IPMI: New MMC firmware IN2P3: DAQ working group (19/06/15) 19/06/2015IN2P3: DAQ working group (06/19)1 PH-ESE-BE collaboration Vincent Bobillier, Stefan.

IPMI: New MMC firmware IN2P3: DAQ working group (19/06/15) 19/06/2015IN2P3: DAQ working group (06/19)1 PH-ESE-BE collaboration Vincent Bobillier, Stefan Haas, Markus Joos, Julian Mendez, Sylvain Mico and Francois Vasey

IPMI: New MMC firmware Outline  PH-ESE-BE: xTCA evaluation project  Hardware Platform Management overview  CERN MMC overview  New MMC firmware  Summary 19/06/2015IN2P3: DAQ working group (06/19)2

IPMI: New MMC firmware PH-ESE-BE: xTCA evaluation project  Major upgrades of the LHC experiments are foreseen over >10 years  Aligned with LHC upgrade long shutdowns: 2013/14, 2018, 2023  Off-detector electronics of the LHC experiments mostly based on VME  Working very reliably  “Old” technology and doubts about long-term availability  Experiments planning to use MTCA & ATCA for upgrades of their back-end electronics  MTCA (and ATCA): CMS  ATCA: ATLAS  xTCA advantages  Choice of form factors  Backplane bandwidth and protocols  Cooling and power supply  Redundancy (PSU, cooling, Mgmt)  Infrastructure monitoring features  Interoperability  MTCA and ATCA developments already on-going at CERN and collaborating institutes  Evaluation project started in 2011 19/06/2015IN2P3: DAQ working group (06/19)4

IPMI: New MMC firmware PH-ESE-BE: xTCA evaluation project (MicroTCA) 19/06/2015IN2P3: DAQ working group (06/19)5 MicroTCA Crates ELMA MTCA.4 043-012 Schroff MTCA.4 + AC/DC CM100 Power Modules (PM) NAT DC840 840W Vadatech UTC010 792W Vadatech UTC001 NATKontron AM4904 AMCs ELMA Load BoardProcessor Kontron AM5030 ESD ADIO24 Processor CCT AM310 Wiener AC/DC 1000W Schroff MTCA.4 11890 ELMA MTCA.4 045-821 Telkoor AC/DC 600W Vadatech UTC020 936W NAT AC/DC AC600 600W

IPMI: New MMC firmware PH-ESE-BE: xTCA evaluation project (AdvanceTCA) 19/06/2015IN2P3: DAQ working group (06/19)6 Commercial ATCA Crates Schroff 14-slot 13U ATCA 11596-150 ASIS 14-slot 13U ATCA 144D422 Comtel 14 slot ELMA 14 slot Custom ATCA Crate Schroff 2-slot ATCA 11990-705 ASIS Vertical cooling

IPMI: New MMC firmware PH-ESE-BE: xTCA evaluation project (Summary)  MicroTCA  Equipment evaluated  Specifications written for DC/DC power modules and shelves  Price inquiry finished  Pre-series of 3 units for final qualification purchased  Once approved, contract can be used to simplify purchase order process (CERN)  AdvanceTCA  Equipment evaluated  Cooling simulation of custom shelves is ongoing (vertical cooling)  Cooling measurement of custom shelves is ongoing (vertical cooling) 19/06/2015IN2P3: DAQ working group (06/19)7

IPMI: New MMC firmware Hardware Platform Management - Introduction  Purpose:  Monitor (temperatures, voltages, current …)  Control (power management …)  Assure proper operation (verify compatibility …)  Management modules:  Module Management Controller (MMC) – for AMC management  IPM Controller (IPMC) / Carrier IPM Controller (CIPMC) – ATCA/ATCA carrier management  Shelf Manager (ShMC) – for ATCA shelf and MicroTCA MCHs  Inter-module communication (IPMI)  Intelligent Platform Management Interface  Base specification was announced September 1998  Hardware Platform Management is based on version 1.5 19/06/2015IN2P3: DAQ working group (06/19)9

IPMI: New MMC firmware Hardware Platform Management - ATCA 19/06/2015IN2P3: DAQ working group (06/19)10

IPMI: New MMC firmware Hardware Platform Management - MicroTCA 19/06/2015IN2P3: DAQ working group (06/19)11

IPMI: New MMC firmware MMC Role and features  MMC Role:  Activating / De-activating an AMC card  Providing information about the AMC card o Maximum current o Ports configuration o Clock configuration  Sending alert events (sensor exceed threshold)  Executing IPMI request  Features  Power management  Sensor monitoring  Clock and ports management 19/06/2015IN2P3: DAQ working group (06/19)12 MMC Temp. sensors current sensors voltage sensors DC/DC PLL Switch FPGA EEPROM

IPMI: New MMC firmware Activation/De-activation state machine 19/06/2015IN2P3: DAQ working group (06/19)13 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed)  CM read the FRU information  Current  E-keying  Clock  M2 (activation requested)  M3 (activation in progress)  Enable power  E-keying control  Clock configuration  M4 (active)  M5 (Deactivation requested)  E-keying de-activation  Clock de-activation  M6 (Deactivation in progress)  Disable power

IPMI: New MMC firmware CERN MMC overview  DESY  Original development  Given to CPPM by Kay Rehlich and Vahan Petrosyan  CPPM  First modification of the source code  Design of a mezzanine card o Can be used as reference design  CERN  Supported by the PH-ESE-BE group  Used in many experiments  CMS: TwinMUX (CERN)  ATLAS: Argon liquid (LAPP)  LHCb and Alice: AMC40 (CPPM) o Architecture changed but the AMC40 will be used for others projects (e.g.: Gamahadron, IN2P3)  IN2P3 projects 19/06/2015IN2P3: DAQ working group (06/19)15

IPMI: New MMC firmware Previous version status  Non-implemented features  FRU information missing (E-keying, clock configuration)  Power sequences fixed  AMC ports control not implemented  Clock configuration not implemented  Remote upgrade not supported  Major issues  Activation / De-activation state machine controlled by the MMC  Sensor event generation: Signed threshold stocked in unsigned variable  Polaris tester outcome  Automatic tests o 14 passed o 14 failed 19/06/2015IN2P3: DAQ working group (06/19)16

IPMI: New MMC firmware Motivations  MMC issues  FRU information missing o Did not work with Pigeon Point IPMC  Activation/De-activation state machine is managed by the MMC itself o One patch should be written to be compliant with Vadatech o Automatically goes from M4 to M6 with Pigeon Point  User customization was too hard o User have to read the Hardware Management Platform of the standard o No documentation  The CERN MMC is used in many experiments and should be fully compliant with the standard 19/06/2015IN2P3: DAQ working group (06/19)17

IPMI: New MMC firmware E-keying (new)  Control  GET/SET AMC PORT STATE commands (Executed after 12V enabling)  Interfaces available for control: I2C, GPIO  Hardware  Buffers  Switches  FPGA  Configuration described in FRU info (P2P connectivity record) 19/06/2015IN2P3: DAQ working group (06/19)19 #define AMC_POINT_TO_POINT_RECORD_CNT 3 #define AMC_POINT_TO_POINT_RECORD_LIST \ GENERIC_POINT_TO_POINT_RECORD(0, PORT(0), ETHERNET, BASE_1G_BX, EXACT_MATCHES) \ GENERIC_POINT_TO_POINT_RECORD(1, PORT(4), PCIE, GEN1_NO_SSC, MATCHES_10) \ GENERIC_POINT_TO_POINT_RECORD(2, PORT(5), PCIE, GEN1_NO_SSC, MATCHES_10)

IPMI: New MMC firmware Clock configuration (new)  Control  GET/SET CLOCK STATE commands (Executed after 12V enabling)  Interfaces available for control: I2C, GPIO  Hardware  Buffers  Switches  FPGA  Configuration described in FRU info (Clock configuration record) 19/06/2015IN2P3: DAQ working group (06/19)20 #define AMC_CLOCK_CONFIGURATION_LIST\ DIRECT_CLOCK_CONNECTION(TCLKA, CIPMC, NO_PLL, RECEIVER, UNSPEC_FAMILY, 0, KHz(8), KHz(7), KHz(9))\ DIRECT_CLOCK_CONNECTION(TCLKB, CIPMC, PLL, RECEIVER, UNSPEC_FAMILY, 0, KHz(2), KHz(1), KHz(3))\ DIRECT_CLOCK_CONNECTION(FCLKA, CIPMC, PLL, RECEIVER, UNSPEC_FAMILY, 0, MHz(12.8), MHz(11), MHz(13))

IPMI: New MMC firmware Additional improvements  Bug fixed  State machine removed (controlled by the carrier manager)  Sensor event generation (negative threshold)  Custom power sequences  Power ON/OFF, Reboot …  Easily defined in a configuration file  LED description  No C functions must be written to define user LED  FRU information  FRU information can be easily customized 19/06/2015IN2P3: DAQ working group (06/19)21

IPMI: New MMC firmware Previous architecture 19/06/2015IN2P3: DAQ working group (06/19)22 Hardware init. Power management Sensors management LED controlIPMI interface user_code.c  Hardware custom initialization  User led control  Sensor definition and monitoring  FRU information  OEM commands Core User

IPMI: New MMC firmware New architecture 19/06/2015IN2P3: DAQ working group (06/19)23 Hardware init. Power management Sensors management LED controlIPMI interface config.hfru_info.hsensors.hsensors.cekeying.cuser_code.c  User led definition  Custom hardware definition  Power sequences customization  FRU information  Sensor definition  Sensor monitoring  AMC port configuration  User specific IPMI commands Core User

IPMI: New MMC firmware Module Management Controller – New release  Simplification of the user customization (E.g.: FRU information) 19/06/2015IN2P3: DAQ working group (06/19)24

IPMI: New MMC firmware Module Management Controller – New release 19/06/2015IN2P3: DAQ working group (06/19)25  Simplification of the user customization (E.g.: LED description)

IPMI: New MMC firmware Module Management Controller – New release 19/06/2015IN2P3: DAQ working group (06/19)26  Simplification of the user customization (E.g.: Power sequences)

IPMI: New MMC firmware Module Management Controller - HPM.1 support  HPM.1 overview  PICMG Standard  Firmware remote upgrade features of IPM Controller (E.g: MMC)  MMC programming evolution 19/06/2015IN2P3: DAQ working group (06/19)27 MMC Application HPM Boot loader 0 0 1 1 0 0 1 0 1 1 1 0 0 0 0 0 0 1 1 1 0 1 1 0 0 0 1 1 0 1 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 0 1 0 1 0 1 1 0 0 1 1 0 0 0 1 0 0 1 1 0 0 1 1 1 0 1 0 1 0 1 1 0 0 1 0 1 0 0 1 0 0 1 1 0 0 1 0 1 1 1 0 0 0 0 0 0 1 1 1 0 1 1 0 0 0 1 1 0 1 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 0 1 0 1 0 1 1 0 0 1 1 0 0 0 1 0 0 1 1 0 0 1 1 1 0 1 0 1 0 1 1 0 0 1 0 1 0 0 1 IPMI Eth. IPMI I2CI2C

IPMI: New MMC firmware Module Management Controller - Additional tools  FRU Editor  Generates FRU binary from header file  Write new FRU in the MMC EEPROM  HPM Downloader  Generates HPM image from.hex file  Download the binary in the MMC flash memory  MTCALib  C library for mTCA (RMCP) 19/06/2015IN2P3: DAQ working group (06/19)28

IPMI: New MMC firmware MMC test with ATCA 19/06/2015IN2P3: DAQ working group (06/19)29  MMC tested successfully with the Pigeon Point starter kit  Adapter card has been designed to adapt connector to DIM-DDR3 VLP for factor.  Goal: adapt the PP solution to an existing ATCA carrier (MMC OK as well) IPMC Shelf Manager AMC1 Emulation AMC2 (CERN MMC) RTM Emulation

IPMI: New MMC firmware MMC test with ATCA 19/06/2015IN2P3: DAQ working group (06/19)30  Successful operation of the CERN MMC in an existing ATCA carrier (LAPP’s IPMC Test board carrier)

IPMI: New MMC firmware Status  MMC source code was re-written  About 90% of the code was modified  HPM.1 is fully supported  New project: boot loader program  Creation of a tool to program the MMC with HPM.1  Improvement of the IPMI compliance  Polaris tester outcome o 26 passed o 2 failed No point to point connectivity information in the MCH (not a MMC issue) IPMI command “set blue led” not filtered by the MCH (not a MMC issue)  Tested with NAT, Vadatech and SAMWAY MCHs as well as with Pigeon Point IPMC  User guide documentation 19/06/2015IN2P3: DAQ working group (06/19)31

IPMI: New MMC firmware Summary  New MMC release will be available soon  Foreseen for next week  Licence GPL  Tested in some configurations  Different MCHs (NAT, Vadatech ans SAMWAY)  ATCA carrier (Pigeon Point)  Different AMC cards: AMC LoadBoard, TwinMUX (CMS), AMC40, GLIB  IPMI compliance improved  Polaris tester outcome (almost 100% compliance)  HPM.1 support for remote upgrades 19/06/2015IN2P3: DAQ working group (06/19)33

IPMI: New MMC firmware 19/06/2015IN2P3: DAQ working group (06/19)34 Thank you for your attention ! Any question ?

IPMI: New MMC firmware Backup slides 19/06/2015IN2P3: DAQ working group (06/19)35

IPMI: New MMC firmware PH-ESE-BE: xTCA evaluation project (MicroTCA) 19/06/2015IN2P3: DAQ working group (06/19)36 MTCA Shelves (up to 40 pcs) DC power modules (up to 100 pcs) Evaluation reports available: https://espace.cern.ch/ph-dep-ESE-BE-uTCAEvaluationProject/default.aspxhttps://espace.cern.ch/ph-dep-ESE-BE-uTCAEvaluationProject/default.aspx ManufacturerELMASchroffVadatech Compliant to spec Pros Cooling efficiency except for slot 12 and RTM Layout with MCH in middle Cooling homogeneity across slots Mechanical robustness Remote support quality To be doneBackplane quality meas. Manufacturer NAT (DC840, different sources) Vadatech (UTC-010)Vadatech (UTC-020) Output power840 W750W890W Compliant to spec Pros Efficiency 100% IPMI compliant Std compliant MTBF Serial interface Std compliant MTBF Serial interface

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)37 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)38 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed) Handle switch monitoring Read FRU Information Current limit E-Keying Clock configuration

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)39 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed)  M2 (activation requested) CM Shelf Manager

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)40 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed)  M2 (activation requested)  M3 (Activation in progress) Enable AMC port Configure clocks Execute power ON sequence

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)41 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed)  M2 (activation requested)  M3 (activation in progress)  M4 (active) Sensor monitoring Power monitoring Handle switch monitoring

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)42 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed)  M2 (activation requested)  M3 (activation in progress)  M4 (active)  M5 (Deactivation requested) CM Shelf Manager

IPMI: New MMC firmware Module Management Controller - Role and feature  MMC Role:  AMC Activation / De-activation and Hardware management 19/06/2015IN2P3: DAQ working group (06/19)43 M0 M1 M2 M3 M4 M5 M6  M0 (not installed)  M1 (installed)  M2 (activation requested)  M3 (activation in progress)  M4 (active)  M5 (Deactivation requested)  M6 (Deactivation in progress) Disable AMC ports Disable clocks Execute Power OFF sequence Inform CM: deactivation finished

IPMI: New MMC firmware Module Management Controller – New release 19/06/2015IN2P3: DAQ working group (06/19)44 CoreNew user code architecture Hardware initialization AMC management IPMI interpretation Sensor management fru_info.h config.h LED definition User GPIO initialization Power sequences ekeying.c Enable/Disable AMC ports sensor.h Sensor definition sensor.c Sensor initialization Sensor reading user_code.c OEM commands Controller specific commands user_code.c LED functions GPIO initialization Sensor definition Sensor monitoring OEM commands user_code.h User’s macros Function prototypes Previous code architecture  Simplification of the user code architecture

IPMI: New MMC firmware Module Management Controller – Before modifications status  Previous architecture 19/06/2015IN2P3: DAQ working group (06/19)45 CoreUser Hardware initialization AMC management IPMI interpretation Sensor management user_code.c / user_code.h FRU: no interpretation (byte array) LED: 6 functions for management Pin init: register value modification Sensor management: Reading and check for event Hard user customization

IPMI: New MMC firmware Module Management Controller – New architecture 19/06/2015IN2P3: DAQ working group (06/19)46 CoreUser Hardware initialization AMC management IPMI interpretation Sensor management fru_info.h config.h LED definition User GPIO initialization Power sequences ekeying.c Enable/Disable AMC ports sensor.h Sensor definition sensor.c Sensor initialization Sensor reading user_code.c OEM commands Controller specific commands Easy user customization => About 90% of the code re-written

IPMI: New MMC firmware Module Management Controller - HPM.1 support  HPM implementation  Configure the microcontroller to automatically boot from the boot loader 19/06/2015IN2P3: DAQ working group (06/19)47

IPMI: New MMC firmware Carrier IPMC – Pigeon Point IPMC  Purpose: Evaluation of a commercial IPMC solution (Pigeon Point IPMC)  Licence: PH-ESE-BE (xTCA Evaluation Project)  Licence for use  Software and Hardware documentation  Starter kit  User guide 19/06/2015IN2P3: DAQ working group (06/19)48 IPMC Shelf Manager AMC1 Emulation AMC2 (CERN MMC) RTM Emulation

IPMI: New MMC firmware Carrier IPMC – IPMC Adapter Card  Goal: Evaluation of the Pigeon Point IPMC on a custom ATCA blade featuring the LAPP footprint 19/06/2015IN2P3: DAQ working group (06/19)49 Pigeon Point connectors DIM connector (LAPP) < 10cm 7cm CPLD USB USB

IPMI: New MMC firmware Carrier IPMC – IPMC Adapter Card 19/06/2015IN2P3: DAQ working group (06/19)50 Pigeon Point IPMC Connector CPLD User I/O Payload / E-Keying SPI: IO config 2 AMCs Hotswap, reset, UART, HA [7..0] User I/O Payload / E-Keying 48V Monitoring USB UART to USB Debug Payload IPMB-L AMC 1 IPMB-L AMC 2 IPMB-B / IPMB-A Master JTAG JTAG (prog) Buffer 9 AMCs (8+1RTM) Phy Ethernet SoL Eth. IPMB-B_Enable / IPMB-A_Enable En Jumpers User I/O 48V Monitoring IPMB-B / IPMB-A IPMB-L 35 13 4 20 16 4 15 LAPPConnectorLAPPConnector I2C buffers

IPMI: New MMC firmware Carrier IPMC – IPMC Adapter Card  Evaluate Pigeon Point IPMC on a custom ATCA blade featuring the LAPP footprint 19/06/2015IN2P3: DAQ working group (06/19)51 CPLD Pigeon point connectors CPLD

IPMI: New MMC firmware  Management of up to 4 AMCs  Communication through IPMB-L bus  Activation/De-activation  SDR / FRU information  Sensor monitoring (events)  HPM.1  Intelligent and non-intelligent RTM  Activation/De-activation  E-keying and clock configuration Pigeon Point IPMC evaluation 19/06/2015IN2P3: DAQ working group (06/19)52  Polaris tester  Ethernet  RMCP  HPM.2 and HPM.3  IPMB-A / IPMB-B  Communication with the Shelf Manager  HPM.1  Management of the ATCA blade  SDR/FRU information  Sensors monitoring (customization, event …)  ATCA e-keying and clock configuration  OEM commands ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

IPMI: New MMC firmware IN2P3: DAQ working group (19/06/15) 19/06/2015IN2P3: DAQ working group (06/19)1 PH-ESE-BE collaboration Vincent Bobillier, Stefan.

Similar presentations

Presentation on theme: "IPMI: New MMC firmware IN2P3: DAQ working group (19/06/15) 19/06/2015IN2P3: DAQ working group (06/19)1 PH-ESE-BE collaboration Vincent Bobillier, Stefan."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

IPMI: New MMC firmware IN2P3: DAQ working group (19/06/15) 19/06/2015IN2P3: DAQ working group (06/19)1 PH-ESE-BE collaboration Vincent Bobillier, Stefan.

Similar presentations

Presentation on theme: "IPMI: New MMC firmware IN2P3: DAQ working group (19/06/15) 19/06/2015IN2P3: DAQ working group (06/19)1 PH-ESE-BE collaboration Vincent Bobillier, Stefan."— Presentation transcript:

Similar presentations

About project

Feedback