Issues concerning Device Access (JAPC / CMW / FESA) With input from: A.Butterworth, E.Carlier, A. Guerrero, JJ. Gras, St. Page, S. Deghaye, R. Gorbonosov, M. Lamont, L. Mestre, E. Roux, N. Stapley, K. Sigerud, J. Wenninger, M. Arruat, Kris Kostro, JL. Nougaret

TC 09-Mar-06 2 Vito Baggiolini AB/CO JAPC / CMW / FESA JAPC CMW FESA Java Applications Hardware Timing

TC 09-Mar-06 3 Vito Baggiolini AB/CO Purpose & Scope Purpose: Identify open issues that need to be addressed –Which might create delays and other problems if not handled Scope: “Device Access” –Interactions between devices and applications –Involving JAPC, CMW, and the communication part of FESA –NOT in scope: Issues between FESA and equipment groups Problems with individual applications or equipment Don’t shoot the messenger ;-)

TC 09-Mar-06 4 Vito Baggiolini AB/CO Experience So Far Experience gathered –in LEIR, HW Commissioning, CESAR Dry Run –And within the equipment groups Positive aspects: we have a standard solution in place –FESA is a great improvement over previous approaches –CMW set/get access is fine, CMW gateways are now more reliable, data format (tags) have been standardized –JAPC is the universally used device access method in Java Areas of concern –Throughput/scaling has not been tested at full scale –Monitoring mode does not fulfill client requirements –CMW/FESA alarms solution is not really convincing –Some functionality is missing

TC 09-Mar-06 5 Vito Baggiolini AB/CO Scaling / Throughput Never tested at LHC scale  real tests needed Most demanding throughput requirements –OASIS logging acquisition mode (BT: 50’000 point/signal at 1Hz) –Logging of diagnostic data to Meas DB (tens of MBytes/cycle) –Post-mortem (e.g. PO: max 400MBytes = 1700 * 256kByte) Most demanding latency/update rate requirements: –OASIS “scrolling mode”: 10 points at 20Hz –3 Hz update rate (REX) –Monitoring at 1-2Hz (PO, RF status, Orbit, Beam Loss, …) Requirements for high numbers of subscriptions: –LHC PO: ~1700 PC on 80 Linux gateways (not FESA) –Up to 400’000 subscriptions for alarm connection as currently proposed by CMW/FESA –Full running LHC: a few subscriptions per device Shall/can these requirements be fulfilled?

TC 09-Mar-06 6 Vito Baggiolini AB/CO Monitoring chain is not reliable enough Requirement –Above JAPC, clients must be sure to receive either data or an error notification Current situation –If no data arrives, the client cannot distinguish: really no data or a problem? –Not sure to be notified on all possible failures Consequence –Operators don’t trust monitoring-based GUIs (“Give me a refresh button”) –We can’t base critical applications on monitoring JAPC CMW FESA Java Client Hardware Timing Current solution: Java Client subscribes to timing –To know when data should arrive and check –Not sufficient to make on-change monitoring reliable!

TC 09-Mar-06 7 Vito Baggiolini AB/CO Some equipments send updates too late Requirement: –Updates for a cycle must arrive by end of that cycle (+ fix delay) –All equipment types must comply with this behavior Current Situation: –Some equipments deliver data too late Consequence –It is very complicated to correlate data from different equipment –Lionel was forced to implement complex work-around inside JAPC –Maintenance nightmare At least newly developed equipment must fulfill this requirement!

TC 09-Mar-06 8 Vito Baggiolini AB/CO Alarms of FESA Equipment LASER Traditional Alarm Source Alarm Source API JMS Broker Gateway FESA Hardware CMW Subscription to alarm properties

TC 09-Mar-06 9 Vito Baggiolini AB/CO CMW/FESA Alarm Connection Proposed mapping of alarms to FESA properties: –One FESA property per potential alarm state (not per active one) –Some fault states can be cycle-dependent (PPM) –Gateway subscribes to all properties (even several times if PPM) Alarm team –Anticipate 250’000 – 400’000 potential fault states for LHC era –Worry about viability of a solution that requires so many CMW subscriptions –Intermediate gateway hampers debugging if alarms don’t arrive Kris –Is not worried about large number of subscriptions –Plans to achieve scalability using several gateways in parallel Jean-Luc –A single alarm property per device should be possible –More thorough analysis needed to confirm this

TC 09-Mar Vito Baggiolini AB/CO Functionality currently missing Access control to protect expert and critical settings –Authentication plus (role-based) authorization –Requested by all equipment groups Transactional set() actions for LSA (PO, RF) –“All-or-nothing” behavior: either all power converters accept new settings or none does Group actions in CMW –CMW only supports individual actions (~ 90ms per subscription) –Reqmt: Single interaction for set/get or monitor a group of devices Restricted to same property on a group of devices E.g. subscribe to Acquisition property on a list of magnets in one call E.g. get Status property of many equipment in one call –This function is ready in JAPC but not implemented in CMW What about UDP infrastructure for real-time feedback?

TC 09-Mar Vito Baggiolini AB/CO Conclusions/Recommendations Real-scale throughput/scaling tests needed –Relevant tests must be identified and planned carefully –Sector test is too late to find problems (should be a validation) Missing functionality must be provided soon –Otherwise many individual solutions will appear Problems are probably more human than technical –Involved people do not manage to find compromises –Consequence: bad solutions and complicated workarounds Workshop? In my opinion No. Better to address issues in small focused efforts –With small technical (not political) teams –With concrete problems to solve –With milestones in the near future (SPS start-up?) –With close follow-up (e.g. short progress reports in TC)