Middleware renovation – technical overview 16th april 2013 Wojciech Sliwinski BE-CO-IN for the Middleware team: Felix Ehm, Kris Kostro, Joel Lauener, Radoslaw Orecki, Ilia Yastrebov, [Andrzej Dworak] Special thanks to: Vito Baggiolini and Pierre Charrue

Wojciech Sliwinski, Middleware Renovation: Technical Overview Agenda Context & Motivation for Renovation Middleware Review process Technical evaluation of the transport layer Changes in the MW Architecture in LS1 Conclusions 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Context & Motivation for Renovation Agenda Context & Motivation for Renovation 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Wojciech Sliwinski, Middleware Renovation: Technical Overview MW Mandate & Scope Standard set of MW solutions Centrally managed services Track & optimize runtime parameters Well defined feedback channel for users Provide support & follow-up issues Scope: CERN Accelerator Complex Operational 24*7*365 Must be Reliable & High Quality 73’000 HW devices, 3’150 servers In all Eqp. groups (4 dpts: BE, EN, GS, TE) 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

CMW in the Controls System CMW client (C++/Java) JAPC GUIs, LabView, RADE JMS client (Java) GUIs CMW client (Java) JAPC Logging, LSA, InCA, SIS CMW client/server (C++/Java) Proxy, DIP, AlarmMon, AQ JMS client (Java) Servers: Logging, InCA, SIS CMW server (C++) FESA, FGC, GM CMW server (C++) PVSS (Cryo, Vacuum) 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Motivations for MW Renovation Current CORBA-based CMW-RDA Integrated in the Control system Used to operate all CERN accelerators Provides widely accepted Device/Property model > 10 years old Why to review & upgrade MW ? CORBA was choosen 15 years ago Technical limitations of CORBA-based transport Functional limitations of the current CMW-RDA Codebase with long history  difficult to maintain, needs architecture review Major issue of long-term support & future evolution Evolution of technology over last 10 years: HW, OS, middleware, 3rd party libraries Human factor  less & less CORBA expertise on the market 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Technical limitations of CORBA transport Became legacy, not actively supported  maintenance issue Shrinking community, slow response time omniORB (C++) – 1 developer/maintainer, last release mid-2011 JacORB (Java) – few developers, small community Major technical limitations Lack of fully asynchronous processing channel Blocking communication  infamous JacORB blocking issue Lack of low-level control of IO resources (sockets, request queues) Development issues Difficult to extend the wire protocol  Backward compatibility issue Complex, error prone API Heavy in memory usage 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Summary: Why change CORBA? CORBA was choosen 15 years ago Not actively maintained  big risk for the MW project Better solutions exist on the market Invest in future solution rather than maintaining old one 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Functional limitations of CMW-RDA Several pending operational issues Difficult (or hardly possible) to resolve with current library Any major change very difficult to introduce Technical Stops & Xmas breaks too short for massive deployment High risk  Major impact on front-end frameworks and applications No protection against ’slow/bad’ client applications Misbehaving application may destabilise front-end server Affects reliability of the subscription channel Workaround: introduction of Proxy Poor scalability when many clients subscribed Stability issues observed when >200 clients subscribed (even for Proxy) Threading model doesn’t scale well with many clients Missing support for priority clients (e.g. SIS, PM, InCA, Logging) Non-critical clients (e.g. GUIs) have the same communication priority + others … 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Summary: Why change CMW-RDA? With current CORBA-based middleware we can’t solve the pending operational issues We can’t provide better scalability & reliability CMW-RDA is difficult to evolve & extend 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Middleware Review process Agenda Middleware Review process 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Middleware Renovation process MW Renovation = MW Review + MW Upgrade MW Review aims to provide the most appropriate technical solution satisfying the user requirements MW Upgrade establishes the plan & strategy for introduction of the new MW Objective: LS1 the unique opportunity for the major MW upgrade Middleware Review Process Gathering of users feedback and requirements (2010-11) Review of communication and serialization libraries (2011-12) Prototyping using selected communication products (2012) Design & impl. of new RDA3: Data, Client & Server (2012-13) Testing & validation of core MW infrastructure (summer’13) Upgrade of all dependent MW libraries & services (2013-14) JAPC, Directory Service, Proxy, DIP Gateway 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Review of users requirements 2010-11 – series of interviews with major users Lars Jensen, Stephen Jackson (BI) Andy Butterworth, Frode Weierud, Roman Sorokoletov (RF) Brice Copy, Clara Gaspar (DIP, DIM) Frederic Bernard, Herve Milcent, Alexander Egorov (PVSS) Alexey Dubrovskiy (CTF), Kris Kostro (DIP gateways) Marine Gourber-Pace, Nicolas Hoibian (Logging) Nicolas De Metz-Noblat (Front-Ends), Alastair Bland (Infrastructure) Michel Arruat (FESA), Stephen Page (FGC) Niall Stapley, Mark Buttner, Marek Misiowiec (LASER & DIAMON) Nicolas Magnin, Christophe Chanavat (ABT) Stephane Deghaye, Jakub Wozniak (InCA, SIS) Vito Baggiolini, Roman Gorbonosov (JAPC & DA systems) + regular feedback from OP + internal team input http://wikis/display/MW/Interviews+with+Experts 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3: Accepted requirements New requirement General Java & C++ API, Win (64-bit) & Linux (SLC5 32-bit & SLC6 64-bit) Accelerator Device Model (i.e. Device/Property) Get, Set, Async-Get, Async-Set, Subscribe Early detection of communication failures Improve error reporting in all the layers: client, server, gateways Admin interface & runtime diagnostics & statistics Data support Data object: primitives, n-dim arrays, data structures Subscription mechanism Subscription behaviour the same regardless condition of the server (active, down) Several client subscription policies (default: continuous) Provide subscription notification ordering First-Update enforced via CMW on server-side Provide callback to front-end framework for the server-side Get Drop support for on-change flag Standardise use of subscription filters and update flags (e.g. immediate update) Add header for acquired Data  common metadata (e.g. acq. stamp, cycle name) All loss of data (dropped updates) must be notified to clients 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3: Accepted requirements New requirement Client side RDA3 client API connects with both: RDA2 (old) & RDA3 (new) servers Efficient mechanism for: connection, disconnection & reconnection Must be able to recover from any interruption of communication with the server Server restarts, IP address change, rename/move of a device to another server Improved semantics of Array Calls, i.e. handling of individual parameters Enhanced diagnostics & collection of statistics Server side Policies for discarding notifications, i.e. deal with overflows and ’bad clients’ Instrument with counters & timings allowing to diagnose the notifications delivery Prioritisation of Get/Set requests for high-priority clients Server-side subscription tree fully managed by CMW Server does not need to manage client subscriptions any more Manage the client connections, e.g. forced disconnect of a client Client lifetime callbacks (i.e. connected, disconnected) 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3: Accepted requirements New requirement Server side (cont.) Client discovery for the diagnostics purposes (i.e. connected clients with payload) Enhanced diagnostics & collection of statistics Ongoing discussions (not accepted yet) Prioritisation of subscription notifications for high-priority clients Technical notes Invest in asynchronous & non-blocking communication Prefer 0-copy & lock-free data structures, message queues http://wikis/display/MW/Design+of+New+RDA 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3: Summary of requirements Unchanged Device/Property model Set of basic operations (Get, Set, Subscribe) Fixes & improvements Subscription mechanism Connection management Diagnostics & statistics New functionality Policies for subscription management (client & server) Client priorities Server-side subscription tree Extended Data support Standardise First-Update concept 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Technical evaluation of the transport layer Agenda Technical evaluation of the transport layer 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Middleware transport requirements Desirable Lightweight Friendly API, documentation Request/reply & pub/sub patterns Mandatory Open source license Asynchronous Active community Stability, Maturity & Longevity Performance & Scalability Fundamental C++/Java Linux/Windows Over TCP/IP LAN 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Evaluation process –> our criteria Appearance Creators specification documentation Users forums bug reports Internet Simple usage Download licensing Compile Linux & gcc Run examples Testing Communication patterns Performance Exceptional situations QoS Configuration CRITERIA QoS Resources, binary size, memory Performance Communications patterns API, look & feel, documentation Community, maturity 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview Andrzej Dworak, ICALEPCS 2011

Evaluated middleware products All opinions are based only on our knowledge and evaluation. Each of the products, depending on the requirements, may constitute a good solution. OpenAMQ CoreDX RTI DDS QPid ZeroMQ OpenSpliceDDS RabbitMQ YAMI Ice omniORB MQtt RSMB JacORB Thrift Mosquito 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview Andrzej Dworak, ICALEPCS 2011

Products comparison (according to the criteria) Sync, async & msg patterns QoS Dependencies & memory f-p Performance Look & feel, API, docs Community & maturity Score ZeroMQ  6 Ice  5 YAMI4 4 RTI 3 Qpid CORBA 2 Thrift 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview Andrzej Dworak, ICALEPCS 2011

Wojciech Sliwinski, Middleware Renovation: Technical Overview Conclusions Several good middleware solutions available The choice is dictated by the most critical requirements Not easy  performance matters but also ease of use, community, … Prototyping was done with the most promising candidates: ZeroMQ, Ice & YAMI Finally we decided to choose ZeroMQ (http://www.zeromq.org/) Asynchronous & non-blocking communication 0-copy & lock-free data structures, message queues Nice API, good documentation & active community 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3 Java – Sync Get round-trip time Test setup: 1kB message payload, cs-ccr-* machines, 1 server host & 10 client hosts 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3 Java – subscription notification latency Test setup: 1kB message payload, cs-ccr-* machines, 1 server host & 10 client hosts 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

New RDA3 Java – subscription notification latency Test setup: 1kB message payload, cs-ccr-* machines, 1 server host & 10 client hosts 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Changes in the MW Architecture in LS1 Agenda Changes in the MW Architecture in LS1 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Current MW Architecture User written Middleware Java Control Programs Central services VB, Excel, LabView C++ Programs Administration console JAPC API Passerelle C++ Clients RDA Client API (C++/Java) Device/Property Model Directory Service Directory Service RBAC A1 Service RBAC Service Configuration Database CCDB CMW Infrastructure CORBA-IIOP RDA Server API (C++/Java) Device/Property Model CMW integr. CMW int. CMW int. CMW int. CMW int. CMW int. Servers Virtual Devices (Java) FESA Server FGC Server PS-GM Server PVSS Gateway More Servers Physical Devices (BI, BT, CRYO, COLL, QPS, PC, RF, VAC, …) 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Changes in MW Architecture in LS1 User written Middleware Central services Java Control Programs VB, Excel, LabView Upgrade in LS1 C++ Programs Administration console JAPC API Passerelle C++ Clients RDA Client API (C++/Java) Device/Property Model Directory Service Directory Service RBAC A1 Service RBAC Service Configuration Database CCDB CMW Infrastructure ZeroMQ RDA Server API (C++/Java) Device/Property Model CMW integr. CMW int. CMW int. CMW int. CMW int. CMW int. Servers Virtual Devices (Java) FESA Server FGC Server PS-GM Server PVSS Gateway More Servers Physical Devices (BI, BT, CRYO, COLL, QPS, PC, RF, VAC, …) 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Wojciech Sliwinski, Middleware Renovation: Technical Overview LS1: Changes in RDA New major version: RDA3 (June’13 – alpha version) Public API NOT backward compatible New protocol, new architecture, new design Same Device/Property model & Get/Set/Subscribe calls Announcement via accsoft-java-announce list Required Actions for RDA Users For Java: Use new version of JAPC (API unchanged) For Java: New JAPC will support communication with RDA2 & RDA3 servers For C++: Upgrade user code to new RDA3 API For C++: RDA3 will support communication with RDA2 & RDA3 servers Consequences if NO Action  staying with old RDA2 NOT possible to communicate with new RDA3 servers (FESA3, FGC, etc.) 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Wojciech Sliwinski, Middleware Renovation: Technical Overview LS1: Changes in JAPC New major JAPC version  upgrade for RDA3 (September’13) Public API backward compatible Possible API extensions, but always compatible Announcement via accsoft-java-announce list Required Actions for JAPC Users Update JAPC jars (via CommonBuild) Re-release your product (via CommonBuild) New JAPC will support communication with RDA2 & RDA3 servers 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Wojciech Sliwinski, Middleware Renovation: Technical Overview Middleware Team Wojtek Sliwinski (Lead) 100% – Directory, RDA, Proxy, RBAC Felix Ehm 30% – JMS, Log/Tracing, Feedback/Metrics Joel Lauener 90% – CMW Admin, Directory, RDA, GM, DIP Gw. Kris Kostro 20% – DIP Gateways, RDA3 Wojtek Buczak 30% – JAPC Core Ilia Yastrebov 100% – RDA, RBAC, Passerelle, Proxy, Log Radoslaw Orecki 100% – Directory, RDA3 Support: cmw-support@cern.ch, rbac-support@cern.ch Docs: http://wikis/display/MW 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview

Wojciech Sliwinski, Middleware Renovation: Technical Overview Conclusions We have to replace CORBA with a new solution We want to resolve the pending operational issues We collected updated users requirements New product (ZeroMQ) was choosen to replace CORBA We will minimize changes to client SW during LS1 16th April 2013 Wojciech Sliwinski, Middleware Renovation: Technical Overview