Siemens openlab Major Review › 15 Oct 2014

Organization 15 Oct 2014Major Review – Siemens CERN openlab2 Siemens/ETM Supervisor CERN Supervisor openlab Researcher Control System Security Filippo TilaroBrice Copy Christoph Fischer, Bernhard Petri Data Analytics Filippo TilaroAxel Voitier M. Roshchin M. Kalinkin C. R. Vintila Database Archiver Kacper Szkudlarek Piotr GolonkaEwald Sperrer Deployment Tool Pavel Fiala Fernando Varela Ewald Sperrer Management Siemens - Thomas HahnCERN - Manuel Gonzalez

Summer Students Projects › Making sense of data streams [Report] [Report]  Kacper Blazej Sokol › Log analysis and classification of CERN control systems [Report] [Report]  Zahari Dimitrov Kassabov 15 Oct 2014Major Review – Siemens CERN openlab3

Typical Control System Architecture 15 Oct 2014Major Review – Siemens CERN openlab4 Fieldbus TN PLCs Sensors & Actuators MOON (Monitoring ) High Voltage DIM/CMW OPC Field layer Process layer Supervision layer Data Analytics SECURITYSECURITY SCADA WinCC OA Front-end PC

Siemens openlab Major Review Author: Filippo Tilaro Supervised by: Brice Copy › 15 Oct 2014 Control System Security

Security project › Main goal:  Improve the Siemens Process Control Systems (PCS) security level › Strategy:  Design of a test-bench to evaluate the PCSs network robustness  Determine key cyber security aspects relevant to CERN in accordance with recent cyber security standards 15 Oct 2014Major Review – Siemens CERN openlab6

Conclusions › Achievements:  Certification Robustness Test extension for IEC standards communication protocols specifications ̵ In-line with ISA Secure Committee Institute (ISCI)  Test-bench Release: ̵ Setup and installation into Siemens Headquarters ̵ Support for test execution and new test definitions  Publication and presentation at ICALEPCS 2013 › Possible future activities:  SCADA System Testing ̵ WinCC OA internal communication protocol ̵ OPC UA (Unified Architecture)  Further communication protocols analysis  Industrial Intrusion Detection System 15 Oct 2014Major Review – Siemens CERN openlab7

Siemens openlab Major Review Author: Filippo Tilaro Supervised by: Axel Voitier › 15 Oct 2014 Data Analytics

Data Analytics project › Main goal:  Build a computing system able to improve the functionality, the efficiency and the predictability of control systems › Strategy:  Use and extend Siemens analysis tools  Extract possible patterns and discover new insights hidden in the controls data itself  Take advantage of the huge amounts of controls data produced by CERN facilities 15 Oct 2014Major Review – Siemens CERN openlab9

Overview of the activities › Off-line analysis [WatchCAT]  QPS time errors analysis › On-line analysis [ELVis]  Statistical time series data analysis  Anomaly detection system based on the generation of the alarms 15 Oct 2014Major Review – Siemens CERN openlab10

OFF-LINE analysis activities 15 Oct 2014Major Review – Siemens CERN openlab11

LHC Quench Protection System (QPS) 1215 Oct 2014Major Review – Siemens CERN openlab Field layer signals Control layer 48 FECs Supervision layer 48 WinCC OA Instances Middleware Nanosecond level synchronization Losses can only be traced in logs (limitation of internal DB) ~2800 LHC radiation-hard devices Long term archive LHC Logging Short term archive WinCC OA RDB WorldFIP bus Desynchronization at control/field layer produces data losses in supervision

QPS time errors analysis Logs - Data Extraction XML Conversion Patterns description Siemens WatchCAT Analysis Tool Fault detection: › Individual time fault signatures › Fault frequency in variable time windows › Root-cause analysis 1315 Oct 2014Major Review – Siemens CERN openlab Faults classification › Field agent not ready to read the time over WorldFIP › The FEC did not handle fast enough the timing data and repeated the same time frame on the WorldFIP bus › Timing error: from the timing card, timing system 48 instances

15 Oct WatchCAT desirable › Conversion into a Service  User-driven analysis without manual actions  On-line mode for continuous control system monitoring over custom time-windows › Data persistence and scalability  Distributed storage to overcome the current single host limitations  Not only in-memory analysis › Common interface for both WatchCAT and ELVis to communicate with WinCC OA › Predictive trending analysis based on time- series process data › Numerical analysis based on CEP Major Review – Siemens CERN openlab

ON-LINE analysis activities 15 Oct 2014Major Review – Siemens CERN openlab15

Fieldbus TN PLCs Sensors & Actuators MOON (Monitoring ) High Voltage DIM/CMW OPC Field layer Process layer Supervision layer Our vision of the analysis framework 12 Feb 2014 Data Analysis Framework Data collection & feedback FFT Machine Learning Neural Network CEPPatterns(R)(LabView)(Java) (WatchCAT) Data Processing Modules Scalable and fault-tolerant !!! Expert Visualisation Analysis memory and configuration Historical Data

ELVis on-line analysis framework › Status:  Under development  Running on CERN OpenStack VMs › High scalability of analysis processes  From laptop to multi-node cluster › Stream based data processing engine › NoSQL data storage engine  under development › Web-based visualisation interface  HTML5, Data pushed by Web-Sockets  Desktop and mobile devices 15 Oct 2014Major Review – Siemens CERN openlab17

Analysis Task Results aggregator Distributed Message Broker Packet Consists of packet id, array of values and timestamp JSON serialization: R execution through STDIN/STDOUT Testing environment: 4 CERN OpenStack virtual machines, whose one for the cluster supervision. Statistical indexes extraction from time series data Major Review – Siemens CERN openlab15 Oct Source Tasks Analysis Task Analysis Task R integration in ELVis

Sampling rate2 VMs, 8 tasks3 VMs, 16 tasks4 VMs, 24 tasks 1000 Hz500 signals900 signals1.4K signals 100 Hz6K signals10K signals13K signals 1 Hz500K signals900K signals1200K signals › Java analysis code runs from 6x to 20x faster than R  IO bottleneck: due to JSON serialization of large data arrays › R good for:  exploratory analysis and building prototypes  low volume data analysis R integration achievements Major Review – Siemens CERN openlab15 Oct › Scale across multiple hosts/VMs  Overcame the initial limitation using JRI to run one R-engine per JVM  Multi-language protocol support via the STDIN / STDOUT interfaces Maximum number of signals processed in R

On-line analysis of control alarms 15 Oct 2014Major Review – Siemens CERN openlab20 MOON: control system infrastructure monitoring MOON: control system infrastructure monitoring ELVis Processing Engine ELVis Processing Engine Alarms feeding Web Reporting › Alarms analysis to detect anomalies or abnormal behaviors  granularity: from system to single devices › Parallelization using the CERN OpenStack cluster › Threshold learning algorithm and outliers detection techniques:  Several versions: statistical techniques are affected by the alarms distribution and system changes › Graphical visualization of the outliers Anomaly detection

Conclusions › Various kinds of analysis but all aiming at monitoring and improve the predictability of CERN control system › Current focus:  Design and implementation of analysis flows specifically tailored for each CERN control application  Use of WatchCAT as an ELVis plug-in for complex event processing ̵ Base line analysis: rule model, analytical data relationships, temporal reasoning › Future activities:  Integration of Siemens analysis tools with our monitoring and control environments ̵ OPC DA solution for low sampling data rate ̵ WinCC OA direct interface for high volume data  Analysis flow configurable by user ̵ + It can use custom analysis software 15 Oct 2014Major Review – Siemens CERN openlab21

Siemens openlab Major Review Author: Kacper Szkudlarek Supervised by: Piotr Golonka › 15 Oct 2014 Database Logging Services for WinCC OA

Activities since last major review › Future SCADA version:  NoSQL/Hadoop workshop for ETM. › Current WinCC OA version 3.X:  RDB Archiver,  Reporting Manager. 15 Oct Major Review – Siemens CERN openlab ARC Archive manager RM Reporting manager SOAP

LHC Quench Protection System (QPS) 2415 Oct 2014Major Review – Siemens CERN openlab Field layer signals Control layer 48 FECs Supervision layer 48 WinCC OA Instances Middleware ~2800 LHC radiation-hard devices Long term archive LHC Logging Short term archive WinCC OA RDB WorldFIP bus User’s Analysis 150K changes/s Filtering › Critical system for safe LHC operation:  Major upgrade during LS1 due to increase LHC energy (13 TeV). › Unprecedented requirements for throughput and data storage:  Constant data rate of 150k changes/s from 133k signals. › Whole data set is transfered to long-term storage DB:  Query + Filter + Insert to secondary DB.

Improvements for QPS Archiving › Use of Oracle Index Organized Tables › Tuning of DB data queries:  Search predicates, time-based partitioning,  Alignment of data in database cluster. › Focus on space reduction. › Optimization of database schema. › Tuning of database parameters. › Integration with LHC logging system. › Necessary changes also in WinCC OA. 15 Oct BE-CO EN- ICE IT-DB Major Review – Siemens CERN openlab

Test results for QPS use case 15 Oct Major Review – Siemens CERN openlab › Successfully implemented performance improvements:  Nominal conditions: ̵ Steady constant rate of 150k changes/s, ̵ 100 MB/s of I/O operations, ̵ 500 GB of data stored each day.  Peak performance: ̵ Exceeded 1 million changes/s, ̵ MB/s of I/O operations.  Presented at Oracle Analytics Innovation Summit, May 2014

WinCC OA RDB Archiver › All improvements fed back to ETM:  Integrated into current versions of WinCC OA,  Made available to all clients. › Deployment at CERN:  Upgraded QPS, several other EN-ICE projects,  Deployment ongoing at experiments,  Goal: all CERN systems updated before end of the LS1 (~600 systems). 15 Oct 2014First Name and Family Name – CERN openlab27

WinCC OA Reporting Manager 15 Oct Major Review – Siemens CERN openlab › New manager in WinCC OA:  Retrieving process and historical values for business reporting tools,  SOAP based interface,  Historical data retrieved from Oracle DB,  To be available in the next version of WinCC OA. RDB Archive WinCC OA RM Reporting Mnager...

Contributions to Reporting Manager 15 Oct Worker thread Working thread Worker thread Thread pool … Connection pool SOAP CERN joins the project at early stage › Oracle data query interface:  Connection pooling. › Structural improvements:  Improved multithreading,  Introduced high-level interfaces,  Code refactoring, Qt. RDB Archive Major Review – Siemens CERN openlab

Reporting Manager: performance › Test case:  Execution of 100 requests at the same time,  Each request for a different Data Point,  historical values returned for each request,  Measured: total time of serving all requests. 15 Oct Major Review – Siemens CERN openlab

Conclusions for Openlab Phase IV › Current version of WinCC OA (3.X):  Work focused around immediate needs for LS1.  RDB Archiver: ̵ Performance for high-throughput systems, ̵ Reduction of the database storage space.  Reporting Manager: ̵ Multipurpose data query interface: available soon, ̵ New horizon for integration with other systems. › Future SCADA versions: Logging Service  Contributing CERN experience with large SCADA systems during R&D phase,  Initial research on the use of NoSQL systems: ̵ NoSQL/Hadoop workshop for ETM in Eisenstadt.  First working version of Oracle database plugin,  First CERN’s setup of Siemens’ future SCADA. 15 Oct Major Review – Siemens CERN openlab

Siemens openlab Major Review Author: Pavel Fiala Supervised by: Fernando Varela › 15 Oct 2014 Future Siemens SCADA Centralized Deployment Tool

Centralized Deployment Tool (CDT) › Large controls applications at CERN comprise >180 interconnected WinCC OA systems › The CDT will allow pushing upgrades onto sets of WinCC OA applications in a centralized fashion 15 Oct › ASCII Manager is a key component of the CDT  Imports/exports of the runtime DB of a project from/to files  Configures communication with the hardware equipment Major Review – Siemens CERN openlab UI User interface Editor UI User interface Runtime UI User interface Runtime D Driver D Driver D Driver CTRL Control manager API API manager DM Data manager EV Event manager ASCII manager DIST Other systems connection This work focuses on the new generation of SCADA currently under development by Siemens/ETM

Objectives for last 6 months › Finish requirement phase › Complete technology evaluation › Continue with architecture design › Start prototyping  Export of the runtime DB  Import of the runtime DB  Verify architecture decisions (feasibility) 15 Oct 2014Major Review – Siemens CERN openlab34

Pre-prototype phase › Requirements were agreed › Technologies were selected  C++ XML parser  XSD XML-C++ binding  Two possibilities to interact with the new SCADA ̵ Direct access to configuration persistency files ̵ High-level interfaces to access the runtime (CHROM API) 15 Oct 2014 Major Review – Siemens CERN openlab35

Architecture design › Decision to use CHROM interfaces  Easier implementation › Correctness of the decision to be proven  Performance issues may be encountered ̵ Messages routed through the event manager  Consider switching to low-level access (RDF framework) › ASCII manager based on CHROM client › XML schema reflects CHROM data model 15 Oct 2014Major Review – Siemens CERN openlab36

Prototyping process › Include XSD compiler in build process › Pioneering the usage of the CHROM SDK  Influence on our development process  Feedback to ETM on SDK tutorial › Need to adapt to the fast development process of SDK  Runtime release every 3 months 15 Oct 2014Major Review – Siemens CERN openlab37

Prototyping achievements › Goals achieved  Export of basic data structures › On-going work  Import of self-produced export file ̵ Found out missing interfaces. SDK to be updated  Verify architecture decisions ̵ Benchmark performance 15 Oct 2014Major Review – Siemens CERN openlab38

Phase IV Summary › Involvement in an early stage of the project  Since initial brainstorming to define the functionality  Requirement gathering and definition of functionality › Slowly moved on the prototyping & testing  Diverted in many different directions and dead-ends  Evaluation of technologies › Underlying frameworks under development 15/10/2014Major Review – Siemens CERN openlab39 Although it took long time to start rolling, the new ASCII manager is now on the right track