Machine Learning for Optics Measurements and Corrections Elena Fol, J. Coello de Portugal, R. Tomas CERN (BE-ABP-HSS), Goethe University Frankfurt  04.06.2019, BE Machine Learning & Data Analytics Forum

Part I. Detection of faulty Beam Position Monitors

Motivation: faulty BPMs in optics analysis Unphysical values coming from faulty BPMs signal still can be observed in reconstructed optics even after cleaning with available tools  Important to remove faulty BPMs since they affect the optics much more than missing good BPMs. ML as an alternative solution to improve the analysis Calculate optics functions (beta-beating, dispersion, etc.) based on harmonic analysis of BPMs signal Optics measurements at LHC BPMs record the turn-by-turn data measuring the oscillations of the excited beam  + data cleaning

Anomaly detection: Detection of faulty BPMs Statistical analysis of the past measurements shows that ~10% of BPMs are faulty General Idea: Since it is unknown, which BPMs are really physically defected, we consider the appearance of non-physical spikes in reconstructed optics as artifact of bad BPMs We do not want to replicate current results, no training data set (input-output pairs) available: Unsupervised learning approach Assuming most of the BPMs measure correctly, the bad BPMs should appear as an anomaly Applied algorithms: K-means[1], DBSCAN[2], Local Outlier Factor[3],  Isolation Forest[4]  using Scikit-Learn 3. Breunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000, May)., LOF: identifying density-based local outliers 4. Liu, Fei Tony, Ting, Kai Ming and Zhou, Zhi-Hua. “Isolation forest.” Data Mining, 2008. ICDM‘08. 1. Stuart P. Lloyd. Least squares quantization in PCM 2. “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise” Ester, M., H. P. Kriegel, J. Sander

Isolating a ”normal” point Isolation Forest (IF) Forest consists of several decision trees, each tree performs splits to isolate each point in given dataset Selects a feature, randomly selects a split between minimum and maximum values of the selected parameter Isolating a ”normal” point Isolating an outlier https://quantdare.com/isolation-forest-algorithm/

Anomaly detection based on combination of differently scaled features Less splits are needed to isolate an anomalous point – anomaly score is based on number of splits Finding structures in the data: algorithm learns a function based on anomaly score averaged over the number of trees The learned function can be used also in “supervised” way to classify new data. There is no data pre-processing needed actual analysis, the ∆ Tune is the deviation of the tune values from the average and is computed for visualization only.

β-beating from the measurement cleaned with old techniques (SVD and threshold cuts) before and after applying IF: Fully integrated into optics measurements at LHC Successfully used during commissioning and machine developments

Part II. Correcting beta-beating using ML: first results

Optics measurements at the LHC BPMs record the turn-by-turn data measuring the oscillations of the excited beam The betatron phase of the BPMs is derived from the harmonic analysis. The phase advance between BPMs is then used to calculate β-function, which are related to the transverse size of the beam Control of β-function guarantees a safe operation as well as providing required luminosity for the experiments 8/18/2019

Optics measurements at the LHC One of the main parameters is beta-beating: ratio of the measured β-function with respect to the nominal designed function Corrections aim to minimize the difference between the measured and design optics by changing the strength of corrector magnets – single quadrupoles and quadrupoles powered in circuits. Optics corrections in the LHC are currently based on a response matrix between available correctors (single quadrupoles or powered in circuits) and observables.

Idea and problem definition Optics correction: identify changes of quads circuits strength needed to minimize the deviation from nominal model  190 quad circuits Optics Measurement Input Output ML model Explain optic corrections as ML process: currently we use RM for the optics correction – computing the response of the optics to the change of magnets strengths using unperturbed model. For corrections, we measure perturbed optics and try to find the corrector delta values required to correct the perturbations. So, for ML model the input should be the perturbed optics, output- corrector strength values. Here the problem is – for supervised learning, we need input-output pairs. So, the required corrector values should be known in advance. How to generate such a training set?

Idea and problem definition Optics correction: identify correctors strength changes needed to minimize the deviation from nominal model  190 quad circuits Optics Measurement Input Output ML model Regression Model, Supervised Learning Explain optic corrections as ML process: currently we use RM for the optics correction – computing the response of the optics to the change of magnets strengths using unperturbed model. For corrections, we measure perturbed optics and try to find the corrector delta values required to correct the perturbations. So, for ML model the input should be the perturbed optics, output- corrector strength values. Here the problem is – for supervised learning, we need input-output pairs. So, the required corrector values should be known in advance. How to generate such a training set?

Idea and problem definition Optics correction: identify correctors strength changes needed to minimize the deviation from nominal model  190 quad circuits Optics Measurement Input Output ML model Regression Model, Supervised Learning Input: 1046 BPMs measurements of phase advance (deviation from the design) Output: 190 quad circuits (strength change values)   Simulations dataset, 100 000 input-output pairs Explain optic corrections as ML process: currently we use RM for the optics correction – computing the response of the optics to the change of magnets strengths using unperturbed model. For corrections, we measure perturbed optics and try to find the corrector delta values required to correct the perturbations. So, for ML model the input should be the perturbed optics, output- corrector strength values. Here the problem is – for supervised learning, we need input-output pairs. So, the required corrector values should be known in advance. How to generate such a training set?

Generation of training data Phase advance measured at 1046 BPMs from ideal optics Ideal optics 190 errors in quad circuits (not in a single quadrupole) MAD-X Phase advance measured at 1046 BPMs from perturbed optics Perturbed optics

Generation of training data Phase advance measured at 1046 BPMs from ideal optics Ideal optics 190 errors in quad circuits (not in a single quadrupole) Difference MAD-X Phase advance measured at 1046 BPMs from perturbed optics Perturbed optics

Generation of training data Phase advance measured at 1046 BPMs from ideal optics Ideal optics Correlation! 190 errors in quad circuits (not in a single quadrupole) Difference ML MAD-X Phase advance measured at 1046 BPMs from perturbed optics Perturbed optics

Training with simulated optics perturbed with circuits 190 qud circuits Phase advance measured at 1106 BPMs from ideal optics Phase advance measured at 1106 BPMs from perturbed optics Difference Input Output ML model

… and once the model has been trained: prediction on optics perturbed with single quadrupoles Magnitude of the quadrupole field errors is 10-6 m2 Gaussian error distribution with σ=3 IP triplets and skew quadrupoles are excluded since these errors are not concern of global correction Phase advance (input) is given gaussian noise of 10-3 2π

Several regression models have been used, e. g Several regression models have been used, e. g. Convolutional Neural Network Used for image processing Looks for spatially depended features  optics is concerned by phase advance between neighboring BPMs Different deep layers look for different features Keras with TensorFlow backend VERY simple model is applied: no parameter tuning, no optimization  lots of improvements are possible

 Even a non-optimized CNN performs on the level of Response Matrix Simulation  Even a non-optimized CNN performs on the level of Response Matrix

Comparison of β-beating averaged over 100 simulations. The measurements are simulated using β∗= 40 cm optics from 2016 for Beam 1 All methods demonstrate similar performance

Comparison of β-beating averaged over 100 simulations. The measurements are simulated using β∗= 40 cm optics from 2016 for Beam 1 All methods demonstrate similar performance Linear Regression ML model achieves best correction

Comparison of β-beating averaged over 100 simulations. The measurements are simulated using β∗= 40 cm optics from 2016 for Beam 1 All methods demonstrate similar performance Linear Regression ML model achieves best correction Linear Regression using ML vs. Response Matrix approach: Both methods are very similar However, the way how we build the model is different: ML models compute a kind of average response that is good for all training samples which contain perturbations introduced by several magnets Response Matrix computes optics response of a single change in a single magnet RM: calculated by applying small changes per magnet and evaluating the difference quotients of the observables of the resulting model via MAD-X. This needs to be done for each magnet (or variable) independently

Problems and possible improvements Bottle neck is the generation of training data using MAD-X (100 000 samples ~ 2 days) Cross –Validation is needed for model parameter tuning in order to optimize CNN  computationally costly task: CNN implemented in Keras can make use of GPUs Training time was an issue only for CNN - simpler regression models take few second to train  No additional resources needed.

Outlook Create a larger dataset in order to build more general models for different optics settings Problem is mostly linear for now. Add more sources of errors  ML model will learn new correlations (coupling, non-linear corrections)  possibly all corrections in one step Study intermediate information representation between the layers  Potentially new correlations or observables can be discovered. Reinforcement Learning Action: strength change in correctors Reward: the reduction of β-beating Environment: MAD-X Generation of dataset is not needed prior to training.

Thank you for your attention!

Backup slides

 How does it affect the quality of optics measurements? Faulty BPMs detection in 2018: Summary of 10 optics measurements obtained under different optics settings * Considering the spikes in reconstructed phase advance and beta beating IF detected BPMs which cause unphysical values in the optics Some of BPMs without obvious spikes are removed as well  How does it affect the quality of optics measurements?

Problems: Are the removed “good” BPMs really good? How many of detected bad BPMs are actually bad (in terms of signal) “Contamination” (the fraction of faulty BPMs) as parameter of IF algorithm  unknown, trade-off between cleaning and removal of good BPMs Simulations with defined bad BPMs: Simulate turn-by-turn signal without any defects Assign known defects to 10% of BPMs randomly Perform IF Compare the result to the known bad BPMs Adjustment of contamination factor Method evaluation

after 0.05 the number of removed bad BPMs increases slower than the number of removed good BPMs Averaged results on 20 simulated measurements using contamination factor = 0.05 The amount of actual good BPMs removed by IF is insignificantly small compared to amount of identified faults