Accelerator Reliability and Software

Slides:



Advertisements
Similar presentations
New Radiation Safety Interlock System for the SPring-8 Accelerator Complex C. Saji, M. Toko, T. Matsushita, R. Furuta, H. Hanaki, S. Hashimoto 1), Y. Hashimoto,
Advertisements

LCLS Control System Overview Mike Zelazny SLAC ICD Software Department Accelerator Directorate.
Manufacturing Productivity Solutions Management Metrics for Lean Manufacturing Companies Total Productive Maintenance (T.P.M.) Overall Equipment Effectivity.
Network Management Overview IACT 918 July 2004 Gene Awyzio SITACS University of Wollongong.
Access 2007 Product Review. With its improved interface and interactive design capabilities that do not require deep database knowledge, Microsoft Office.
Ernest L. Williams Jr. FAC November 2008 Control Systems Software Section Configuration Management Ernest Williams.
SNS Integrated Control System SNS Machine Protection System EPICS Workshop April 27, 2005 Coles Sibley.
Patch Management Module 13. Module You Are Here VMware vSphere 4.1: Install, Configure, Manage – Revision A Operations vSphere Environment Introduction.
by Evolve IP Managed Services
INTRODUCTION TO CLOUD COMPUTING Cs 595 Lecture 5 2/11/2015.
Terri Lahey EPICS Collaboration Meeting June June 2006 LCLS Network & Support Planning Terri Lahey.
Software Engineering Modern Approaches
SNS Integrated Control System EPICS Collaboration Meeting SNS Machine Protection System SNS Timing System Coles Sibley xxxx/vlb.
© 2010 VMware Inc. All rights reserved Patch Management Module 13.
Quality Attributes of Web Software Applications – Jeff Offutt By Julia Erdman SE 510 October 8, 2003.
CLS: Control System E. Matias, R. Berg, G. Wright, T. Wilson, T. Johnson, R. Tanner and H. Zhang Canadian Light Source University of Saskatchewan Invited.
Richard Farnsworth –Controls and IT Manager Australian synchrotron OUTSOURCING, INSOURCING, AND INTEGRATION OF CONTROL SYSTEMS IN THE AUSTRALIAN SYNCHROTRON.
1 SEG4912 University of Ottawa by Jason Kealey Software Engineering Capstone Project Tools and Technologies.
Engr. M. Fahad Khan Lecturer Software Engineering Department University Of Engineering & Technology Taxila.
Dec 8-10, 2004EPICS Collaboration Meeting – Tokai, Japan MicroIOC: A Simple Robust Platform for Integrating Devices Mark Pleško
E.Matias Canadian Light Source. Where is Saskatoon?
1 Kenneth Osborne, 9/14/07 Inter Group communications at the Advanced Light Source. Overview of the different methods of communication between different.
(Sub)Version Control. 2 Keep large teams working on the same code Back up your work so you don't lose it all Compare changes to previous versions Revert.
Elder Matias, Diony Medrano, Dong Liu (At Michigan State University - Nov 2011) IRMIS at CLS.
CLS Control System Progress Report Elder Matias Canadian Light Source University of Saskatchewan Saskatoon Saskatchewan.
March 2008EPICS Meeting in Shanghai1 KEKB Control System Status Mar Tatsuro NAKAMURA KEKB Control Group, KEK.
Dave Muirhead Director of Electronic Customer Self-Service Oracle Corporation.
Integrating EPICS and LabVIEW on Windows using DCOM Freddie Akeroyd ISIS Computing Group.
Control Hardware Design Control Hardware Wiring IOC Software (Low Level) OPI Software (High Level) Final Test Front-End (Connection to SR1 Ring) Done Insertion.
Reconfigurable Communication Link Between FASTER and RTSim Interface Matthew McCollum Mark Krause Derek Keibler.
EPICS Release 3.15 Bob Dalesio May 19, Features for 3.15 Support for large arrays - done for rsrv in 3.14 Channel access priorities - planned to.
WCL303 Business Desktop Deployment (BDD) 2007: Part 2, Deploying the 2007 Office system Michael Niehaus Systems Design Engineer Microsoft
HNDIT23082 Lecture 06:Software Maintenance. Reasons for changes Errors in the existing system Changes in requirements Technological advances Legislation.
A U.S. Department of Energy Office of Science Laboratory Operated by The University of Chicago Argonne National Laboratory Office of Science U.S. Department.
EPICS Release 3.15 Bob Dalesio May 19, Features for 3.15 Support for large arrays Channel access priorities Portable server replacement of rsrv.
Jefferson Lab Report Karen S. White 11/14/00. Overview  Status of Jefferson Lab Control System  Work In Progress  Transitioning to Operations.
1 The ILC Control Work Packages. ILC Control System Work Packages GDE Oct Who We Are Collaboration loosely formed at Snowmass which included SLAC,
An Overview of Support of Small Embedded Systems with Some Recommendations Controls Working Group April 14, 2004 T. Meyer, D. Peterson.
INNOVATE THROUGH MOTIVATION MSP Services Overview KEVIN KIRKPATRICK – OWNER, MSP INC LOGO.
11 th February 2008Brian Martlew EPICS for MICE Status of the MICE slow control system Brian Martlew STFC, Daresbury Laboratory.
Control System Considerations for ADS EuCARD-2/MAX Accelerators for Accelerator Driven Systems Workshop, CERN, March 20-21, 2014 Klemen Žagar Robert Modic.
Final Fantasy ½: IOC Overview Team Q CSE 403 Winter ‘03 I’ve got something special for you.
Operations Machine Simulator.
CompTIA Security+ Study Guide (SY0-401)
SESAME Control System Status
Manufacturing Productivity Solutions
Redcell™ Management Essentials, Juniper Networks Enterprise Edition
InGenius Connector Enterprise Microsoft Dynamics CRM
Accelerator control at iThemba LABS
Critical Security Controls
Constructing Deploying and Maintaining Enterprise Systems
SNS Status Report Karen S. White 10/15/08.
Beam-Synchronous Data Acquisition (BS-DAQ)
Chapter 18 Maintaining Information Systems
Database High-Level Overview
DLS Standard PLC System
The ILC Control Work Packages
Concurrent Version Control
Nessus Vulnerability Scanning
SharePoint Administrative Communications Planning: Dynamic User Notifications for Upgrades, Migrations, Testing, … Presented by Robert Freeman (
Networks Software.
5 Tips for Upgrading Reports to v 6.3
What Do I Want from an RDB?
JENKINS TIPS Ideas for making your life with Jenkins easier
Lecture 06:Software Maintenance
Hardware-less Testing for RAS Software
EPICS: Experimental Physics and Industrial Control System
Automation of Control System Configuration TAC 18
TANGO from an EPICS perspective
DOE Review of the LCLS Project 7-9 February 2006
Presentation transcript:

Accelerator Reliability and Software A Controls Engineer’s perspective

INTRODUCTION A bit about me Controls system overview Controls faults statistics What we did well What we might to better Acknowledgments Questions

MACHINE CONTROLS SYSTEM OVERVIEW Uses EPICS (3.14.8.2 though to 3.14.12-2) 175 IOCs 174 Linux (CentOS, Debian, Libera), 1 Windows 12 OPI boxes (8 recent replacements) Delphi, EDM, MEDM, Alarm Handler, MatLab 8 PLCs 1 Archiver (27000+ PVs) 1 File server 28 MOXA terminal servers 4 Gateways boxes – 27 EPICS gateways Network separation – separate sub nets for accelerator, accelerator EPS, each beamline, build network, integration lab, office, wireless, etc.

CONTROLS FAULTS YEARLY ANALYSIS 2007 2008 2009 2010 2011 2012 Number of Faults 17 7 3 8 5 Total Downtime (h:mm) 14:34 15:24 6:47 4:18 9:05 6:46 MTBF (h:mm) 157:40 278:00 699:42 1619:45 612:15 987:00 MDT (h:mm) 0:51 0:54 0:58 1:26 1:08 1:21 % of Total Downtime (10.3%) 9.50% 13.00% 7.41% 5.79% 11.37% 14.68%

PERCENTAGE DOWNTIME – CONTROLS FAULTS Controls Fault Break Down

GOOD STUFF 1 Selection of EPICS as control system (2004) Good community support (teck-talk) Collaboration meeting twice a year. Access control lists (Top-up related IOCs only so far). A good team for initial development Good interface with accelerator/operations team for on going maintenance and development. Good response times for issues New functionality integrated in timely manor Consistent OPI interface helps here.

USED SYSTEMS ENGINEERING PROCESS 1 Captured functional requirements for the Storage Ring Injection System, RF delivered by 3rd party vendors. Creates a System Breakdown Structure (SBS) Produced and reviewed design documentation for major sub-systems e.g. Vacuum, Magnets, Front End Control, Diagnostics etc. Produced system block diagrams Produced and reviewed interface specifications. Produced system test requirement specifications

USED SYSTEMS ENGINEERING PROCESS 2

USED SYSTEMS ENGINEERING PROCESS 3

CONFIG MANAGED BUILD SYSTEM Automated Uniform build and deploy process. Quarantined build boxes (avoids side effects). All source code extracted from source code repository (perforce) Only source code changes linked to bug tracking system (bugzilla) available for build. Completed build committed to build repository But not locked down so as prevent patching and/or development builds. Trust system.

ACCELERAOR OPERATOR INTERFACE Reliable User Interface Operators like it Attributes: Simple, intuitive, comprehensive, consistent. Not so good: Essentially only I have the in depth of knowledge. Also not embraced by the EPICS community. Use of confirmation dialogs on some actions – but we don’t over do it. Cosmetically pleasing – not bland Make training easier Provides a PV value configuration management capability E.g. magnet power supply and timing system settings.

OPI EXAMPLE 1

OPI EXAMPLE 2

NOT SO GOOD STUFF / RISKS 1 Turnkey systems. Quick to get going but… Lack hands-on experience with some items (e.g. PLCs) Extra work to maintain and modify. Needs some reverse engineering. This impact time to fix. Often we mitigate a problem rather than doing a proper fix (band-aid solutions) Still work, but not always ideal. Always in catch up mode for h/w and s/w to provided advanced functionality.

NOT SO GOOD STUFF / RISKS 2 Accelerator file server failure Highlighted a hole is the IT backup process – need to be vigilant. Some IOC now 7+ years old (e.g. 30 uIOCs) Relatively small maintenance team Utter dependence on the network. Recent network upgrade caused havoc, not all implications considered (duplicate routing tables). Last year, a security audit involving full port scan swamped the 10MHz link to EPS which then didn’t know state of shutters which caused a beam dump.

ACKNOWLEDGEMENTS Joel Trewhella (statistics and associated graphics) Don McGilvery.