Controls Debbie Rogind EPICS Collaboration Oct SLAC LCLS First Year in Operation and Controls Lessons Learned Debbie Rogind SLAC Controls Software

Controls Debbie Rogind EPICS Collaboration Oct Outline  LCLS Status –Commissioning Success –Enhancements in Progress  Controls Lessons Learned  LCLS Future Plans –LCLS II –R & D

Controls Debbie Rogind EPICS Collaboration Oct X-ray FEL uses last 1-km of existing 3-km linac Injector (35º) at 2-km point Existing 1/3 Linac (1 km) (with modifications) New e  Transfer Line (340 m) Undulator (130 m) Electron Beam Dump Linac Coherent Light Source at SLAC

Controls Debbie Rogind EPICS Collaboration Oct SLAC/LCLS Main Control Center (MCC) Gun brightness beats LCLS Goals Max x-ray pulse energy of 4.5 mJ surpasses design of 2.0 mJ (6.4 Gev) FEL is fully tunable in rep rate, pulse length, photon energy, peak power,…

Controls Debbie Rogind EPICS Collaboration Oct Linac-to-Undulator (340m)

Controls Debbie Rogind EPICS Collaboration Oct Undulator Hall (130m) - 33 Undulator Segments Beam saturates in 60m rather than 90m; 25 of 33 undulators are installed

Controls Debbie Rogind EPICS Collaboration Oct Beam Dump

Controls Debbie Rogind EPICS Collaboration Oct LCLS Commissioning Success ( ) M M A A M M J J J J A A S S D D LTU/Undulator First Light in FEE First Light in FEH NEH J J F F M M A A M M J J J J A A S S D D J J F F M M A A M M J J J J A A S S O O N N D D J J F F M M A A M M J J J J ON A A Linac/BC2 FEL/FEE Install Undulators M M D D PEP-II ends X-Rays in NEH D D ON Injector/BC1 Down First e  to dump First e  from gun Install BC2 First e  through linac First lasing CD-4 Review ~ 4 years Gradual migration from Legacy Most BPMs & Magnets EPICS Most BPMs and Magnets are EPICS All BPMs and Magnets are EPICS, Linac Upgrade in Progress Mix of Legacy and EPICS new devices A A S S O O Drive Laser Installation (Aug-Nov, 2006) Today Challenging combination of Legacy and EPICS First Users First Year Of Operations EPICS new devices

Controls Debbie Rogind EPICS Collaboration Oct LCLS - all EPICS early 2011  Linac Upgrade underway (T Himel) –Move all critical functionality off of legacy VMS system and onto Linux –Legacy micros replaced with VME IOCs –Move existing CAMAC control to VME module –RF control and timing control challenging –100% EPICS early 2011! Status

Controls Debbie Rogind EPICS Collaboration Oct Linac Upgrade Status

Controls Debbie Rogind EPICS Collaboration Oct Hz Operation Jan Hz Fast Feedback (D Fairley) –Dedicated FNET multicast network; FCOM protocol (T. Straumann) –Pattern Aware Unit (PAU); RF Control (K Kim) –Replaces MATLAB feedbacks 120Hz Beam Synch Acquisition plus MPS work fb03 LI21 LI24BSYLTU core IN20 Li22LI23Li25LI26LI27Li28LI29LI30UNDDMP bp01 bp02 rf01 bp01bp bl01 1 Controller IOCs fb02 fb01 LTU1 mg01 rf01bl01bp02 DMP bp01 UND bp03 UND bp04 UND bpo1 UND bp02 LTU1 bp04 LTU1 bp03 LTU1 bp02 LTU1 bp01 LTU0 bp01 BSY bp01 BSY bp RF EIOCs LTU0 mg01 Status

Controls Debbie Rogind EPICS Collaboration Oct Total IOCs

Controls Debbie Rogind EPICS Collaboration Oct LCLS Controls Success  Major commitment to EPICS based control –Sept 2010: 465 Total IOCs 165 VME IOCs, 218 EIOCs, 82 Soft IOCs Total PVs: 697,330, and growing with linac upgrade 115,000 PVs archived, and growing with linac upgrade  Control system ready on time  Great collaboration with operations and physicists  Reliability/Availability is unusually high Status

Controls Debbie Rogind EPICS Collaboration Oct LCLS Controls Availability – Last User Run Controls Dept Goal: 98.8% Status

Controls Debbie Rogind EPICS Collaboration Oct Controls Lessons Learned  EPICS and Legacy System Integration  Physicist Contributions  Controls HLA development  IOC Development  Controls System Monitoring  Policy Lessons

Controls Debbie Rogind EPICS Collaboration Oct EPICS and Legacy  Move off of legacy system sooner –Incorrectly assumed existing legacy system physics applications would dominate Years of effort expended to develop complex EPICS-based bridge that provides legacy system with EPICS data not worth the large effort –Underestimated the utility of AIDA (Accelerator Independent Data Access, G White) Provides legacy data to EPICS (more lucrative direction) MATLAB + AIDA + VMS based CAS + labCA bridges the gap between legacy and EPICS. Physicists can rule! Lessons

Controls Debbie Rogind EPICS Collaboration Oct Harvest the Power of Physicists  Harvest the Power of Physicists –Provide MATLAB / EPICS infrastructure sooner –Look for synergy between physicists and Controls - not competition Lessons

Controls Debbie Rogind EPICS Collaboration Oct MATLAB High Level Physics Apps …

Controls Debbie Rogind EPICS Collaboration Oct Matlab Support PVs Soft IOC framework for physicists - Banks of user-configurable PV records for physics data (Mike Z)

Controls Debbie Rogind EPICS Collaboration Oct Harvest the Power of Physicists  PROS for physicists’ MATLAB GUIs –Makes use of a huge resource of physicists; –Domain knowledge; no need to translate requirements to engineers –Rapid prototyping – agile methodologies –Automatic access to machine time / shifts –GUIs are intuitive  CONS for physicists’ MATLAB GUIs –No structured requirements, design reviews; limited documentation –No code reviews; code can be difficult to follow –Reliability issues –Limited support beyond original author –What happens to the maintenance after the physicists transfer to another project? Lessons

Controls Debbie Rogind EPICS Collaboration Oct Controls Java-based HLA  Java-based apps built from common jars with GUIs are too heavy-weight and feature rich (complex) –JVM tuning is challenging –Ops desire performance and reliabilty over feature rich –Evaluate service based architecture  XAL Model roll-out took more effort than expected –“Like fitting a square peg into a round hole”, Mark Woodley –Physicists went ahead and developed their own MATLAB LEM and Model Lessons

Controls Debbie Rogind EPICS Collaboration Oct Controls Model Based Apps - LEM P Chu, G White et al

Controls Debbie Rogind EPICS Collaboration Oct Controls Model Based Apps – Model Manager P Chu, et al

Controls Debbie Rogind EPICS Collaboration Oct Controls HLA - MPS S Chevstov, S Norum et al

Controls Debbie Rogind EPICS Collaboration Oct IOC development  Place requirements, design review and documentation in all project schedules from beginning  Lack of code reviews at all levels (driver,…,DB) cause cleanup work later; can cause downtime  EPICS IOC configuration and deployment standards needed day 1  Naming standards verification tool needed to enforce naming convention  Tight schedules trump above! Lessons

Controls Debbie Rogind EPICS Collaboration Oct IOC development  Underestimated RTEMS skill set required  Train RTEMS personnel earlier for personnel load balancing –Drivers, Board support  Strong OSI policy needed day 1 for device support Lessons

Controls Debbie Rogind EPICS Collaboration Oct  Require “QA” IOC external interface checklist prior to IOC deployment – gets left undone –Publish IOC APIs ! –Establish archiver policy - rates, PV fields (.EGU,.LOLO, etc ) –Complete configs/integration for RDB, SCORE, IRMIS, ALH, EDM, autosave, archiver, message logging, Help System –“Boot Fests” were required to ensure IOC restores to desired state invalid PV, autosave parameters plagued us  Manual and disparate client configurations above should be automated  Archive Engine reliability /scaleability a surprise –Forced to establish rules for PVs that should/should not be archived IOC Interfaces Lessons

Controls Debbie Rogind EPICS Collaboration Oct Control Systems Monitoring  Engineers need a more thorough monitoring and alert mechanism to inform them of errant / alarmed IOC or linux system process prior to being contacted by Operations Lessons

Controls Debbie Rogind EPICS Collaboration Oct Policy  All software, no matter how small the modification, needs a code review prior to release into production  Every subsystem engineer needs a trained backup  Comprehensive test plan for any software release is required. All test plans are reviewed by physicists and operations for approval prior to release into production. Requires lots of time!  Every test plan requires a backout plan  Controls Deputy (Controls SW engineers rotate weekly) coordinates test plan /approvals /scheduling/ oversight. Lessons

Controls Debbie Rogind EPICS Collaboration Oct Policy  All software should be scheduled via one of the following mechanisms: –CATER (trouble report from user) –Software Request Database –An approved Project schedule –Supervisor request –NOT in the “hallway” from a user Lessons

Controls Debbie Rogind EPICS Collaboration Oct Policy  Invest in test stands. –Needed, given the high demand for machine time. Competition especially keen after user runs started.  For new project development, relieving engineers of their support and maintenance duties gives the schedule more determinism –Worked well for LCLS! Lessons

Controls Debbie Rogind EPICS Collaboration Oct Future Looks Bright!  LCLS II  R & D Future

Controls Debbie Rogind EPICS Collaboration Oct Future is bright: LCLS II  Builds on the success of and demand for LCLS experimental time  The LCLS II will essentially be a clone of the LCLS, 10 sectors upstream, and with a challenge of supporting multiple beam lines (~$400M project)  Goal is to support more users with reliable high-quality beam  CDR work commencing  Construction start ~2012  Expected lifetime is years  Lots of R&D before that (new platform, remove CAMAC) Future

Controls Debbie Rogind EPICS Collaboration Oct LCLS II Layout undulator X L1 L2L3BC1BC2 RFgun-1 L GeV sector-11sector-21sector-24sector-14 LCLS existing enclosure exists at sector 10 und-hall L3 undulator 3-7 GeV bypass line 3-7 GeV X RFgun-2 L1 L2BC1BC2 L0 FACET wall Future

Controls Debbie Rogind EPICS Collaboration Oct Current R&D Direction  Move to a micro-TCA platform –Research already underway  Design and test controls for all items in a sector –RF (funds for major upgrade exist now) –BPM, Timing, Magnets, Toroids, Movers, etc  Propose modern upgrade for LCLS II and later LCLS as well  Team: R Larsen, Q Yang, T Himel, and many others Future

Controls Debbie Rogind EPICS Collaboration Oct Conclusion  EPICS is now assumed by everyone at SLAC for new projects  Many lessons have been and continue to be learned by SLAC Controls. –Use to establish future direction for LCLS II  SLAC Controls is actively pushing for a viable long- term controls hardware solution.  Lots of fun, challenging, work ahead! Future

Controls Debbie Rogind EPICS Collaboration Oct The End  Thank you for listening.