1. Controls Debbie Rogind EPICS Collaboration Oct. 2010 SLAC LCLS First Year in Operation and Controls Lessons Learned Debbie Rogind SLAC Controls Software

2. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Outline  LCLS Status –Commissioning Success –Enhancements in Progress  Controls Lessons Learned  LCLS Future Plans –LCLS II –R & D

3. Controls Debbie Rogind EPICS Collaboration Oct. 2010 X-ray FEL uses last 1-km of existing 3-km linac Injector (35º) at 2-km point Existing 1/3 Linac (1 km) (with modifications) New e  Transfer Line (340 m) Undulator (130 m) Electron Beam Dump Linac Coherent Light Source at SLAC

4. Controls Debbie Rogind EPICS Collaboration Oct. 2010 SLAC/LCLS Main Control Center (MCC) Gun brightness beats LCLS Goals Max x-ray pulse energy of 4.5 mJ surpasses design of 2.0 mJ (6.4 Gev) FEL is fully tunable in rep rate, pulse length, photon energy, peak power,…

5. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Linac-to-Undulator (340m)

6. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Undulator Hall (130m) - 33 Undulator Segments Beam saturates in 60m rather than 90m; 25 of 33 undulators are installed

7. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Beam Dump

8. Controls Debbie Rogind EPICS Collaboration Oct. 2010 LCLS Commissioning Success (2007-2010) M M A A M M J J J J A A S S D D LTU/Undulator First Light in FEE First Light in FEH NEH J J F F M M A A M M J J J J A A S S D D J J F F M M A A M M J J J J A A S S O O N N D D J J F F M M A A M M J J J J ON A A Linac/BC2 FEL/FEE Install Undulators M M D D PEP-II ends X-Rays in NEH D D ON Injector/BC1 Down First e  to dump First e  from gun Install BC2 First e  through linac First lasing CD-4 Review ~ 4 years Gradual migration from Legacy Most BPMs & Magnets EPICS 2007200820092010 Most BPMs and Magnets are EPICS All BPMs and Magnets are EPICS, Linac Upgrade in Progress Mix of Legacy and EPICS new devices A A S S O O Drive Laser Installation (Aug-Nov, 2006) Today Challenging combination of Legacy and EPICS First Users First Year Of Operations EPICS new devices

9. Controls Debbie Rogind EPICS Collaboration Oct. 2010 LCLS - all EPICS early 2011  Linac Upgrade underway (T Himel) –Move all critical functionality off of legacy VMS system and onto Linux –Legacy micros replaced with VME IOCs –Move existing CAMAC control to VME module –RF control and timing control challenging –100% EPICS early 2011! Status

10. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Linac Upgrade Status

11. Controls Debbie Rogind EPICS Collaboration Oct. 2010 120Hz Operation Jan 2011 120Hz Fast Feedback (D Fairley) –Dedicated FNET multicast network; FCOM protocol (T. Straumann) –Pattern Aware Unit (PAU); RF Control (K Kim) –Replaces MATLAB feedbacks 120Hz Beam Synch Acquisition plus MPS work fb03 LI21 LI24BSYLTU core IN20 Li22LI23Li25LI26LI27Li28LI29LI30UNDDMP bp01 bp02 rf01 bp01bp02 1027 76 bl01 1 Controller IOCs fb02 fb01 LTU1 mg01 rf01bl01bp02 DMP bp01 UND bp03 UND bp04 UND bpo1 UND bp02 LTU1 bp04 LTU1 bp03 LTU1 bp02 LTU1 bp01 LTU0 bp01 BSY bp01 BSY bp02 1041 54 7 4 4699 8 7 9 RF EIOCs LTU0 mg01 Status

12. Controls Debbie Rogind EPICS Collaboration Oct. 2010 465 Total IOCs

13. Controls Debbie Rogind EPICS Collaboration Oct. 2010 LCLS Controls Success  Major commitment to EPICS based control –Sept 2010: 465 Total IOCs 165 VME IOCs, 218 EIOCs, 82 Soft IOCs Total PVs: 697,330, and growing with linac upgrade 115,000 PVs archived, and growing with linac upgrade  Control system ready on time  Great collaboration with operations and physicists  Reliability/Availability is unusually high Status

14. Controls Debbie Rogind EPICS Collaboration Oct. 2010 LCLS Controls Availability – Last User Run Controls Dept Goal: 98.8% Status

15. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Controls Lessons Learned  EPICS and Legacy System Integration  Physicist Contributions  Controls HLA development  IOC Development  Controls System Monitoring  Policy Lessons

16. Controls Debbie Rogind EPICS Collaboration Oct. 2010 EPICS and Legacy  Move off of legacy system sooner –Incorrectly assumed existing legacy system physics applications would dominate Years of effort expended to develop complex EPICS-based bridge that provides legacy system with EPICS data not worth the large effort –Underestimated the utility of AIDA (Accelerator Independent Data Access, G White) Provides legacy data to EPICS (more lucrative direction) MATLAB + AIDA + VMS based CAS + labCA bridges the gap between legacy and EPICS. Physicists can rule! Lessons

17. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Harvest the Power of Physicists  Harvest the Power of Physicists –Provide MATLAB / EPICS infrastructure sooner –Look for synergy between physicists and Controls - not competition Lessons

18. Controls Debbie Rogind EPICS Collaboration Oct. 2010 MATLAB High Level Physics Apps …

19. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Matlab Support PVs Soft IOC framework for physicists - Banks of user-configurable PV records for physics data (Mike Z)

20. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Harvest the Power of Physicists  PROS for physicists’ MATLAB GUIs –Makes use of a huge resource of physicists; –Domain knowledge; no need to translate requirements to engineers –Rapid prototyping – agile methodologies –Automatic access to machine time / shifts –GUIs are intuitive  CONS for physicists’ MATLAB GUIs –No structured requirements, design reviews; limited documentation –No code reviews; code can be difficult to follow –Reliability issues –Limited support beyond original author –What happens to the maintenance after the physicists transfer to another project? Lessons

21. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Controls Java-based HLA  Java-based apps built from common jars with GUIs are too heavy-weight and feature rich (complex) –JVM tuning is challenging –Ops desire performance and reliabilty over feature rich –Evaluate service based architecture  XAL Model roll-out took more effort than expected –“Like fitting a square peg into a round hole”, Mark Woodley –Physicists went ahead and developed their own MATLAB LEM and Model Lessons

22. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Controls Model Based Apps - LEM P Chu, G White et al

23. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Controls Model Based Apps – Model Manager P Chu, et al

24. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Controls HLA - MPS S Chevstov, S Norum et al

25. Controls Debbie Rogind EPICS Collaboration Oct. 2010 IOC development  Place requirements, design review and documentation in all project schedules from beginning  Lack of code reviews at all levels (driver,…,DB) cause cleanup work later; can cause downtime  EPICS IOC configuration and deployment standards needed day 1  Naming standards verification tool needed to enforce naming convention  Tight schedules trump above! Lessons

26. Controls Debbie Rogind EPICS Collaboration Oct. 2010 IOC development  Underestimated RTEMS skill set required  Train RTEMS personnel earlier for personnel load balancing –Drivers, Board support  Strong OSI policy needed day 1 for device support Lessons

27. Controls Debbie Rogind EPICS Collaboration Oct. 2010  Require “QA” IOC external interface checklist prior to IOC deployment – gets left undone –Publish IOC APIs ! –Establish archiver policy - rates, PV fields (.EGU,.LOLO, etc ) –Complete configs/integration for RDB, SCORE, IRMIS, ALH, EDM, autosave, archiver, message logging, Help System –“Boot Fests” were required to ensure IOC restores to desired state invalid PV, autosave parameters plagued us  Manual and disparate client configurations above should be automated  Archive Engine reliability /scaleability a surprise –Forced to establish rules for PVs that should/should not be archived IOC Interfaces Lessons

28. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Control Systems Monitoring  Engineers need a more thorough monitoring and alert mechanism to inform them of errant / alarmed IOC or linux system process prior to being contacted by Operations Lessons

29. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Policy  All software, no matter how small the modification, needs a code review prior to release into production  Every subsystem engineer needs a trained backup  Comprehensive test plan for any software release is required. All test plans are reviewed by physicists and operations for approval prior to release into production. Requires lots of time!  Every test plan requires a backout plan  Controls Deputy (Controls SW engineers rotate weekly) coordinates test plan /approvals /scheduling/ oversight. Lessons

30. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Policy  All software should be scheduled via one of the following mechanisms: –CATER (trouble report from user) –Software Request Database –An approved Project schedule –Supervisor request –NOT in the “hallway” from a user Lessons

31. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Policy  Invest in test stands. –Needed, given the high demand for machine time. Competition especially keen after user runs started.  For new project development, relieving engineers of their support and maintenance duties gives the schedule more determinism –Worked well for LCLS! Lessons

32. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Future Looks Bright!  LCLS II  R & D Future

33. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Future is bright: LCLS II  Builds on the success of and demand for LCLS experimental time  The LCLS II will essentially be a clone of the LCLS, 10 sectors upstream, and with a challenge of supporting multiple beam lines (~$400M project)  Goal is to support more users with reliable high-quality beam  CDR work commencing  Construction start ~2012  Expected lifetime is 20-30 years  Lots of R&D before that (new platform, remove CAMAC) Future

34. Controls Debbie Rogind EPICS Collaboration Oct. 2010 LCLS II Layout undulator X L1 L2L3BC1BC2 RFgun-1 L0 3-15 GeV sector-11sector-21sector-24sector-14 LCLS existing enclosure exists at sector 10 und-hall L3 undulator 3-7 GeV bypass line 3-7 GeV X RFgun-2 L1 L2BC1BC2 L0 FACET wall Future

35. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Current R&D Direction  Move to a micro-TCA platform –Research already underway  Design and test controls for all items in a sector –RF (funds for major upgrade exist now) –BPM, Timing, Magnets, Toroids, Movers, etc  Propose modern upgrade for LCLS II and later LCLS as well  Team: R Larsen, Q Yang, T Himel, and many others Future

36. Controls Debbie Rogind EPICS Collaboration Oct. 2010 Conclusion  EPICS is now assumed by everyone at SLAC for new projects  Many lessons have been and continue to be learned by SLAC Controls. –Use to establish future direction for LCLS II  SLAC Controls is actively pushing for a viable long- term controls hardware solution.  Lots of fun, challenging, work ahead! Future

37. Controls Debbie Rogind EPICS Collaboration Oct. 2010 The End  Thank you for listening.