SNS Status Report Karen S. White 10/15/08

EPICS Collaboration Meeting October 15-17, 2008 SNS Background Spallation Neutron Source World’s most powerful pulsed neutron source Built by ORNL and 5 partner labs Construction complete (CD-4) April 2006 Current operating power ~625 kW Full operating power 1.4MW Multiyear ramp up to full production capability Full power, high availability, max hours 24 instruments on 18 beamlines Planning power upgrade, 2nd target, current upgrade EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 Controls Large operational EPICS system ~180 VME IOCs (VxWorks) ~80 Soft IOCs (Linux) ~320 Diagnostics IOCs (Windows) ~210 PLCs ~500,000 Process Variables, including ~125,000 for diagnostics Estimated 100,000 channels EPICS Collaboration Meeting October 15-17, 2008

Production Ramp Up Plan FY08 Actual 72.0 2807 4032 950 Availability (%) Neutron Production Hours Total Operating Hours Integrated Beam Power Commitment [MW-hrs] FY07 75.0 1500 3500 117 FY08 82.5 2700 4000 887 FY09 90.5 5000 3575 FY10 92.5 4300 5133 FY11 94.5 4500 5350 EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 SNS Culture Change Operations Commissioning Construction “Make it work right all the time” “Just build it” (on schedule & budget) “Make it work” EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 Phases Construction Budget and Schedule rule Technical deliverables sometimes compromised to stay on schedule or within budget Commissioning Demonstration that technical systems can operate, often at less than full design specifications Learn how to operate and get diverse systems to work together Identify and fix problems, then run some more EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 Phases Operations Repeatability, Reliability, Availability, Operability Deliver neutrons to users Gradual ramp up to full power In parallel Upgrade, improve or repair problematic systems Push for higher availability Controls Availability Goal – Downtime < 50% of FY07 Actual – FY08 downtime was 37% of FY07 EPICS Collaboration Meeting October 15-17, 2008

All Systems Downtime Hours EPICS Collaboration Meeting October 15-17, 2008

Controls Downtime Hours EPICS Collaboration Meeting October 15-17, 2008

Controls Availability Analyze causes of past downtime and target improvements to address biggest loss and highest frequency problems Downtime Not due to EPICS Hardware failures Application programs Unclear interfaces  shooting the messenger Controls can also help other systems improve availability EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 Infrastructure Network, servers, file storage Failures poses biggest risk for widespread outage with potentially long recovery times Backups, patching, file organization, bumpless reboots, hot spares Improve configuration control, documentation and system monitoring Decouple services for accelerator operations from ORNL IT (e.g. Oracle) EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 Alarms Operations does not like our alarm system (not alh) Alarms based on automatically generated EPICS IOC logic and edm screens Monitors ~12,000 Process Variables Selected alarms are audibly annunciated Non-annunciated alarms are basically ignored Getting from a summary alarm indication to the root cause requires drill down through numerous screen System is difficult to maintain and expand Difficult to define what is an alarm; get good limits and configurations EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 New Developments New controls back-end developments use: CSS/Eclipse/Java Oracle database for configuration and logging Other talks at this meeting New Archiver (Kay Kasemir) Databrowser (Kay Kasemir) Database (Dave Purcell) Alarms (Kay Kasemir) XAL (Dave Purcell for Tom Pelaia) CAML (Dave Purcell for Tom Pelaia) EPICS Collaboration Meeting October 15-17, 2008

Controls Improvement Projects PPS MEBT Beam Stop Mode Allows downstream tunnel entry during front end operations Timing Improve maintainability and reliability; reduce number of modules Design 3 new timing system boards Master – replaces 20 existing VME cards (1) Receiver - FPGA based board (25) Fan-out – distributes low jitter signals to new receivers Vacuum Improve maintainability and reliability Replace custom chassis and interlock modules with standard PLC controls EPICS Collaboration Meeting October 15-17, 2008

EPICS Collaboration Meeting October 15-17, 2008 Hiring Transition from construction to operations, need different skills mix Now hiring software and hardware engineers jobs.ornl.gov select Neutron Science EPICS Collaboration Meeting October 15-17, 2008