1. SNS Status Report
Karen S. White
10/15/08

2. EPICS Collaboration Meeting October 15-17, 2008
SNS Background
Spallation Neutron Source
World’s most powerful pulsed neutron source
Built by ORNL and 5 partner labs
Construction complete (CD-4) April 2006
Current operating power ~625 kW
Full operating power 1.4MW
Multiyear ramp up to full production capability
Full power, high availability, max hours
24 instruments on 18 beamlines
Planning power upgrade, 2nd target, current upgrade
EPICS Collaboration Meeting October 15-17, 2008

3. EPICS Collaboration Meeting October 15-17, 2008
Controls
Large operational EPICS system
~180 VME IOCs (VxWorks)
~80 Soft IOCs (Linux)
~320 Diagnostics IOCs (Windows)
~210 PLCs
~500,000 Process Variables, including ~125,000 for diagnostics
Estimated 100,000 channels
EPICS Collaboration Meeting October 15-17, 2008

4. Production Ramp Up Plan
FY08 Actual
72.0
2807
4032
950
Availability (%)
Neutron Production Hours
Total Operating Hours
Integrated Beam Power Commitment [MW-hrs]
FY07
75.0
1500
3500
117
FY08
82.5
2700
4000
887
FY09
90.5
5000
3575
FY10
92.5
4300
5133
FY11
94.5
4500
5350
EPICS Collaboration Meeting October 15-17, 2008

5. EPICS Collaboration Meeting October 15-17, 2008
SNS Culture Change
Operations
Commissioning
Construction
“Make it work right
all the time”
“Just build it”
(on schedule & budget)
“Make it work”
EPICS Collaboration Meeting October 15-17, 2008

6. EPICS Collaboration Meeting October 15-17, 2008
Phases
Construction
Budget and Schedule rule
Technical deliverables sometimes compromised to stay on schedule or within budget
Commissioning
Demonstration that technical systems can operate, often at less than full design specifications
Learn how to operate and get diverse systems to work together
Identify and fix problems, then run some more
EPICS Collaboration Meeting October 15-17, 2008

7. EPICS Collaboration Meeting October 15-17, 2008
Phases
Operations
Repeatability, Reliability, Availability, Operability
Deliver neutrons to users
Gradual ramp up to full power
In parallel
Upgrade, improve or repair problematic systems
Push for higher availability
Controls Availability
Goal – Downtime < 50% of FY07
Actual – FY08 downtime was 37% of FY07
EPICS Collaboration Meeting October 15-17, 2008

8. All Systems Downtime Hours
EPICS Collaboration Meeting October 15-17, 2008

9. Controls Downtime Hours
EPICS Collaboration Meeting October 15-17, 2008

10. Controls Availability
Analyze causes of past downtime and target improvements to address biggest loss and highest frequency problems
Downtime
Not due to EPICS
Hardware failures
Application programs
Unclear interfaces  shooting the messenger
Controls can also help other systems improve availability
EPICS Collaboration Meeting October 15-17, 2008

11. EPICS Collaboration Meeting October 15-17, 2008
Infrastructure
Network, servers, file storage
Failures poses biggest risk for widespread outage with potentially long recovery times
Backups, patching, file organization, bumpless reboots, hot spares
Improve configuration control, documentation and system monitoring
Decouple services for accelerator operations from ORNL IT (e.g. Oracle)
EPICS Collaboration Meeting October 15-17, 2008

12. EPICS Collaboration Meeting October 15-17, 2008
Alarms
Operations does not like our alarm system (not alh)
Alarms based on automatically generated EPICS IOC logic and edm screens
Monitors ~12,000 Process Variables
Selected alarms are audibly annunciated
Non-annunciated alarms are basically ignored
Getting from a summary alarm indication to the root cause requires drill down through numerous screen
System is difficult to maintain and expand
Difficult to define what is an alarm; get good limits and configurations
EPICS Collaboration Meeting October 15-17, 2008

13. EPICS Collaboration Meeting October 15-17, 2008
New Developments
New controls back-end developments use:
CSS/Eclipse/Java
Oracle database for configuration and logging
Other talks at this meeting
New Archiver (Kay Kasemir)
Databrowser (Kay Kasemir)
Database (Dave Purcell)
Alarms (Kay Kasemir)
XAL (Dave Purcell for Tom Pelaia)
CAML (Dave Purcell for Tom Pelaia)
EPICS Collaboration Meeting October 15-17, 2008

14. Controls Improvement Projects
PPS MEBT Beam Stop Mode
Allows downstream tunnel entry during front end operations
Timing
Improve maintainability and reliability; reduce number of modules
Design 3 new timing system boards
Master – replaces 20 existing VME cards (1)
Receiver - FPGA based board (25)
Fan-out – distributes low jitter signals to new receivers
Vacuum
Improve maintainability and reliability
Replace custom chassis and interlock modules with standard PLC controls
EPICS Collaboration Meeting October 15-17, 2008

15. EPICS Collaboration Meeting October 15-17, 2008
Hiring
Transition from construction to operations, need different skills mix
Now hiring software and hardware engineers
jobs.ornl.gov select Neutron Science
EPICS Collaboration Meeting October 15-17, 2008