1. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 1 The LCLS Timing & Event System - An Introduction – John Dusatko / Accelerator Controls

2. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 2 Outline 1.Introduction to LCLS 2.Some background on SLAC Timing 3.The SLAC Linac Timing System 4.The LCLS Timing System 5. Issues

3. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 3 Introduction

4. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 4 LCLS Introduction The Linac Coherent Light Source is an X-ray FEL based on the SLAC Linac: 1.0nC, 14GeV e - are passed thru an undulator, a Self Amplifying Stimulated Emission process produces 1.5 Angstrom X-Rays. LCLS is an addition to the existing SLAC Linac: it uses the last 1/3 of the machine ►This is important to note because we have to integrate the New LCLS Timing System with the Existing Linac (SLC) Timing System.

5. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 5 The LCLS – Schematic View (ignoring photon beamline) Single bunch, 1-nC charge, 1.2-  m slice emittance, 120-Hz repetition rate… (RF phase:  rf = 0 is at accelerating crest) SLAC linac tunnel research yard Linac-0 L =6 m Linac-1 L  9 m  rf   25° Linac-2 L  330 m  rf   41° Linac-3 L  550 m  rf   10° BC-1 L  6 m R 56   39 mm BC-2 L  22 m R 56   25 mm LTU L =275 m R 56  0 DL-1 L  12 m R 56  0 undulator L =130 m 6 MeV  z  0.83 mm    0.05 % 135 MeV  z  0.83 mm    0.10 % 250 MeV  z  0.19 mm    1.6 % 4.54 GeV  z  0.022 mm    0.71 % 14.1 GeV  z  0.022 mm    0.01 %...existing linac new rfgun 21-1b21-1d X Linac-X L =0.6 m  rf =  21-3b24-6d25-1a30-8c

6. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 6

7. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 7 LCLS Timing – Some Definitions The LCLS Timing System can be viewed as consisting of three parts: Part 1: ‘Standard’ Accelerator Timing 10ps Triggers for Acceleration and Diagnostics Part 2: S-Band Timing 2856MHz LCLS RF Phase Reference Distribution Part 3: Ultra-Precise Timing 10fs Synchronization for Experiments (LBL System)

8. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 8 LCLS Timing – Some Definitions The LCLS Timing System can be viewed as consisting of three levels: Part 1: ‘Standard’ Accelerator Timing 10ps Triggers for Acceleration and Diagnostics ‘Triggers’ are signals from the timing system used by HW to accelerate & measure the beam Part 2: S-Band Timing 2865MHz RF Phase Reference Distribution Part 3: Ultra-Precise Timing 10fs Synchronization for Experiments (LBL System)

9. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 9 LCLS Timing – Performance Requirements LCLS Timing System Requirements Maximum trigger rate:360 Hz (120Hz) Clock frequency:119 MHz Clock precision:20 ps Delay Coarse step size:8.4 ns ± 20 ps Delay range;>1 sec Fine step size:20 ps Max timing jitter w.r.t. clock;2 ps (10ps) rms Differential error (skew), location to location: 8 ns Long term stability:20 ps Signal Level:TTL

10. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 10 Requirements Comparison Timing Reqmts for earlier SLAC systems: Original Linac: (~1968) - Resolution: 50 nSec - Jitter: 15 nSec - Main Trigger Line Waveform: + / - 400 Volts PEP – II: (~1998) - Resolution: 2.1 nSec - Jitter: 20 pSec - NIM Level Waveform: 0 to –0.7 V into 50 Ohms

11. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 11 Background

12. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 12 SLAC’s Timing Systems In order to explain the New LCLS timing system, we first need to understand how the old SLAC timing system works – i.e. how we got from there to here The SLAC Accelerator complex consists of several machines: Linac, Damping Rings, Stanford Linear Collider, PEP-II, FFTB, NLCTA / each with its own timing sub-system ►The overall timing system consists of incremental add- ons to the original system ►Design Challenge for LCLS Timing System was that it had to know about and work with the existing system

13. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 13 (pre-LCLS) SLAC Accelerator Complex ( Lots of Pieces)

14. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 14 SLAC Linac Timing System

15. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 15 Old SLAC Timing System We’ll talk a little about the existing SLAC Linac Timing System: The Linac is a Pulsed Machine (get a packet of beam per pulse) runs at a max of 360Hz Three Main Timing Signals: 476MHz Master Accelerator Clock (runs down 2mile Heliax Main Drive Line cable) 360Hz Fiducial Trigger (used to ‘tell’ devices when the beam bunch is present) / encoded onto the 476MHz master clock 128-Bit PNET (Pattern Network) Digital Broadcast

16. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 16 Some More Details Why 360Hz? Original design rate of the Linac / derived from the 3- phase, 60Hz AC power line frequency: want to trigger devices (Klystrons, etc.) consistently so as to not create huge transients on the Power Line Sync’d to 476MHz PNET Broadcast A special computer called the Master Pattern Generator (triggered by the 360Hz fiducial) broadcasts a 128-bit digital message containing information (conditions, rate, charge, etc.) about the beam This is used by the Trigger Generator computers to set up triggers and their delays Sent over SLAC coax cable TV network

17. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 17 How The 360Hz is Generated The Sequence Generator creates a 360Hz signal as well as 6 Timeslot pulses (used for further synchronization) Source: SLAC Blue Book c.1962

18. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 18 Timing HW at Head- End of Linac For Phase Stabilization

19. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 19  The 360Hz Signal is Amplitude Modulated onto the 476MHz Accelerator Clock and propagated down the Main Drive Line.  The AM process is not ideal and some FM occurs; in addition, the signal gets more dispersed as it heads down the 2 mile MDL Generation of The Linac Timing Signal (AM Modulator)

20. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 20 Linac RF Phase Reference Distribution  This slide is to give you an idea of how the Linac Phase Reference is used Each sector (30 total) taps off the MDL, to extract the RF clock This is just the Phase Ref, trigger generation is accomplished by a different set of HW

21. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 21 Old Timing System (CAMAC based) generates triggers by combining the RF Clock, 360Hz Fiducial and PNET Data Timing CAMAC Crate PNET Data on Serial Link  476MHz is divided/4 to get 119Mhz + fiducial by another system / This is because the older technology HW could not run at 476MHz  The Programmable Delay Unit (PDU) Module generates the triggers. It contains digital counters that get set with delay values from PNET and started when the Fiducial Pulse comes along Trigger Generation

22. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 22 The LCLS Timing System

23. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 23 Finally – The LCLS Timing System Old CAMAC System is no longer viable for new Systems (performance limited, obsolete) Seek to implement a new Timing System that has similar functionality, better performance, and can be laid atop the old system, working alongside it In addition, LCLS has have its own master oscillator (PLL sync’d with Linac MO) and local phase reference distribution system at S20 ►LCLS System is VME based, using High-Speed digital serial links to send Clock, Trigger and Data all on one optical Fiber to timing clients

24. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 24 LCLS RF Front End LCLS Master Osc – slaved to Linac MO / Lower Phase noise req’d by LCLS

25. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 25 DEVDEV LCLS Timing/Event System Architecture ~ Linac main drive line Sync/Div SLC MPG PNETPNET 119 MHz 360 Hz SLC events LCLS events PNETPNET PP PDUPDU EVREVR OCOC TTL-NIM convert. Digitizer LLRF BPMs Toroids Cameras Wire Scanner SLC klystrons TTL SLC Trigs FANFAN OCOC Low Level RF EPICS Network Precision<10 ps fiber distribution LCLS Timing System components are in RED *MicroResearch * * EVGEVG LCLS Master Oscillator 476 MHz Linac Master Osc System is based around the EVent Generator and EVent Receiver

26. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 26 The Event System Based on Commercial Hardware (MicroResearch Finland) Which was based on a design from ANL-APS Timing System Designed Around Xilinx Virtex-II FPGAs Uses FPGA’s internal High-Speed 2.38 Gb/s Serial xcvr, which connects to a fiber optical transceiver FPGA logic implements all of the timing functions Uses 119MHz clock from Linac which get multiplied up to by FPGA internal PLL to 2.38GHz EVG sends out 8-bit event code to EVRs along with clock and trigger information over one fiber EVR Receives event code & with its associative memory, generates a trigger with a delay set by digital counters

27. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 27 The Event System ►Upon RX’ing a 360Hz Fid, the EVG sends out a stream of serial Data to the EVRs over a fiber link. The serial stream consists of 16-bit words sent every 8.4ns. ►Each word contains an Event Code byte and some trigger setup data

28. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 28 Event Generator ►EVG contains a RAM that gets loaded with event codes, based on PNET data. 360Hz Fiducial causes the RAM to get sent out over the Serial fiber link to the EVRs.

29. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 29 Event Receiver ►EVR contains another RAM that looks for matches of event codes to its contents. If match, it starts a counter running that generates a trigger

30. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 30 EVG Hardware Event Generator VME-64x Module: Sits in Master Timing Crate with VME PNET receiver and Master Timing CPU.  Receives 119MHz reference and 360Hz master timing fiducial from SLC timing system.  Receives PNET pattern from SLC system.  Broadcasts timing system data in the form of a high-speed 25Gb/s serial data stream to the EVRs over an optical fiber.

31. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 31 Event Receiver Comes in two flavors: VME and PMC. Receives 2.5Gb/s serial datastream from EVR and generates triggers based on values of the event codes. Also receives and stores PNET timing pattern, EPICS timestamp and other data and stores them in an internal data buffer. Can output 14 total pulsed-output triggers and several more level-type Triggers are output via a rear transition module (not shown) Trigger delay, width, duration and polarity are fully programmable Trigger signal level format is TTL VME Version has 10ps jitter PMC Version has 25ps jitter EVR Hardware

32. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 32 System SW

33. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 33 What Happens during one 2.8mS machine interval: F3 0 Fiducial B0 Acq Trigger 1023 Beam Kly Standby Record processing (event, interrupt) Fiducial Event Received Event Timestamp, pattern records, and BSA ready Receive pattern for 3 pulses ahead Hardware Triggers 500 0.3 18 100 Triggering Event Codes Start Kly Accel

34. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 34 Performance 9 ps rms jitter EVR jitter w.r.t. fiducial  The Event System Trigger Jitter was measured using an Agilent Infinium 54845A Digital Oscilloscope in Jitter Histogram Mode / data was collected for 30 minutes  The EVR output (shown) was measured against the system input trigger (360Hz fiducial)  The Actual jitter performance is much better after subtracting off the intrinsic jitter of the scope: Actual Jitter: JITTER system = [ (JITTER sys_meas ) 2 – (JITTER scope ) 2 ] 1/2 Event System Jitter: JITTER EVR =[ (9.7472ps) 2 – (6.3717ps) 2 ] 1/2 = 7.3763 ps

35. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 35 Issues Current LCLS Timing System has growing pains (integration w/old system, SW, HW) Use of commercial HW doesn’t quite fit our needs / having to modify EVG & EVR System scalability / with an eye towards eventually replacing the old SLAC timing system with this one (we think it will work..) Phase Drift on Fiber (where Russell’s Laser System will help us)

36. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 36 Issues – Fiber Phase Drift Temperature-Dependent Phase Delay  The Fiber-Optic cable will experience temperature- induced optical transport and length variations which will affect the timing. What should we expect? Calculation ( warning hand-waving guesstimate! ): Delay Coefficient: 30ps/C/Km (source J. Frisch ILC timing notes) Worst Case (max length / largest temp variation): Length: Sector 20  FEH (~2.2Km) Temperature Variation: 32F – 100F (0C – 40C) (typ winter/summer)

37. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 37 Temperature-Dependent Phase Delay (cont.)  How will this affect the timing system?  Badly, but its not a 1:1 correlation between the trigger timing and the fiber phase delay  Why? Because the fiber contains a 20-bit serial digital data stream (w/embedded clock) that gets recovered and decoded at the receiver. So its not straightforward to understand the effect  Here’s a guess:  20-bit datastream @ 2.38Gb/s  1 bit cell = 420ps  With = 2204ps  5 to 6 bit slots get shifted. This will affect when the Event Code gets de-serialized and loaded in the memory to produce a trigger  The trigger could jump around by 8.4ns (119Mhz clk cycle) increments > This is one of the LCLS applications that needs the Stabilized Laser Distribution System

38. John Dusatko USPAS Fundamentals of Timing & Synchronization jedu@slac.stanford.edu January 25, 2008 / Santa Rosa, CA 38 End of Talk