KEKB Accelerator Control System KEKB Controls Group by T. Katoh

Control System l Control Computer System l Timing Control System l Beam Gate Control System l Personnel Safety Control System l Communication System l Control Room and Operators Consoles

Contents l Construction History of KEKB Accelerator Control System l KEKB Control System Design l System Hardware Configuration l System Software Configuration l Central Control Room

Construction History Started System Design for KEKB Controls Systems Evaluation Committee Recommendation to adopt EPICS First Version of Computer System Specifications Started to Dismantle TRISTAN Main-Ring Final Version of Specifications Opened the Bids abco1 and 15 IOCs were Installed Hardware Specifications were Fixed BT Lines were Commissioned The Rest of IOCs were Installed

Construction History(contd.) Started Software Installation with Link- Persons and Engineers from the Company Commissioning of KEKB Accelerators Commissioning with the BELLE Detector Started Designing PF-AR Control System Installation of abco2 and 10 IOCs for PF-AR Started to Dismantle PF-AR Devices Started to Install or Re-install Equipment Commissioning of PF-AR

KEKB Control System Design l System Requirements l Constraints l Basic Concepts l System Architecture

System Requirements l Sources –Equipment Groups Magnet and Power Supply RF Beam Monitor Vacuum Beam Transport Feedback, Facilities, Physics, etc. –Operations Group –Accelerator Physicists or Commissioning Group –Controls Group

System Requirements(contd.) l Requirements –All the data that are possible to take should be taken. –All the data that are taken should be saved for later analyses. –All the operation should be recorded for later inspection. –All the machine parameters and information about the machine components should be stored in the database.

System Requirements(contd.) l Requirements(contd.) –The Man-Machine Interface should be Operator- Friendly. –The Programming Environment should be Programmer-Friendly. –The Overall Response Time to an Operator’s Request should be less than a Second unless the Progress of the Process is Indicated.

Constraints l Use CAMAC as an Equipment Interface l Schedule –Control Systems Final Design January, 1995 –Removal of MR equipment January, 1996 –Control Systems Installation September, 1996 –Completed Hardware Installation March, 1997 –KEKB CommissioningDecember, 1998

Constraints(contd.) l Limited Man-Power –9 KEKB Controls Group Members 3 Physicists and 6 Engineers –12 Link-Persons from Other Groups 2 from Magnet and Power Supplies 2 from RF 2 from Vacuum 2 from Beam Transport 2 from Beam Monitor and Feedback 1 from Linac, 1 from Physics

Basic Concepts l Standard Model Architecture l International Standards –CAMAC, VME, VXI, GPIB, FDDI, etc. l Existing Software Environment : EPICS l Separate Computer-bus and Field-buses l “Link-person” System : Applications l “Out-Sourcing” : 5 People from Companies

Standard Model Architecture l Presentation Layer (Server) –Man-Machine Interface(OPI) –Logging, Analyses, Alarm Displays, Database, Global Feedbacks l Equipment Control Layer (IOCs) –Data Acquisition, Sequence Control, Local Feedbacks l Device Interface Layer (Field-buses) –Hardware Interfaces

The most Important Points l Presentation Layer –Software Development Tools User/Programmer Friendly Tools –Ergonomic Consoles LCDs, Macintosh’s, PCs, PDPs, Flat Desks l Equipment Control Layer –VMEbus : High Reliability, Flexibility l Device Interface Layer –CAMAC : High Reliability, Well-known

Presentation Layer l Operator’s Consoles : X-Terminals l Database Management : ORACLE 7 l Alarm Generation/Recording l Data Logging l Data Display : CATV Network l Simulation : SAD Program l High-Speed Network –FDDI, Distributed Shared-Memory Network, –100 Base/TX and 10 Base/T l Gateway to KEK Laboratory Network : acsad

Equipment Control Layer l Provides Standard Interfaces –CAMAC Serial Highway Drivers(Hytec) –MXI-bus Drivers for VXI Main Frames(HP) –ARCNET Drivers, GPIB Drivers(NI), etc. l Computers : VMEbus based IOCs –FORCE CPU-40(MC68040) –FORCE CPU-60(MC68060) –FORCE PowerCore6603(PPC 603e) –FORCE PowerCore6750(PPC 750) l Operating System : VxWorks

Device Interface Layer l Standard Interfaces –CAMAC : RF and Vacuum –VXI : Beam Position Monitors –GPIB : RF, Vacuum, Magnet Readouts, etc. –ARCNET : Magnet Power Supplies TV Signal Switches –MODBUS+ : PLCs –RS232: Vacuum Measuring Instruments

System Architecture(1) l Functional Configuration Database Console Simulation Logging Alarm Gateway Display Laboratory Network Console CAMAC VME CAMAC VME CAMAC Presentatikon Layer Equipment Control Layer Device Interface Layer High-Speed Network CATV CAMAC

System Architecture(2) l Actual Configuration Gateway Display Laboratory Network CAMAC VME CAMAC VME CAMAC Presentation Layer Equipment Control Layer Device Interface Layer CATV CAMAC FDDI Network Switch X-Term Alarm, Logging, Database, Console, Simulation Consoles F/E

System Configuration Power Supply Unit ARCNET Driver Module GPIB Controller Module CAMAC Serial Highway Driver Module CPU Module Various Equipment Magnet Power Supplies FDDI Switch Server Workstation Consoles & Peripherals Measuring Instruments VXI Main Frame RS232 Module Central Control Building Local Control Building VME-MXIbus Driver Module CAMAC Crates

Central Control Room Cisco Catalyst 1200 Bridge Phaser 550JX2 QMS2425 Server abco1 ME/RK 460 Cisco C1400 Concentrator FDDI GIGA Switch 26 Local Control Rooms Lattis System 810M X Terminals for Consoles VME IOC Timing, etc.

abco1 Server Workstation l PA-RISC 7200 Architecture –120MHz CPU Clock –4 CPUs –2GB of Main Memory –4GB Hard Disk Drives –FDDI Interface –20GB RAID Disk

acsad Server Workstations l Compaq Alpha Server –4 Alpha CPUs of 440MHz –6 Alpha CPUs of 330MHz –1 GB Memory –50 GB RAID –FDDI Network Interface –True 64 Unix Operating System

Former KEKB Control System abco1 acsad Laboratory Network KEKB Control Network 94 IOCs PCs KEKB Accelerator Operation EPICS Software Development Relational Database KEKB Accelerator Operation Beam Optics Simulation Operators’ Consoles Real-time Control

KEKB/PF-AR Control System abco2 acsad Laboratory Network KEKB Control Network 104 IOCs PCs KEKB & PF-AR Accelerator Operation EPICS Software Development Relational Database KEKB Accelerator Operation Beam Optics Simulation Operators’ Consoles Real-time Control abco1

abco2 Server Workstation l PA-RISC 8500 Architecture –440MHz CPU Clock –2 CPUs –1GB of Main Memory –36GB Hard Disk Drives –FDDI Interface –140GB RAID Disk

IOC Configuration Power Supply Module ARCNET Driver GPIB Controller （ NI GPIB 1014 ） CAMAC SD （ HYTEC VSD 2992 ） CPU Module （ PowerCore 6750 ） CAMAC Crates Measuring Instruments Magnet Power Supplies FDDI ｰ Ethernet Bridge Cisco Catalyst 1200 Ethernet HUB Lattis 810M Measuring Instruments VXI Main Frame RS-232C Module MXIbus Interface Module Terminal Server Cisco 2509/2511 System Monitor Module X Terminals RS-232C Ethernet From CCR

VME IOC System l Subrack –Schroff 16 Slots Subrack l Power Supply Module –259 Watts, 5V 35A, 12V 5A, -12V 2A –14 HP Wide, Schroff MPS l System Monitor Module –Mitsubishi DRSJ-01 Remote System Reset DC Power Lines Monitor

VME Modules Used l CPUs –FORCE CPU-40MC MHz –FORCE CPU-60MC MHz –FORCE PowerCore 6603e PPC 603e –FORCE PowerCore 6750 PPC MHz –FORCE PowerCore 6750 PPC MHz l CAMAC Serial Highway Driver –HYTEC VSD 2992

VME Modules Used (contd.) l GPIB Controller –National Instruments GPIB 1014 l ARCNET Driver –Advanet ARCNET-4 l VME-MXI Driver –Hewlett Packard VME-MXI –National Instruments VME-MXI II l MODBUS+ Interface Controller

Number of IOCs and VME Modules MGRFMOVABTMVENTIPHLITotal IOC CAMAC GPIB RS-232C ARCNET MXI Total

Equipment Connected l 2,517 Magnet Power Supplies –176 ARCNET Segments l 800 Beam Position Monitors

EPICS Records on IOCs l 242,597 EPICS Records on 94 IOCs l 25,147 EPICS Records on IOCMGD06 l 2,788.5 EPICS Records in Average l 48,149 kB Max. Allocated Memory l 1,324 kB Min. Allocated Memory l 12,142 kB Ave. Allocated Memory

Hardware Summary l Use Standard Buses –Reliability: VME, Compact PCI, CAMAC Not ISA, EISA, PCI, Proprietary Buses –Separate CPU Bus from Field Buses Isolation, Analog Signal Handling l Choose Components Carefully –Power Supplies –Capacitors, Fans, Connectors l Use Ergonomic Equipment

Control Consoles

TRISTAN Consoles

KEKB and PF-AR Consoles

KEKB Consoles l Voice Generator using Power Macintosh l Multiple Screen Display for Consoles –DELL Optiplex NX1 –Intel Pentium II 400MHz, 1GHz –Colorgraphic Evolution 4 –Macintosh with Multiple Video Cards l X-terminal using IBM Network Station –Power PC 406(133MHz) –48MB Memory –Disk-less / Fan-less System

Control Consoles l 18” TFT LCDs(1280x1024 pixels) l 16” TFT LCDs(1280x1024 pixels) l 14” TFT LCDs(1024x800 pixels) l Book-type Personal Computers –DELL Optiplex NX1 and IBM Network Station –Multi-Screen Display Controllers –Wireless Keyboard/Mouse l Power Macintosh with 2/4 Video Controllers l Six 40” Plasma Displays as TV Monitors l Easy to change Configuration

Software

l Two Language Architecture –TRISTAN Experiences : NODAL and PCL –Interpretive Language for Applications SAD and python Short Turn-around Time Safe Debuging Environment Not Dedicated Application Software Programmer –Compiler Language for Low-Level Software C or C++ for Low-Level Software Quick Response Time Runs Fast Realizes Everything You Want

Software (contd.) l Relational Database Software –From the Designing Stage –Put All the Accelerator Information Cable Connection Lists Equipment Parameters –Physical Parameters : Sizes, Weight, … –Calibration Constants, Fitting Curve Parameters, … Histories –Installation Dates, Repair Histories, Costs, …. Property Numbers and more … –Generates EPICS Database Automatically

Software (contd.) l EPICS Software Toolkit –medm Generated from Relational Database –Ah : Alarm –Ar : Archiving –SNL : State Notation Language

EPICS Software Used l medm l SAD l python

medm Applications

python Applications

SAD Applications

EPICS Databse Files

Linac Portable CA Server Linac Console Linac Main Controls Linac Sub-Controls Linac Equipment KEKB Equipment KEKB IOCs KEKB CA Clients Linac CA Server

Linac Portable Channel Access

Application Software in Use l SAD141 l medm 74 l python 42 l Miscellaneous 6 l Total263

EPICS Records on IOCs l 242,597 EPICS Records on 94 IOCs l 25,147 EPICS Records on IOCMGD06 l 2,788.5 EPICS Records in Average l 48,149 kB Max. Allocated Memory l 1,324 kB Min. Allocated Memory l 12,142 kB Ave. Allocated Memory

Software Summary l EPICS R3.13 Toolkit on HP-UX 10.2 l medm for Man-Machine Interface l X-Window Server Software on the Terminals l python for Application Programming l SAD for Accelerator Simulation / Operation l VxWorks on IOCs l Tornado Cross Software Development Sys. l CAPFAST for EPICS Runtime Database Dev. l ORACLE 7 for Relational Database l Portable Channel Access Server for Linac Controls

VME CPU Board Benchmarks

CPU Board Benchmark Test FORCE Pcore 6604 FORCE CPU-40 CPU Power PC 604e MC Clock 200 MHz 25 MHz Memory 16 MB16 MB L2 Cache 512 kB --

CPU Load Arising from Scanning Database PCore 6604CPU sec0.60 % 5.8 % 0.5 sec1.2 % 12 % 0.2 sec3.0 % 27 % 0.1 sec6.1 % 56 %

Transactions Time Required for Channel Access PCore 6604CPU-40 ca_search 953 us 1,189 us ca_put 23 us 109 us ca_get 58 us 118 us ca_put&ca_get 76 us 246 us

VME-MXI Problems

VME-MXI Problem l HP VME-MXI Driver Module : National Instruments Products l FORCE PowerCore 6603 or 6750 CPU Modules : Power PC l Tandra Universe or Universe II Chips : PCI-VME Bus Bridge Chips l CPU Module Halts and Never Restarts Responding to the Reset Signal without Power is switched off

Improvements in ARCNET l A HUB Box contains 3 ARCNET HUBs l 33 HUB Boxes were Installed for 97 ARCNET Lines l Decreases 10 2 /Day/Line ARCNET “Reconfiguration” Errors down to Zero

Configuration of the Test Bench CPU module VME-MXI module System Monitor module VXI-MXI Controller MPX module RF Voltmeter CPU Modules tested FORCE CPU40: MC68040, 33 MHz FORCE CPU64: MC68060, 66 MHz FORCE PowerCore 6603: PPC 603, Universe Chip FORCE PowerCore 6750: PPC 750, 266 MHz, Univ. II FORCE PowerCore 6750: PPC 750, 400 MHz, Univ. IIB

Normal and Abnormal Bus Cycles Normal Bus Cycle Abnormal Bus Cycle

Abnormal Bus Cycle

Normal and Abnormal DTACK* Signals PowerCore 6603, with Universe Chip Normal Abnormal

Abnormal DTACK* Signal PowerCore 6750, 266 MHz, with Universe II Normal Abnormal

Realtime Linux

Linux l Disk I/O causes ms of Latency, possibly Latency can be around 100 ms or more. l Cause: –Non-preemptive Kernel –Scheduling Algorithm –Interrupt Disabling

Non-Preemptive Kernel Interrupt Low Priority High Priority Kernel Latency

Preemptive Kernel Low Priority High Priority Interrupt Kernel

EPICS iocCore Network Channel Access Run-time Database Device Access Layer I/O Buses (VME/PCI/ISA)

EPICS under L4-Linux L4 real-time micro-kernel X Linux Server MEDMiocCore

Linux System Calls Linux Server Process Interrupt L4 Kernel ( IPC ) Linux Kernel Process Interrupt Standard LinuxL4-Linux

RT-thread Preempts Linux Linux Server RT- thread Interrupt L4 Kernel Process

RT-thread Calls Linux Linux Server RT- thread Interrupt L4 Kernel Process

What Happens if RT-thread Calls Preempted Linux? Linux Server RT- thread Interrupt L4 Kernel Process Latency