David Abbott CODA3 - DAQ and Electronics Development for the 12 GeV Upgrade

Outline  What is CODA  DAQ Issues / Aging Technologies  Requirements for 12 GeV  CODA3 Developments  Software  Hardware  Summary

The CODA mission  CODA is a software toolkit with specialized hardware support  Modular software components use the network for inter-process communication and event transport.  Use open standards and minimize the use of commercial software while maximizing use of commercial hardware.  DAQ systems for each experimental Hall can be “built-up” from common components to fit their needs.

What is CODA

Data Acquisition Status  The Data Acquisition Group is charged with the task of providing a functional, unified acquisition system appropriate for ALL experimental programs at JLAB.  Recent experiments have begun to test the limits of the current CODA system.  Development in both hardware and software technologies will be necessary to provide continued support of the future 6GeV program.  Current DAQ projects reflect the philosophy that we can progress to support the physics of the 12 GeV program through an “upgrade” of the existing proven system.

Front-End Issues…  The CODA ROC is a customizable software object that provides a standard for the User to access front-end hardware - whatever that may be. It must evolve.  Obsolete FASTBUS digitizing electronics needs a replacement alternative (in VME). There are far fewer appropriate commercial solutions today.  Real-Time readout on a per event basis limits the maximum accepted L1 trigger rate (~10 KHz).  32 crate limit on the trigger distribution system is nearly reached in Hall B.  DAQ - Trigger - Electronics - All must be designed to work together, efficiently, “customized” !!

Other DAQ Limitations  Event transport limitations in the current CODA architecture are being seen for moderately complex systems.  Aging software technologies and reliance on third party packages are making code portability and upkeep difficult.  Monitoring and control of large numbers of distributed objects are not handled in a consistent way (too many protocols).  Slow controls only minimally supported ~30 ---> 1 EB ROC MB/sec KHz Trigger Hall B “bottlenecks”

A General Plan  Replace aging technologies  Run Control  Tcl/Tk based DAQ Components  mSQL  FASTBUS/CAMAC  Hall D (GlueX) requirements drive development direction  Maintain Cross platform support  Linux - Solaris - vxWorks - OS X - 64 bit…  Support and use new commercial advances where possible.  More custom hardware design will be required and an electronics support group to maintain.

CODA3 - Requirements/Goals  Pipelined Electronics (FADC, TDC)  Dead-timeless system  Replacement for obsolete electronics  Eliminate large numbers of delay cables  Integrated L1/L2 Trigger and Trigger Distribution System  Support up to 200 KHz L1 Trigger  Use FADC for L1 trigger input  Support 100+ crates  Parallel/Staged Event Building  Handle ~100 of input data streams  Scalable (>1 GByte/s) aggregate data throughput  L3 Online Farm  Online (up to x10) reduction in data to permanent storage  Integrated Experiment Control  DAQ RunControl + “Slow” control/monitoring  Distributed, scalable, and “intelligent”

Anatomy of a CODA3 DAQ System

Example - Hall D DAQ Existing Halls

Front-End Goals  L1 Trigger rate KHz  Block up Events (200 event block -> 2kHz)  Move some ROL runtime code to modules (FPGAs)  ADCs provide L1 trigger data ( hence we need a distributed high speed clock MHz  New Trigger Supervisor  Perhaps 100+ crates  Support pipeline, event blocking  Manage flow control into DAQ system backend

Front-End Systems VME CPU VME CPU -(MV6100) PowerPC, GigE vxWorks or Linux CODA ROC Readout ~160+ MB/s Trigger Interface Trigger Interface - (V3) Pipeline Trigger Event Blocking Clock distribution Event ID Bank Info F1 TDC Flash ADC R&D to support fully pipelined crates capable of 200 KHz trigger rates

Level 1 Trigger Distributed high speed clock  Distributed high speed clock derived from Accel clock 499 MHz derived from Accel clock 499 MHz stepped down at crate - inter crate jitter <50ps stepped down at crate - inter crate jitter <50ps  Subset of ROCs collect summed ADC data and send it to L1 Trigger - in sync and send it to L1 Trigger - in sync  12 bit sums/crate x 250MHz --> 3 Gbit/s links  Trigger decision goes to Trigger Supervisor for pipeline distribution to all crates for pipeline distribution to all crates

Staged/Parallel Event Building Divide total throughput into N streams (1GB/sec -> N*xMB/sec). Two stages - Data Concentration -> Event Building. Two stages - Data Concentration -> Event Building. Each EMU is a software component running on a separate host. Each EMU is a software component running on a separate host. 2N hosts are interconnected though one BIG switch

Event Management Unit EMU built around the ET system for customizable system for customizable processing/distribution of processing/distribution of event streams. event streams.Examples: Data Concentrator for ROCs Data Concentrator for ROCs Sub-Event builder Sub-Event builder Farm distribution point Farm distribution point Event Recorder Event Recorder User Processes can attach to User Processes can attach to any EMU in the system any EMU in the system

Level 3 Farm  Can be used for analysis or filtering  Support 100s of nodes  Nodes can come and go during event taking  Filtered events will be time ordered on the back-end. 1 GB/s 100 MB/s

Current DAQ Projects - Overview Components:  Run Control  Coda ROC  Coda EB/ER (EMU) Software Tools:  cMsg  ET  evio Hardware:  FADC/F1TDC  Trigger Interface (VME/PCI)  Trigger/Clock Distribution R&D:  Embedded Linux  Experiment Control  Staged/Parallel Event Building  VXS - High speed serial  200KHz Trigger/readout  Low jitter clock distribution

Some Current Projects  cMsg - homegrown publish-subscribe messaging which will become the basis for DAQ online communication  JAVA Agent-based Run Control  EMU - C/C++ based, modular, threaded  Event Blocking  Hardware - New TI to support existing Trigger Sup.  Software - new ROC, parallel readout lists, EVIO  Embedded Linux - establish a stable maintainable Linux distribution for SBCs with access to VME and PCI hardware.

Software Tools  cMsg - Homegrown publish-subscribe messaging - the basis for all DAQ online communication.  ET - Event Transport - High speed, shared memory based event distribution system. Also supports network transport.  EVIO - Library/Tools for reading, writing, building, analyzing and displaying events in the CODA framework.

cMsg - CODA Messaging System cMsg  Publish-subscribe messaging system  JAVA,C++,C implementation and APIs  Replaces older separate packages with one maintainable system

Experiment Control JVM Host 1 A ROC A HV RC  CODA Java-Based (v 1.5) “Intelligent” agents (AFECS)  JADE extensions provide a runtime “distributed” JVM.  Agents provide a customizable intelligence and communication with external processes. Host 1 Host 2 Host 3 AA ROCHV RC

CODA ROC CODA EMU EPICS IOC 1 EPICS CAG Trigger soft Trigger hard Online ANA A A A A A A A S S S S S NRNANC Physical Components Normative Agents Supervisor agent Grand supervisor Front-End ACC AFECS Platform IPC GUI/user IPC WEB IPC Hierarchy of Control IPC

Run Control GUI

Experiment Monitoring and Control

JLAB Pipeline TDC TDC ASIC (8 channels) Hits Trigger Time -->

JLAB Flash ADC FPGAs Hits Trigger Time --> 8µs

VME64X - VXS Interconnect J total pins 45 differential pairs 6 GHz Bandwidth 18 VME Payload Slots 2 Switching slots

Electronics Development

VXS - L1 Trigger VME CPU VME CPU -(V7865) Intel, GigE Linux CODA ROC VME Readout of Event Data Switch Sum and Trigger Distribution Modules (VXS) Collect Sums/Hits Pass Data to Master L1 Clock distribution Trigger Distribution Flash ADC Use VXS High speed serial backplane (P0) to collect Energy sum and hit data from FADCs Flash ADC P0

ATCA as a Standard Becoming a widely adopted Commercial standard 8U Blade Form-factor High-Power/High Availability Few applications in Physics yet and not particularly cheap Possible use for Master Trigger processor and Trigger Supervisor

Summary  CODA version 3 is now being molded - and can be adapted to future requirements!!  Our plan is to phase in new tools to provide a smooth transition from CODA2 --> CODA 3 (CODA 2.6 Release)  New custom electronics will be available for use soon.