Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LDAQ – the New Lujan Center Data Acquisition Application Frans Trouw, Gary Cooper and Paul Lewis Lujan Center, Los Alamos National Laboratory

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Outline Brief Overview of Lujan Center Description of hardware Existing data acquisition application (LOS) Issues with LOS (reminder, see Paul’s talk) Guiding principles for new application (LDAQ) Overview of existing new LDAQ application Experiences with LDAQ (good!) Summary

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Data Acquisition at the Lujan Center - hardware Detectors are all 3 He gas tubes or area detectors Pre-amps at detectors generate a differential signal Custom “Time-of-Flight” boards take differential input and convert into events in a buffer VME (VXI) crate with single board computer collects events from TOF units

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Overview of Hardware Configuration Detector & Preamp CPU ROC TOF CPU ROC TOF CPU ROC TOF T0 from accelerator Private DAQ Network 100Mbit/s DAQ Host PC Windows Chopper EPICS IOC VXI Crates VME Crate Processors VxWorks "Public" LANL Network 100Mbit/s LANSCE Facility EPICS Gateway Data Archive PC Panels CPU ROC TOF CPU ROC TOF CPU ROC TOF CPU ROC TOF CPU ROC TOF CPU ROC TOF CPU ROC TOF T0 from accelerator Private DAQ Network 100Mbit/s DAQ Host PC Windows XP Remote access Chopper EPICS IOC VXI Crates VME Crate Processors VxWorks "Public" LANL Network 100Mbit/s LANSCE Facility EPICS Gateway Data Archive PC Controls PC Sample Environments

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Crate contains SBC, ROC module, and 1-11 TOF units TOF unit ROC module SBC To Detectors

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LOS Development Methodology Domain: class diagram, class diagram, state charts, state charts, action language action language Domain: class diagram, class diagram, state charts, state charts, action language action language Domain X: class diagram, class diagram, state charts, state charts, action language action language “Model compiler”: templates templates generation scripts (rules) generation scripts (rules) Source code generator Build “Model compiler”: run-time library run-time library Imported software Source file Build script Executable Executable Executable

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D xtUML is in principle attractive High level description generates consistent code and less errors Extending or changing model generates new code that takes into account interdependencies Less hand-coding => less manpower Quality is consistent and assured

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D xtUML has Significant Drawbacks Requires the desire to think at a very high level of abstraction – requires expert Not all of the code can be generated Highly dependent on message passing The community using xtUML is very small (< 100 at SMUG) - “early adopters” Only available for Windows Debugging next to impossible!

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D xtUML – communication risky? Message passing (MP) is critical to xtUML MP is done using Windows message queues or sockets A single communication failure kills the whole system VxWorks 5 network stack obsolete Windows network stack documentation limited (being generous here)

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LOS move to Linux – MP problems! As Windows is so opaque and a single user environment, xtUML system libraries were ported to Linux Communication was implemented using a typical client server model (TCP/IP sockets) Linux and VxWorks 5 network stacks are incompatible

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LDAQ “design principles” VXI crate LDAQ processes should be independent entities – fault tolerant No “host” computer controlling task Minimal code base Homogenous and state of the art operating system environment Open source preferred – transparency, quality & cost

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D What is LDAQ? Threw out all of LOS except hand coded pieces that do VME communication Moved VXI CPU’s from VxWorks to Linux “Host” computer is Linux Wrote a minimal application that integrates the VME calls – about 1 month of effort LDAQ runs independently as a user application on each VXI crate

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LDAQ visual description LDAQ MySQL (on host) ROC module TOF units 20 Hz Local Network Computer LANL Trigger Start Stop Ready Data

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LDAQ benefits versus LOS Code can be changed and recompiled in seconds using Eclipse (> 1/2h for LOS) LDAQ failure on any VXI crate does not affect others – can be restarted Remote cross-debugging using gdb successful Linux is a true multi-user system (security) Open source – no licensing costs

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Single Event and Histogram LDAQ does single event “out of the box” – temporary NFS limitations Single event -> large amount of data, use only when useful LOS histograms implemented in LDAQ “Live data” transfer call implemented Defined “time-focusing” histogram – e.g. live GSAS coming very soon!

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Experience with LDAQ Pharos has run LDAQ for most of this year – only failures were power cuts! HIPD and more recently HIPPO are successfully using LDAQ with legacy scripting LDAQ is a user application; changing SBC’s is straightforward (e.g. PPC -> Intel) – cost, effort Restart takes seconds Infrequent/development VME bus “hangs” now limiting reliability

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D Why is LDAQ successful? Each VXI crate is an independent agent – does not know about others No controlling task Minimal features, implement bells and whistles externally (e.g. run timing) Linux is robust and transparent Keep it simple!

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LDAQ development - observations Exclusively using Linux - economical (up front and manpower) – cultural challenges Does require significant Linux expertise MySQL straightforward, robust & free Agile development, develop features as needed – requires IS collaboration Close collaboration between physical and computer scientists

Operated by Los Alamos National Security, LLC for NNSA U N C L A S S I F I E D LDAQ - future LDAQ development is essentially done! Documentation Needs external setup & control scripts Scripting – leave to IS, provide, or port existing control scripts? Integration with slow controls – EPICS Visualization toolkit Real time analysis