Workshop in ALMA Logs Prepared by Juan Pablo Gil – Arturo Hoffstadt ADC Software Group ALMA Observatory March, 2015

ALMA Hardware Layout 66 Antenna ~45 device per antenna AOSTiming Photonic References 64 ant-correlator 16-antenna correlator

ALMA Software Layout ALMA Software is built upon ACS Different Subsystems CONTROL Antennas Time reference SCHEDULING DATACAPTURER BLCORR (64 ant) ACACORR (16 ant) TELCAL QUICKLOOK EXECUTIVE ARCHIVE

ACS Deployment dv01-abm Component Container Host Actual code that controls one physical device, or does one specific task Eg. IFproc, DTX, ARCHIVE_CORR_BULKRECEIVER_NT, CONTROL_MASTER Container Logic grouping of components. Has PATH/containerName Eg. CONTROL/DV01/cppContainer has IFProc, DTX Host Computer which runs containers. Eg. dv01-abm runs CONTROL/DV01/cppContainer xx-abm, gasXX, cob-cpn-xx, lo-lmc-x, etc dv01-abm CONTROL/DV01/cppContainer IFProc ADC Mount WCA Other containers…

ALMA Logging Human-readable description of one action at one specific time ~30 GB of every day log Container Logs: Coarse grained (Info) Local to machine Stored in plain text files XML Logs: Fine grained (Debug) System wide Stored in XML files Introducing high performance distributed logging service for ACS - Avarias et all, 2010

XML Log Example A typical XML log entry <Debug TimeStamp="2014-06-07T17:14:50.658" File="WVRImpl.cpp” Line="665” Routine="void WVRImpl::publishWVRData(const Control::WVRData#, ACS::Time)” Host="pm01-abm" Process="CONTROL/PM01/cppContainer" Thread="CONTROL/PM01/WVRMonitorThread" Context="” SourceObject="CONTROL/PM01/WVR" Audience="Developer” > <![CDATA[Sucessfully published the following WVR event. Antenna: PM01 Mid-point of data: 17:14:49.920 Temperatures: 284.069, 284.029, 284.105, 284.014 deg. K. Sampled every 1.152 seconds. Event is valid. Data monitored at: 17:14:50.496]]> </Debug> Log Levels: TRACE, DEBUG, INFO, NOTICE, WARNING, ERROR, EMERGENCY, … 2.5 millions of XML entries in 4 hours of observatory operation. Valuable information hidden in Debug and below. Main tool: jlog

Container Log Example Subset of XML log. Bird of Eye view of operation (By default, only stores up to INFO level) Generated at container level as text log line. 2015-03-19T11:52:07.546 [CONTROL/DV10/FrontEnd/ACD - void ACDImpl::setCalibrationDeviceBandInternal(CalibrationDeviceMod::CalibrationDevice, Receiver BandMod::ReceiverBand)] Setting Hot load in front of band 4 Vector Size: 1 Starttime: 1e4ce2e5e639700 Band: 3 Freq: 7.68268e+10 Caldevice 2 2015-03-19T11:52:07.631 [CONTROL/DV10/FrontEnd - void FrontEndImpl::lockFrontEndInternal(Control::FrontEnd::SubscanInformation, bool, bool)] Receiver band was already locked at 1.537186e+11 ACD Movement: Time: e602f8 To: 3 Set As: 2 ACDCSV,e602f8,3,2 2015-03-19T11:52:15.461 [CONTROL/DV10/cppContainer-GL - void PositionStreamConsumer::processData(Control::MountStatusData)] At 11:52:15.461 got the l ast needed data with a timestamp of 11:52:15.360. The delay is 0.101 seconds. 2015-03-19T11:52:16.253 [CONTROL/DV10/MountController - virtual void Control::MountControllerImpl::abortDataCollection()] Cleaning up past data colle ction that started at 11:51:54.000 and was due to stop at 11:52:15.360. Time now is 11:52:16.253

What is not an ALMA Log It is not observational data. It is behavior description. Linux messages: dmesg, /var/log/messages Mount circular buffers BL CORR internal coding Engineering scripts HVAC shutdowns Utility Modules

Where are the Logs? Container logs /alma/logs/{hostname} XML Logs /mnt/gas02/data1/AcsLogs-8.1/ Don´t be afraid of the name! log2015-03-20T19:42:07.557_2015-03-20T20:03:30.984.xml Other places to search Computing Logs: http://computing-logs.aiv.alma.cl Kibana (beta): http://hypatia.osf.alma.cl:5601

Useful locations at AOS CONTROL /alma/logs/gas01/CONTROL/ACC/javaContainer /alma/logs/gas01/CONTROL/ACC/pythonContainer Antennae /alma/logs/ANT-abm/CONTROL/ANT/cppContainer Photonic References /alma/logs/lo-lmc-1/CONTROL/LMC1/cppContainer /alma/logs/lo-lmc-2/CONTROL/LMC2/cppContainer BL Corr /alma/logs/cob-cdpm-b/CORR/CDPMaster/cppContainer /alma/logs/cob-cc/CORR/CCC/cppContainer ACA Corr /alma/logs/coj-cc-1/ACACORR/OBSERVATION_CONTROL/cppContainer /alma/logs/coj-cc-1/ACACORR/CMD_TRANSFER/cppContainer /alma/logs/coj-cpm-1/ACACORR/CDPM/cppContainer

Thank you! The Atacama Large Millimeter/submillimeter Array (ALMA), an international astronomy facility, is a partner- ship of Europe, North America and East Asia in cooperation with the Republic of Chile. ALMA is funded in Europe by the European Organization for Astronomical Research in the Southern Hemisphere (ESO), in North America by the U.S. National Science Foundation (NSF) in cooperation with the National Research Council of Canada (NRC) and the National Science Council of Taiwan (NSC) and in East Asia by the National Institutes of Natural Sciences (NINS) of Japan in cooperation with the Academia Sinica (AS) in Taiwan. ALMA construc- tion and operations are led on behalf of Europe by ESO, on behalf of North America by the National Radio Astronomy Observatory (NRAO), which is managed by Associated Universities, Inc. (AUI) and on behalf of East Asia by the National Astronomical Observatory of Japan (NAOJ). The Joint ALMA Observatory (JAO) provides the unified leadership and management of the construction, commissioning and operation of ALMA.