ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop N. Kornweibel & P. Baksai Operations at Paranal

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Introduction Instrument Control LAN Console Layout Data Flow System Autrep Logging

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Instrument Control LAN

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Control LAN Data 1.Control Information: involves all LANs. Maximum 200 ms delay for a reply on a command is met “typically” by keeping the network load low (< 30%). 2.Data from technical CCDs (e.g. slit viewing, guiding). Per camera up to ~5 Mbit/s for short periods (i.e. data is not sent continuously to WS). 3.Astronomical data from CCDs or IR detectors. Instrument LAN, and via the Backbone LAN also to the User Station and General Services LANs (for Archiving). Support sustained data rates of 1000 mbps. 4.Other data: Field stabilization, Rapid Guiding and X.

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop X Terminals CDE window manager Console Layout

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop DHS Data Flow System P2PP (visitor) OT (service) Instrument WS BOB vcsolac volac FITS DCSICSTCS OS Logs vcsolac DHS WS (archive) OB OHS WS (archive)

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Autrep System Extracts operations logs from log files Archives operations logs in SYBASE Generates automatic daily reports and plots Web interface for long term trending

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Autrep Automatic Plot

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Autrep Architecture IWS Ops logs Web Report Interface

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Instrument Software Life and Adventures Life Cycle Daily Operations User Feedback

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Life Cycle of an Instrument at Paranal After PAE…shipping… AIV Commissioning PAC Paranalization SciOps Maintenance Refurbishment Decommissioning Recycling

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Life Cycle of its Software Software activity mainly during: Commissioning Paranalization Maintenance

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Commissioning activities Mainly support, we do not touch the software Maintain the SW Punch List in Remedy Standby for start of sky observations to tackle problems interacting with other systems (TCS, OHS, ASM, DFS, etc.) Contribute to the analysis of problems and propose solutions based on operations experience

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Paranalization activities Update PECS for new targets (operations, development) Update integration module xxins for operations target, to achieve smooth rebuild sequence Create integration module for development target Create xxmtoo module Create xxocal module Update template signature files and create Instrument Package for the first period of operation Update Technical Instrument Package Register instrument WBS in Remedy PPRS Register keywords for AUTREP, create Autrep plots

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Software Maintenance Instrument is assigned a 1 st and 2 nd software responsible, but software could be modified and installed by any software engineer Instrument must be able to be rebuilt from “scratch” using pkginBuild access to Garching cmm archive. Integration module is central configuration control for instrument. (e.g. xxins)

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Software Maintenance Remedy (PPRS) Central Database for Incidents, Problems, Maintenance Change requests.

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Software Maintenance regular activities Configuration update and backup Change request commissioning with Instrument Scientist Daily startup Weekly reboot Rebuild twice per semester Period changeover VLTSW Release upgrade and Patch installation

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Daily Operations Shutdown and Startup Modes: Night, Day, Calibration CalobBuild

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Daily Operations Cycle Calibrating Night Operations (Observing or ready) Day Operation (Daily checks Maintenance) 1. Shutdown 2. Startup 3. Night Mode 1. Calib Mode 2. Execute Daily Calibrations 1. Check Calibrations 2. Day mode

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Operations Menu

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Shutdown and Startup xxinsStop and xxinsStart shutdown and startup INS WS software according to the configuration in xxmcfgSTART.cfg determines panel displays and system access. xxinsStartup GUI is used to edit the config.

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Shutdown and Startup

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Instrument Modes NIGHT: Online, TCS enabled. CALIB: Online as required for calibrations. DAY: Standby, dark state, TCS disabled. Executed with script xxinsMode with possible SETUP from reference files.

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Build calibration OB from observation data and standard calibration set Executed daily at end of night operations Should complete in under 2 hours in CALIB mode and no TCS. Configured in xxocalCALOB.cfg CalobBuild

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop CalobBuild # Initial OBDs to be included CALIB.TPL.OBD1 "HAWKI_calob_begin.obd"; CALIB.TPL.OBD2 "HAWKI_calob_bias.obd"; # Final OBD to be included CALIB.TPL.ENDOBD "HAWKI_calob_end.obd" # FITS keywords used to select the calibrations CALIB.TPL.FITS "INS.MODE"; # Calibrations for Mode=IMG; exclude TECHNICAL CALIB.TPL1.INDEX "IMG"; CALIB.TPL1.VALUE1 "HAWKI_img_cal_Darks.obdx"; CALIB.TPL1.COND1 "(({$IMAGE:DPR.CATG} == {SCIENCE} || {$IMAGE:DPR.CATG} == {CALIB} || {$IMAGE:DPR.CATG} == {ACQUISITION}) && {$IMAGE:DPR.TYPE} != {DARK})"; CALIB.TPL1.DAYS "1";

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop User Feedback Principal users of Instrument Software are: Night time — Staff Astronomers / Instrument Scientist — TIOs Day time — Instrument Scientist — Instrumentation Engineers

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Importance of User Feedback INS SW is in use almost 24h per day Time of use in operation adds up to more than was invested in the development and commissioning phases INS SW authors are usually not available for troubleshooting Ask any user about SW features: they know exactly What is slow The boring and bothersome tasks The useful and fast commands How things could be improved How some problems can be reproduced

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Our Common Goal at Paranal Minimize downtime, specially thecnical, and SW above all

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Three things that play against our goal Slow procedures in instrument function drivers Difficulties for the operator/engineers to identify and cure the problem cause (Root Cause Analysis) Administrative obstacles for safe and efficient troubleshooting

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Considerations that will help us During design, consider the impact of: Parting from standards (procedures, devices, models) Poor error handling and unclear diagnostics Tollerance to failure Speed of recovery (recover or switch focus?)

ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop Conclusion Operations at Paranal are oriented towards minimum downtime and maximum data quality A positive impact is quickly noticed after close interaction between Users and Engineers A better knowledge of systems integration and INS SW cycles at Paranal might empower developer’s contribution to smoother and shorter Commissioning, Paranalization times (INS offered sooner to community)