1. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 1 N. Kornweibel & P. Baksai Operations at Paranal

2. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 2 Introduction Instrument Control LAN Console Layout Data Flow System Autrep Logging

3. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 3 Instrument Control LAN

4. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 4 Control LAN Data 1.Control Information: involves all LANs. Maximum 200 ms delay for a reply on a command is met “typically” by keeping the network load low (< 30%). 2.Data from technical CCDs (e.g. slit viewing, guiding). Per camera up to ~5 Mbit/s for short periods (i.e. data is not sent continuously to WS). 3.Astronomical data from CCDs or IR detectors. Instrument LAN, and via the Backbone LAN also to the User Station and General Services LANs (for Archiving). Support sustained data rates of 1000 mbps. 4.Other data: Field stabilization, Rapid Guiding and X.

5. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 5 X Terminals CDE window manager Console Layout

6. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 6 DHS Data Flow System P2PP (visitor) OT (service) Instrument WS BOB vcsolac volac FITS DCSICSTCS OS Logs vcsolac DHS WS (archive) OB OHS WS (archive)

7. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 7 Autrep System Extracts operations logs from log files Archives operations logs in SYBASE Generates automatic daily reports and plots Web interface for long term trending

8. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 8 Autrep Automatic Plot

9. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 9 Autrep Architecture IWS Ops logs Web Report Interface

10. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 10 Instrument Software Life and Adventures Life Cycle Daily Operations User Feedback

11. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 11 Life Cycle of an Instrument at Paranal After PAE…shipping… AIV Commissioning PAC Paranalization SciOps Maintenance Refurbishment Decommissioning Recycling

12. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 12 Life Cycle of its Software Software activity mainly during: Commissioning Paranalization Maintenance

13. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 13 Commissioning activities Mainly support, we do not touch the software Maintain the SW Punch List in Remedy Standby for start of sky observations to tackle problems interacting with other systems (TCS, OHS, ASM, DFS, etc.) Contribute to the analysis of problems and propose solutions based on operations experience

14. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 14 Paranalization activities Update PECS for new targets (operations, development) Update integration module xxins for operations target, to achieve smooth rebuild sequence Create integration module for development target Create xxmtoo module Create xxocal module Update template signature files and create Instrument Package for the first period of operation Update Technical Instrument Package Register instrument WBS in Remedy PPRS Register keywords for AUTREP, create Autrep plots

15. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 15 Software Maintenance Instrument is assigned a 1 st and 2 nd software responsible, but software could be modified and installed by any software engineer Instrument must be able to be rebuilt from “scratch” using pkginBuild access to Garching cmm archive. Integration module is central configuration control for instrument. (e.g. xxins)

16. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 16 Software Maintenance Remedy (PPRS) Central Database for Incidents, Problems, Maintenance Change requests.

17. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 17 Software Maintenance regular activities Configuration update and backup Change request commissioning with Instrument Scientist Daily startup Weekly reboot Rebuild twice per semester Period changeover VLTSW Release upgrade and Patch installation

18. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 18 Daily Operations Shutdown and Startup Modes: Night, Day, Calibration CalobBuild

19. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 19 Daily Operations Cycle Calibrating Night Operations (Observing or ready) Day Operation (Daily checks Maintenance) 1. Shutdown 2. Startup 3. Night Mode 1. Calib Mode 2. Execute Daily Calibrations 1. Check Calibrations 2. Day mode

20. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 20 Operations Menu

21. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 21 Shutdown and Startup xxinsStop and xxinsStart shutdown and startup INS WS software according to the configuration in xxmcfgSTART.cfg determines panel displays and system access. xxinsStartup GUI is used to edit the config.

22. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 22 Shutdown and Startup

23. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 23 Instrument Modes NIGHT: Online, TCS enabled. CALIB: Online as required for calibrations. DAY: Standby, dark state, TCS disabled. Executed with script xxinsMode with possible SETUP from reference files.

24. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 24 Build calibration OB from observation data and standard calibration set Executed daily at end of night operations Should complete in under 2 hours in CALIB mode and no TCS. Configured in xxocalCALOB.cfg CalobBuild

25. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 25 CalobBuild # Initial OBDs to be included CALIB.TPL.OBD1 "HAWKI_calob_begin.obd"; CALIB.TPL.OBD2 "HAWKI_calob_bias.obd"; # Final OBD to be included CALIB.TPL.ENDOBD "HAWKI_calob_end.obd" # FITS keywords used to select the calibrations CALIB.TPL.FITS "INS.MODE"; # Calibrations for Mode=IMG; exclude TECHNICAL CALIB.TPL1.INDEX "IMG"; CALIB.TPL1.VALUE1 "HAWKI_img_cal_Darks.obdx"; CALIB.TPL1.COND1 "(({$IMAGE:DPR.CATG} == {SCIENCE} || {$IMAGE:DPR.CATG} == {CALIB} || {$IMAGE:DPR.CATG} == {ACQUISITION}) && {$IMAGE:DPR.TYPE} != {DARK})"; CALIB.TPL1.DAYS "1";

26. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 26 User Feedback Principal users of Instrument Software are: Night time — Staff Astronomers / Instrument Scientist — TIOs Day time — Instrument Scientist — Instrumentation Engineers

27. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 27 Importance of User Feedback INS SW is in use almost 24h per day Time of use in operation adds up to more than was invested in the development and commissioning phases INS SW authors are usually not available for troubleshooting Ask any user about SW features: they know exactly What is slow The boring and bothersome tasks The useful and fast commands How things could be improved How some problems can be reproduced

28. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 28 Our Common Goal at Paranal Minimize downtime, specially thecnical, and SW above all

29. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 29 Three things that play against our goal Slow procedures in instrument function drivers Difficulties for the operator/engineers to identify and cure the problem cause (Root Cause Analysis) Administrative obstacles for safe and efficient troubleshooting

30. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 30 Considerations that will help us During design, consider the impact of: Parting from standards (procedures, devices, models) Poor error handling and unclear diagnostics Tollerance to failure Speed of recovery (recover or switch focus?)

31. ESO-PSW – N. Kornweibel & P. Baksai ESO Instrumentation Software Workshop 2008 - 31 Conclusion Operations at Paranal are oriented towards minimum downtime and maximum data quality A positive impact is quickly noticed after close interaction between Users and Engineers A better knowledge of systems integration and INS SW cycles at Paranal might empower developer’s contribution to smoother and shorter Commissioning, Paranalization times (INS offered sooner to community)