The Blanco telescope and its instruments: a status report Timothy M. C. Abbott , Alistair R. Walker, Sean D. Points, David J. James, Brooke Gregory, Roberto Tighe, Nicole David, Esteban Parkes, Rolando Cantarutti, Michael Warner, Omar Estay, Manuel Martinez, Marco Bonati, Edison Bustos, Andres Montane, Freddy Muñoz, Patricio Schurter Cerro Tololo Inter-American Observatory, Casilla 603, La Serena, Chile. Summary In recent years the V. M. Blanco 4-m telescope at Cerro Tololo Inter-American Observatory (CTIO) has been renovated for use as a platform for a completely new suite of instruments: DECam, a 520-megapixel optical imager, COSMOS, a multi-object optical imaging spectrograph, and ARCoIRIS, a near-infrared imaging spectrograph. This has had considerable impact, both internally to CTIO and for its wider community of observers. In this poster, we report on our experiences renovating the facility, ongoing improvements, lessons learned during the deployment of the new instruments, how we have adapted to them, including unexpected phenomena and subsequent responses. DECam 520 Mpixel CCD camera at prime focus[1]. Built as an NOAO facility-class instrument by the DES (Dark Energy Survey[2]) Collaboration. As of May 2016, more than 66 refereed papers have appeared that use DECam data[3]. DECam has been impressively reliable; a wavefront sensor (WFS) system and hexapod maintain focus and alignment automatically to deliver consistent image quality. Intuitive GUIs and scripted observations make the astronomer's life easy. On-sky time has totaled 7,400 hours and almost 400,000 images have been read out. The NOAO Science Archive (NSA) contains nearly a quarter million astronomical images from DECam. Approximately 1000 channels of telemetry facilitate support, diagnosis, and improvement.  A system of alarms, telephone calls, and action by the DECam system itself takes place in response to faults. The camera's fully-depleted CCDs have some oddities [4], and a tendency to fail under over-illumination.  2 of the 62 science CCDs are lost, one due to over-illumination, the second failed electrically. One science CCD has an amplifier with unstable gain, one WFS CCD has developed a very hot pixel, and two others have a signal chain fault compromising full well, but sill usable.  Automatic protection has been introduced to limit excess light risks. No changes in the positions of the optical elements. Changes to the dry air exhaust ports near the second element, will prevent the ingress of dust in high wind. The LN2 pump must be replaced after 8 months of use – much less than anticipated. R&D continues at Fermilab to build a pump with longer life.   Two flexible, vacuum-jacketed lines have been replaced after the originals developed thermal shorts. No issues with imager vacuum - the instrument has not been bought up to air since installation (2012).   The hexapod (from ADS, Italy) is very reliable, its only problems derived from a failed connector and recovery from complete power loss due to a fire alert in the building. Issues traceable to control software (SISPI) and computer hardware are very rare. The computers are rebooted on a monthly basis to avoid lengthy consistency checks (fsck). All 72 hard drives replaced on reports of poor reliability of the original models. Updating software is a vexing question, between "not changing what is working" and the need to avoid obsolescence. We have proceeded very cautiously so far. The community (NOAO) and DES (NCSA) data reduction pipelines have evolved together. DECam data is transmitted in real time to NCSA (DES) and/or to Tucson (all images), and raw data then appears in the NOAO Science Archive (NSA). Processed data appears in the NSA typically a few days after the raw data arrives. (See also paper by Diehl et al., this conference) COSMOS A new optical spectrograph at the Blanco f/8 Cassegrain focus covering 350nm – 1030nm. Based on OSMOS (Ohio State Multi-Object Spectrograph, [5], [6], [7]). a high-efficiency instrument for imaging, long-slit, and multi-object spectroscopy over a 100 square arcminutes. Contains two VPH grisms, “blue” one “red”, providing a resolution of ~2500 with 0.9” slit. “Blue” grism covers ~280nm at 0.7Å/pixel. “Red” grism covers ~400nm at 1Å/pixel. Available since May 2015. Commissioning 2014: existing calibration lamp units found to be inadequate for use with either COSMOS or ARCoIRIS. Designed and built a new calibration unit [8]. Commissioning spectra exhibited a feature corresponding to a direct image of the slit, derived from light passing between the grism and its cell; this was suppressed by installing baffles on the on the disperser wheel. Repeatability of slit positioning is better than 0.2 pixels, allowing streamlined object acquisition for spectroscopy, distinct advantage in the absence of a slit viewing camera. ARCoIRIS A new, facility-class near-infrared spectrograph, a near copy of the TripleSpec instruments at ARC 3.5m, Hale 5m and KECK 10-m [9]. A cross-dispersed, single-object longslit imaging spectrograph covering 0.81 < λ < 2.47 microns with resolution R~3500. Contains no moving parts, and has a rectangular entrance slit of dimensions 1.1 by 28 arseconds, and two 3.81-arcseconds offset square “outrigger” apertures, 1.15” on a side. The outriggers allow for better wavelength calibration by improved interpolation of the background spectrum. The collimated input beam illuminates a 110.5 line/mm reflection grating, blazed at 6.79 microns, with cross-dispersion provided by two ZnSe and one Infrasil prism. The stacked spectra, in six (6) spectral orders, are recorded on a science-grade HAWAII-II RG detector (0.417 arcsecs/pixel). A Leach III controller, with double correlated sampling mode, has read noise of ~14 e-/pixel, to about 4.5 e-/pixel for Fowler16 sampling. Acquisition and guiding is provided by a J-band slit-viewer camera, with an engineering-grade HAWAII-II RG detector (0.264 arcsecs/pixel and about four arcminutes square). ARCoIRIS was commissioned on the Blanco Telescope in 2015 and was first used for science observations in March of 2016. Telescope Control System The upgraded TCS is documented in [10]. The cRIO platform has been a total success. Maintainability, scalability, and accessibility have little or no impact to the normal operation of the telescope when intervention is needed.  Integration of new instruments, COSMOS and ARCoIRIS, has proved straightforward and f/8 guiding has been integrated via the distributed architecture policy adopted at the start. We have since added non-sidereal tracking by modifying the pointing kernel to include a polynomial-based trajectory generator.  Serendipitously, this also decreased steady state convergence time in the sidereal case.  Final integration with DECam is ongoing. Infrastructure Improvements A large (28m2, crane hook height 2.4m) cleanroom facility was designed at CTIO and built at a local workshop to provide clean conditions for assembly and maintenance of the detector dewars NEWFIRM (a wide field IR imager developed at KPNO[18]) and DECam. The conditions sought were to permit opening the dewars for direct access to filters, IR arrays, and CCD arrays with low particle counts and stringent static suppression. Total building glycol cooling capacity was upgraded from 45T to 80T.  The building and dome cooling circuits were rationalized to support the new instrumentation and upgraded to adequately cool the new instruments and their supporting hardware.  Two 10T air cooling units were installed in the dome and permit us to more nearly match the daytime temperatures to anticipated conditions at night.  Glycol temperature and flow, air temperature and oil pressure monitoring devices have been incorporated in the system and their output used to tune the system and adjust it for changing conditions. Active Optics Air Pressure Controller Blanco active optics incorporate 33 axial air bags. A new pressure controller has replaced the aging system. The new controller is form-fit and power compatible allowing a quick upgrade with no system downtime. It was designed with no calibration, or adjustment necessary, and higher offset and noise performance. The new pressure controller design is based on a pair of Clippart EV-PM-20-6025 Proportional Valves, replacing the original EP-310 On/Off valves, an Analog Devices SDX15G2A precision pressure sensor, and precision 0.1% components. The new pressure controller is a simple type II analog servo, with closed loop bandwidth of 2Hz, adequate for the expected disturbances in the Blanco. A novel non-linear amplifier stage compensates for the proportional pressure valve dead-band. Integration was completed in 3 days during daytime (Aug 2014) with no telescope downtime. It has performed flawlessly for 20 months. A 5-10x performance improvement over the old system is observed (below). Primary Mirror Cleaning Every 3 to 4 months: a semi-wet wash using HCl-treated natural sponges wet with deionized water and a little neutral soap.  Every week: CO2 snow. (Last coat: March 2011.) On a number of occasions the reflectivity decreased and scattering increased significantly.  This seems to occur at the end of the summer and is believed to derive from contamination of mirror cooling air with aerosolized hydraulic oil from the telescope bearings.  However, replacing the air ducts where such oil was evident has not had the desired effect and investigations are ongoing. New air compressors and driers have been installed to supply the primary mirror active optics system and provide purge air of the corrector optics.  The achieved dewpoint is -22°C. Reliability Immediately prior to the installation of DECam in 2012, typical uptime losses to technical failures from all sources were around 4%, representing a steady improvement over the decade preceding (from 6-8%).  Some temperamental instruments have since been retired and, of course, the general efficiency of DECam is much higher than was Mosaic II, with a 20sec readout time, compared to Mosaic II’s 100sec, regardless of the nearly 8 times areal coverage. DECam Reliability statistics - for the whole system (facility, telescope, DECam) while DECam has been scheduled - are impressive. For 2013 9.1% of scheduled observing time was lost, dominated by a LN2 pump failure, and a hexapod connector failure; for 2014 2.0% of time was lost, for 2015, with our worst winter weather for 34 years and when facility power was unreliable, 3.2% of time was lost, and until March 30 2016 0.8% of time was lost. Overall, the telescope and its subsystems account for less than one percent downtime in each year, the remainder being due to instrument problems and external issues such as earthquakes. References Flaugher, B., et al., “The Dark Energy Camera,” AJ, 150, 43, (2015) www.darkenergysurvey.org http://www.ctio.noao.edu/noao/content/DECam-Science-papers Workshop on Precision Astronomy with Fully Depleted CCDs, Brookhaven National Laboratory, 2013, IOP Journal of Instrumentation, Volume 9, (2014) Martini, P., Stoll, R., Derwent, M.A., et al., PASP, 123, 187-197 (2011) Stoll, R., Martini, P., Derwent, M.A., et al., Proc. SPIE 7735, 154-163 (2014) Martini P., Elias J., Points S. et al., Proc. SPIE 9147 91470Z, (2014) Points, S. D, James, D. J., Tighe, R., et al., Proc SPIE (2016) Schlawin, E., et al.., Proc SPIE Vol. 9147 (2014) Warner, M., et al.,, Proc. SPIE Vol.8451 (2012)