1. CTIO Facilities Improvement Project (CFIP) Tim Abbott Joint Review Jan 2008

2. Project Components CFIP consists of five independent tasks: – Replace the telescope control system (TCS). – Definitively repair the primary mirror radial support system. – Evaluate the effectiveness of the Environment Control System (ECS). – Build a cleanroom adjacent to the telescope that will accommodate DECam. – Install utility and cryogen lines to DECam specifications. 2T. Abbott, CFIP

3. TCS Upgrade

4. TCS Upgrade – Motivation & Goals Keep Blanco operating as a state-of-the art facility – Normal cycle of upgrade & enhancement Retire & replace obsolete components – Replace wire-wrap & bread-boarded components with off-the-shelf and PCB components – Use technology with an estimated >5 year market life Maximize telescope efficiency for DECam/DES/NEWFIRM operations – Integrate all existing instruments Modernize servo – Use new encoders – Decouple mount, dome, rotator/guider from TCS Modernize software & control – Introduce independent local and/or remote intelligence where appropriate – Modularize Improve maintainability Use SOAR/LSST model & components – LSST prototyping, SOAR backfits Record and make available all telemetry from telescope to weather The upgrade is to be performed with existing TCS in place 4T. Abbott, CFIP

5. Specifications Slew requirement: 2 degrees in 17 sec – To match DECam readout time Slew, project goal: 3 degrees in 20 seconds, track-to- track Tracking: <0.5” drift in 60 sec everywhere within the DES area Jitter: 100 mas r.m.s. Pointing accuracy: 10” rms (goal: 5” rms) Guide error signal delivery rate: 1 Hz Operational deadline: Sept 2010. 5T. Abbott, CFIP

6. Blanco 4-m Upgraded TCS Diagram Operator GUI Mount Control TCS App Kernel Dome/ Utility 4MAPIMANROT4MF8SECCFADC DECAMISPIHYDRAMOSAIC Guider Interface Lamps Interface Dome Interface Ethernet RS-485 Mount Interface Dome PLC Dome Encoders Comparison Lamps Guider Motors Guider Encoders Ra/Dec Motors Handpaddle Interlocks Encoders UDP, TCP/IP Ethernet/ Serial Instruments Motor Controllers (SMC) Facility Database Weather, MASS DIMM, Sky Cam. Time IRIG-B Remote Ops IMAN PC-Guider NEWFIRM TCS Router Linux PC Interface Instrument SMC Subsystem 6T. Abbott, CFIP Key

7. Servo Upgrade Of the telescope drive servo, only the existing motors will remain. The encoders, motor control hardware and software, and the power drivers will all be replaced. 7T. Abbott, CFIP

8. Present Servo Racks and the Mount Control Loop M1 Tach 1 M2 Tach 2 Inc. RA Encoders Abs. RA TCS on VME Chassis VME PMAC Heurikon Processor RA Speed Ref. Slew/Track Select. M3 Tach 3 M4 Tach 4 Inc. DEC Abs. DEC DEC Speed Ref. Slew/Trac kSelect. RA Servo Rack DEC Servo Rack Mount RA Manual Slew DEC Manual Slew VME 9100 8T. Abbott, CFIP

9. Servo Drivers Schematics Velocity Comparator Card PMAC Interface Card Alarm Control Card Power Drivers DC Current Amplifier Control Card Motors and Tacho- Generators Slew Control 50 VDC Power Supply 9T. Abbott, CFIP

10. Delta Tau Products Compact UMAC Geobrick UMAC: Universal Multi-Axes Controller All those discrete components convert to remotely modifiable code 10T. Abbott, CFIP

11. 11 Proposed Tape Encoder Signal Diagram ERA881 Tape with 2 Read Heads ERA881 Tape with 2 Read Heads RA AXIS DEC AXIS PMAC 1:4096 Interpolators (4) PMAC CARD This design is used on almost all modern telescopes Read Head Tape T. Abbott, CFIP

12. 12 Locations of new tape encoders RA ( D=4.3m ) DEC ( D=1.25m ) Read Head1 Read Head2 RA TAPE DEC TAPE Read Head2 Read Head1 Mounted on RA oil bearing unused surface, 9m tape provides 240° coverage 2 read heads separated by 60° Custom mount, 3m tape provides 270° coverage 2 read heads at 90° separation T. Abbott, CFIP

13. 13 Enhanced telescope safety The existing system is lacking a redundant rate limiter independent of the drive system. We will install an inertial rate measuring system, in both axes which will shutdown the motors in the event the rate limit is exceed. Use a similar system as SOAR, using inexpensive MEMS inertial sensors. T. Abbott, CFIP

14. TCS Upgrade, Software Upgrade Control System to overcome shortcomings on performance: – Slew-Track transition DECAM requirements – Accuracy of motion demand computation – Upgrade/Replace obsolete components Use Blanco 4M as a prototype for LSST TCS: » Roots in the SOAR project. – Kernel TCS Software – Wallace, RAL – Communications Middleware (DDS) – Facility Database – LSST Modularization: Supervisory Control Distributed Processes – Remote Operations 14T. Abbott, CFIP

15. TCS Software Tools PC Linux Operating System – RTAI extensions. DDS Middleware (RTI baseline) LabVIEW C/C++ MySQL Database – Real Time Connect Module SubVersion package for Version Control. 15T. Abbott, CFIP

16. Principle of Control: Supervisory Control Control (PID) IN OUT Device Process setpoint Status Control Application Log - CA performs supervisory control: computes the “setpoint” - Time critical loops are closed locally. - Status is needed for monitoring and for maintenance. + telemetry 16T. Abbott, CFIP

17. Control System Concept Integration of a distributed system SCADA (Supervisory Control And Data Acquisition) is a technology utilized to integrate distributed systems. – Send control instructions to distributed subsystems – Provides synchronization – Collects data from remote subsystems. Multi-tier Hierarchical Control with Local Autonomy – Hierarchical nature of the Control System is realized with a Master/Slave paradigm. Control is exerted via message passing. 17T. Abbott, CFIP

18. TCS FACILITY DATABASE ● The current estimation of data load for the TCS Facility Database is 535 Mbytes/day. T. Abbott, CFIP

19. TCS GUI For the GUI design, the SOAR model will be taken. A set of GUIs for regular operations and a set of specialized GUIs for engineering and maintenance purposes. ● LabVIEW is the main tool for the GUIs, and other web-based new technologies will be explored. T. Abbott, CFIP

20. TCS Upgrade, Project Staff Tim Abbott (Project manager/scientist, DECam deputy project manager) German Schumacher (Software manager) Ricardo Schmidt (Electronics group manager) Eduardo Mondaca (Electronics engineer, interfaces, hardware, servos) Michael Warner (Servos, hardware) Manuel Martinez (Connectivity, motor controllers) Rolo Cantarutti (Software) Omar Estay (Software) Francisco Delgado (Software) Andres Montane & group (Mechanics, fab) CTIO DSS – Gale Brehmer (mgr), Enrique Schmidt, David Rojas, Javier Rojas, Humberto Orrego. 20T. Abbott, CFIP

21. Software Timeline Jul-Sep 07Oct-Dec 07Jan-Mar 08Apr-Jun 08Jul-Sep 08Oct-Dec 08Jan-Mar 09Apr-Jun 09Jul-Sep 09Oct-Dec 09Jan-Mar 10 Servo Control Mount Software Serial Software TCS Application SMC Control Dome/ Utility Guider Instruments Interfaces Operator GUI Control Design Purchases Commis. GS 50% MW 50%MW 70% MW 50% OE 75% MM 50% RC 10%RC 70% FD 50% Facility Database FD 70% 200820092010 FD: Francisco Delgado GS: Germán Schumacher MM: Manuel Martínez MW: Michael Warner OE: Omar Estay RC: Rolando Cantarutti 21T. Abbott, CFIP

22. Electronics Timeline Jul-Sep 07Oct-Dec 07Jan-Mar 08Apr-Jun 08Jul-Sep 08Oct-Dec 08Jan-Mar 09Apr-Jun 09Jul-Sep 09Oct-Dec 09Jan-Mar 10 SMC Tests TCS Application Tests Support Dome/Utility Interface D&I Guider Tests Instruments Interfaces D&I Conceptual Design Mount Interfaces and Power Drivers Purchases Commis. EM 60% ET 30% ED 30% DOME Tests 200820092010 Mount Tuning Encoder Tests Guider Interface D&I Instruments Tests D&I = Design and Implementation 22T. Abbott, CFIP

23. Capital costs 23T. Abbott, CFIP

24. Radial Supports

25. Primary mirror radial supports 25T. Abbott, CFIP

26. Radial Supports The Blanco radial support system has a history of breaking off individual supports, with increased numbers of failed supports in recent years (up to 2005). One consequence is that the primary mirror can move on its cell with a path that depends on its recent pointing history (i.e. there is hysteresis) and an amplitude which significantly affects image quality (through coma). Recent work has greatly improved our knowledge of how the telescope actually works as opposed to how it is supposed, or was intended, to work. 26T. Abbott, CFIP

27. 4 Surprises 1.Primary mirror moves on its cell 2.Radial support attachment to the primary mirror is marginal 3.Radial supports bend 4.Misalignments have accumulated with each radial support repair 27T. Abbott, CFIP

28. Surprise #1: Primary Mirror Moves on its Cell With amplitudes on the ~1mm scale depending on the number of broken radial supports. – Sufficient to contribute significantly to image quality With hysteresis – Location of primary mirror depends on history of telescope pointing, i.e. unpredictable. Displacement gauges installed between mirror & cell Laser & camera combination used to explore primary mirror movements with respect to the prime focus corrector. 28T. Abbott, CFIP

29. Recent displacement measurements Note: even if the primary mirror did not move on its cell, there are still significant other sources of prime focus translations/tilts with respect to the primary mirror – probably caused by flexure in the telescope structure. 29T. Abbott, CFIP

30. Surprise #2: The “H-bar” is marginal Significant peeling stresses can develop in the epoxy between invar and glass and the join is marginal. When a join fails, it can take glass plugs with it. 30T. Abbott, CFIP

31. A new design H-bar Rotational freedom at the ends of the legs reduces peeling stresses in the Invar/Cervit join. There is no significant slack in the joints. 4 units were installed in 2005, including some very vulnerable locations – none have yet failed. 31T. Abbott, CFIP

32. Surprise #3: Radial supports bend Radial supports (& the mirror) want to fall. Contact with the plenum or cell is likely, throwing the system out of balance and encouraging mirror movement on the cell. The plenum has been removed with no apparent collateral consequences. 32T. Abbott, CFIP

33. Surprise #4: Accumulated misalignments of radial supports Errors in reinstalling broken supports have accumulated o Due to misunderstanding, miscommunication Available lateral slack in radial supports is ~±1mm o If mirror moves, supports are inevitably driven into critical conditions. 33T. Abbott, CFIP

34. Back on the straight and narrow The primary mirror cannot be directly constrained. Realigning the radial supports will expand the mirror’s comfort zone again, making radial support breaks less likely. Replacing the remaining old style H-bars with the new design, on fresh glass, will make radial support breaks less likely. The current plan, still under study to test the completeness of our theory, is to hold a major shutdown, remove all the radial supports, rotate the primary mirror 5° on the cell, then reattach the supports at carefully calibrated locations using new design H-bars throughout. Shutdown is expected in 2 nd semester 2008 or 1 st semester 2009 and will last ~20 days. The majority of preparation is in the manufacture of 20 new H-bars and the design of an installation procedure. 34T. Abbott, CFIP

35. ECS, Cleanroom, Supplies

36. Environment Control System A survey with a thermal imaging camera in 2004 showed that the primary heat sources in the Blanco dome are the prime focus instrumentation, the Cassegrain instrumentation, the old control room and the telescope hydrostatic bearings. The old control room will be rendered redundant by the new TCS and decommissioning will be completed. DECam will incorporate active thermal controls, of which Mosaic II possesses none. Further thermal control measures will be explored as time and resources allow. Such work is not expected to impact on the DECam deployment schedule. 36T. Abbott, CFIP

37. Instrument Maintenance Facility (Cleanroom+) A cleanroom and associated equipment suitable for the handling needs of all existing instruments, NEWFIRM and DECam, will be permanently installed in the large Coudé room. The cleanroom will provide clean air of order class 10,000, and clean hoods will be provided for direct handling of particularly vulnerable components. Cleanroom Compressor room NW service platform 37T. Abbott, CFIP We are currently gathering specifications for DECam and NEWFIRM

38. Utility & Cryogen Lines DECam will require supplies of: – Dry N2 – Clean, dry air – High pressure gas lines – Glycol, water/alcohol coolants – Power – Signal & fiber optic lines CTIO is responsible for the installation of these lines. The installation plan will develop as the DECam final design matures. 38T. Abbott, CFIP

39. Spare slides 39T. Abbott, CFIP

40. Patch Panel To/From Telescope To/From VME Chassis Patch Connectors 40T. Abbott, CFIP

41. Replace: TCS s/w & platform – Kernel – Communications – Dome & utility control GUI Mount Control Telescope drive power amplifiers Encoders (SMCs later) Leave: Telescope motors Guiders Active optics Rotator f/8 control Instruments (SMCs, for the time being) 41T. Abbott, CFIP

42. 42 Existing Servo Block Diagram (Single Axis) PMAC MOTOR AMP1 Motor1 Tach1 Motor2 Tach2 Rate Cmd TCS Incr Enc ABS MOTOR AMP1 RATE LOOP Preload Slew/Track Gain & Filter Select Interlocks 25:1 Interpolator MOTOR DRIVER RACK Interface Card VME + + Motor Drivers are single ended T. Abbott, CFIP

43. 43 New System Block Diagram ( Single Axis ) PMAC DUAL MOTOR H-BRIDGE DRIVER Motor1 Tach1 Motor2 Tach2 Torque Cmd1 TCS Incr Enc ABS Interlocks Rate Loop, and Motor Preload Logic is done by PMAC + - - + Torque Cmd2 T. Abbott, CFIP

44. New Servo Model Developed lumped mass model in each axis Calculated maximum acceleration – There is plenty of margin for DECam Synthesized optimal slew trajectory A 3° slew in 15 seconds has maximum velocity of 0.5°/sec and maximum acceleration of 0.1°/s 2, within the rate limits established by the RA oil bearing. 44T. Abbott, CFIP

45. Encoders Current incremental encoders alone limit slew speed to 1°/s due to mechanical slippage and are therefore used for tracking and small offsets. The current system uses the (coarser) absolute encoders to achieve higher slew speeds. The upgrade will use Heidenhain tape encoders for all telescope speeds in both axes. These will be calibrated off the existing absolute encoders. 45T. Abbott, CFIP

46. 46 Incremental Encoder Comparison T. Abbott, CFIP

47. 47 RA oil bearing surface, radial deflection with motion RA oil bearing surface, axial deflection with motion T. Abbott, CFIP

48. 48 RA bearing deflection when oil bearing is energized OIL BEARING ENERGIZED T. Abbott, CFIP

49. 49 RA bearing surface Data shows that bearing surface motions are repeatable and stay within a narrow range, the radial motion meets Heidenhain’s requirements, we need to better understand the axial motions. The gauges were mounted in the oil bearing structure, for convenience, we need to repeat the tests at the final read head position, including the support structure, as the results might be different. Work continues… T. Abbott, CFIP

50. TCS GUI The GUIs will inject the commands and receive the status through the middleware, allowing seamless coexistence of local GUIs, remote GUIs and external commandments from the master instrument. DDS middleware LOCAL GUI REMOTE GUI ENGINEERING GUI TCS T. Abbott, CFIP

51. MESSAGE TYPES Signals – Rapidly generated and time-critical data. In most instances, it is more important to get the next issue than to retry a dropped one. Events – Asynchronously generated, priority messages which must be delivered reliably. Commands – Sequential instructions which must be received in order. Status – Persistent data about state or goals. Its timeliness differs from one application to the next Requests – Two-way request-reply transactions for a specific service or data. A messaging system allows separate, uncoupled applications to reliably communicate asynchronously. 51T. Abbott, CFIP

52. Data Distribution Service Technology to handle control requirements (message passing) and telemetry requirements, in a distributed environment. Publish-Subscribe paradigm. – Producer-Consumer Just declare your intent to publish or receive data. – Data abstracted in terms of topics. Quality of Service (QoS) layer for real time conformance. RTI (Real-Time Innovations) implementation is baseline. OMG DDS standard released. API: Application Programming Interface T. Abbott, CFIP

53. COMMON SERVICES Connection Service – Means for applications in a distributed environment to locate and connect to other applications. Event Service – Support high-performance publish/subscribe communications. Command Service – Provide Client/Server communications for application control. Logging Service (Telemetry Capture) – Collects, records, distribute and analyze system messages. Persistent Store Service – Holds system configuration information, calibration information, performance data, etc. Error Handling Service – Monitors for improper behavior. Supports recovery operations. Reports errors to applications and users. 53T. Abbott, CFIP

54. Transport DDS Infrastructure The Software Infrastructure Distributed Applications Per Topic Quality of Service Configuration Topic Based Anonymous Communications Publish / Subscribe Interface Dynamic Network Architecture COMMON SERVICES 54T. Abbott, CFIP

55. Use Case: Storing Published Data Into a Database TCS (DDS) MOUNT (DDS) UTILITY (DDS) Analysis (SQL) Monitor (SQL) Relational Database RTI Real-Time Connect DDS Write DDS TakeSQL UPDATE / INSERT SQL SELECT 55T. Abbott, CFIP

56. Global Data Space DDS: Pub/Sub Scenarios One to One Data Writer PublisherSubscriber Data Reader Subscriber Data Reader Subscriber Data Reader Data Writer Publisher One to Many Many to One Many to Many Topic SSS 56T. Abbott, CFIP

57. Event Screens Data Subscribers Publisher Loop Event Loop Analysis Loop INTRANET Telemetry Data Science Metadata Real Time Database Devices Site Telescope System Telemetry System Facility Database Telemetry Concept 57T. Abbott, CFIP

58. Client/Server Point to Point Connection 58T. Abbott, CFIP

59. Facility Database SensorsControllers Engineering Screens Operator Screens TCSInstruments MountOpticsDome Communications Middleware (DDS) System Diagram 59T. Abbott, CFIP

60. Version Control All the Software development is under version control. The organization of the software in the repository is module based instead of file based, allowing the tagging and release of meaningful SW components. Open source tool SUBVERSION is used, with the corresponding additions to allow full functionality over LabVIEW binary code and other text based languages and components. A version control server and repository PC will be installed at CTIO computer room. T. Abbott, CFIP

61. N WE S 16 Current status of primary radial supports = broken = new H-bar 10 14 3 4 20 22 12 5 6 7 8 9 11 12 13 15 17 18 19 21 23 24 61T. Abbott, CFIP

62. The road to ruin The primary mirror can have no hard restraints against movement on its cell. Some movement is inevitable - imbalance, abuse, earthquake If the primary mirror moves so as to drive a radial support into a bound condition, it may break. If the primary mirror is shocked while at high zenith angle, a radial support may break. Errors of alignment of the repaired radial supports systematically reduce the safe range of movement of the primary mirror on the cell. As the safe range of movement of the primary mirror on the cell decreases, it becomes easier to drive a support into a bound condition and support breaks become more likely. Iterate. 62T. Abbott, CFIP

63. ! 63T. Abbott, CFIP