1. Remote Observing with the Keck Telescopes
Robert Kibrick, Director of Scientific Computing, University of California Observatories / Lick Observatory
Science, Culture, and Education over Internet2 Networks, April 4, 2001

2. Overview of Presentation
Background
The Keck Telescopes
Mauna Kea Observatories
Telescope Scheduling
Modes of Observing
Remote observing with the Keck Telescopes
From Keck Headquarters in Waimea, Hawaii (32 km)
From Santa Cruz, California via Internet2 (3200 km)
Operational models and issues
Live videoconference with astronomers at Keck
11/12/2018

3. The Keck Telescopes
11/12/2018

4. The Keck Telescopes
11/12/2018

5. Keck Telescope Facts Twin, 10-meter optical/infrared telescopes
Largest telescopes of this type in the world
Construction funded by W. M. Keck Foundation
Observing time shared between 4 institutions
California Institute of Technology (Caltech)
University of California (UC)
National Aeronautics and Space Adminstration (NASA)
University of Hawaii (UH)
Located atop 4,200 meter summit of Mauna Kea
11/12/2018

6. Mauna Kea summit on the island of Hawaii
11/12/2018

7. Mauna Kea Summit Dormant volcano
Premier astronomy site in N. Hemisphere
Above 90% of water vapor in atmosphere
Non-turbulent airflow over the summit
Sub-arcsecond atmospheric seeing
Home to many international observatories
11/12/2018

8. Mauna Kea Optical Observatories
UH 0.6 meter
UH 2.2 meter
Gemini North
CFHT
NASA IRTF
Keck-2
Keck-1
Subaru
11/12/2018

9. Largest Optical Telescopes in the Northern Hemisphere
Hubble Space Telescope Mirror is similar in size to UH 2.2-meter
Combined light gathering power of Mauna Kea telescopes is 50 times greater than HST
11/12/2018

10. Common Facilities Operated by University of Hawaii
Dormitories at Hale Pohaku
Altitude is 2,800m
All water must be trucked in
Acclimatization required before ascent to the 4,200m summit
11/12/2018

11. Mauna Kea Observatories Serve An International Community
Argentina
Australia
Brazil
Canada
Chile
France
Japan
Netherlands
Taiwan
United Kingdom
United States
11/12/2018

12. Assigning Telescope Time: Classical Scheduling
Committee evaluates observing proposals
Produce a 3 or 6 month telescope schedule
Approved proposals assigned a set of dates
Observing dates known many weeks in advance
Airline tickets can be purchased at lower rates
Astronomers can adjust course schedules
Proposal writers conduct their own observations
Can adjust observing program to weather conditions
Can alter program in case of unexpected discovery
11/12/2018

13. Assigning Telescope Time: Queue Scheduling
Committee evaluates observing proposals
Approved proposals placed in a queue
Computer selects queue entries nightly:
Specific visibility or timing requirements
Best match of current sky conditions
Proposal priorities
Observing dates not known in advance
Proposal authors do not conduct observations
11/12/2018

14. Queue scheduling works best if instruments can be changed quickly
Smaller, more recent telescopes, like the Subaru, have robotic mechanisms that permit instruments to be rapidly changed in the middle of the night.
This flexibility makes queue scheduling easier to implement.
11/12/2018

15. Keck instruments must be changed manually and only during the day
Instrument size scales with telescope size.
Keck instruments are massive – some weigh nearly 8,200 kilograms.
Keck instruments can only safely be changed manually and during the daytime.
11/12/2018

16. Keck Telescopes use Classical Scheduling
Kecks not designed for queue scheduling
Schedules cover a semester (6 months)
Approved proposals get 1 or more runs
Each run is between 0.5 to 4 nights long
Gaps between runs vary from days to months
Half of all runs are either 0.5 or 1 night long
Separate schedules for the two Kecks
11/12/2018

17. Keck Telescope Schedule
11/12/2018

18. Non-queue Scheduled Observing Modes
Local Observer
Controls observation from the telescope site
Remote Observer
Controls observation from a remote site
Service Observer
Observes in place of a remote observer
Remote observer submits detailed object list
Service observer conducts observation locally
11/12/2018

19. Keck Telescope Observing Modes
Modes neither supported nor planned:
Service observing
Queue scheduled observing
Supported observing modes:
Local observing
Remote observing
Astronomers who are granted observing time conduct their own observations
11/12/2018

20. From 1993 to 1995, all Keck observing was done at the summit
Observers at the summit work from control rooms located adjacent to the telescope domes
11/12/2018

21. Conducting observations involves coordinated effort by 3 groups
Telescope operator (observing assistant)
Responsible for telescope safety & operation
Keck employee; normally works at summit
Instrument scientist
Expert in operation of specific instruments
Keck employee; works at summit or Waimea
Observers
Select objects and conduct observations
Employed by Caltech, UC, NASA, UH, or other
11/12/2018

22. Keck 2 Control Room at the Mauna Kea Summit
Telescope operator, instrument scientist, and observers work side by side, each at their own computer.
11/12/2018

23. Observing at the Mauna Kea summit is both difficult and risky
Oxygen is only 60% of that at sea level
Lack of oxygen reduces alertness
Observing efficiency significantly impaired
Altitude sickness afflicts some observers
Some are not even permitted on summit:
Pregnant women
Those with heart or lung problems
11/12/2018

24. Initiative to support remote observing from Keck Headquarters
1995: Remote control rooms built at Keck HQ
Initial tests via 1.5 Mbps (T1) link to the summit
1996: Videoconferencing connects both sites
Remote observing with Keck 1 begins
1997: >50% of Keck 1 observing done remotely
Link to the summit upgraded to 45 Mbps (DS3)
1999: remote observing >90% for Keck 1 and 2
2000: remote observing is now the default mode
11/12/2018

25. Keck 2 Remote Control Room at the Keck Headquarters in Waimea
Observer and instrument scientist in Waimea use video conferencing system to interact with telescope operator at the summit
11/12/2018

26. Videoconferencing has proved vital for remote observing from Waimea
Visual cues (body language) important!
Improved audio quality extremely valuable
A picture is often worth a thousand words
Chose dedicated versus PC-based units:
Original (1996) system was PictureTel 2000
Upgrading to Polycom Viewstations
11/12/2018

27. Keck 2 Remote Observing Room as seen from the Keck 2 summit
Telescope operators at the summit converse with astronomer at Keck HQ in Waimea via the videoconferencing system.
11/12/2018

28. The Remote Observing Facility at Keck Headquarters in Waimea
Elevation of Waimea is 800 meters
Adequate oxygen for alertness
Waimea is 32 km NW of Mauna Kea
45 Mbps fiber optic link connects 2 sites
A remote control room for each telescope
Videoconferencing for each telescope
On-site dormitories for daytime sleeping
11/12/2018

29. The Keck Headquarters in Waimea
Most Keck technical staff live and work in Waimea. Allows direct contact between observers and staff. Visiting Scientist’s Quarters (VSQ) located in same complex.
11/12/2018

30. Limitations of Remote Observing from Keck HQ in Waimea
Most Keck observers live on the mainland.
Mainland observers fly > 3,200 km to get to Waimea
Collective direct travel costs exceed $400,000 U.S. / year
11/12/2018

31. Remote Observing from Waimea is not cost effective for short runs
Round trip travel time is 2 days
Travel costs > $1,000 U.S. per observer
About 50% of runs are for 1 night or less
Cost / run is very high for such short runs
Such costs limit student participation
11/12/2018

32. Motivations for Remote Observing from the U.S. Mainland
Travel time and costs greatly reduced
Travel restrictions accommodated
Sinus infections and ruptured ear drums
Late stages of pregnancy
Increased options for:
Student participation in observing runs
Large observing teams with small budgets
Capability for remote engineering support
11/12/2018

33. Fast and reliable network needed for mainland remote observing
1997: 1.5 Mbps Hawaii -> Oahu -> mainland
1998: 10 Mbps from Oahu to mainland
1999: First phase of Internet-2 upgrades:
45 Mbps commodity link Oahu -> mainland
45 Mbps Internet-2 link Oahu -> mainland
2000: Second phase upgrade:
35 Mbps Internet-2 link from Hawaii -> Oahu
Now 35 Mbps peak from Mauna Kea to mainland
11/12/2018

34. Internet-2 links
11/12/2018

35. Mainland remote observing goals and implementation strategy
Target mainland facility to short duration runs
Avoid duplicating expensive Waimea resources
Avoid overloading Waimea support staff
Strategy:
No mainland dormitories; observers sleep at home
Access existing Waimea support staff remotely
Restrict mainland facility to experienced
Restrict to mature, fully-debugged instruments
11/12/2018

36. Mainland remote observing facility is an extension of Keck HQ facility
Only modest hardware investment needed:
Workstations for mainland remote observers
Network-based videoconferencing system
Routers and firewalls
Backup power (UPS) – especially in California!!!
Backup network path to Mauna Kea and Waimea
Avoids expensive duplication of resources
Share existing resources wherever possible
Internet-2 link to the mainland
Keck support staff and operational software
11/12/2018

37. Keck software is accessed the same regardless of observer’s location
The control computers at the summit:
Each telescope and instrument has its own computer
All operational software runs only on these computers
All observing data written to directly-attached disks
Users access data disks remotely via NFS or ssh/scp
The display workstations
Telescopes and instruments controlled via X GUIs
All users access these X GUIs via remote displays
X Client software runs on summit control computers
Displays to X server on remote display workstation
11/12/2018

38. Why did we choose this approach?
Operational Simplicity
Operational control software runs only at the summit
All users run identical software on same computer
Simplifies management between independent sites
Allowed us to focus on commonality
Different sites / teams developed instrument software
Large variety of languages and protocols were used
BUT: all instruments used X-based GUIs
11/12/2018

39. Focus effort on X standardization and optimization over long links
Maintain consistent X environment between sites
Optimize X performance between sites
Eliminates need to maintain:
Diverse instrument software at multiple sites
Diverse telescope software at multiple sites
Coordinate users accounts at multiple sites
Fewer protocols for firewalls to manage
11/12/2018

40. Accessing Keck software and data from Keck HQ in Waimea
Telescope operator uses display workstation at summit.
Instrument scientist and observers use display workstations in Waimea.
11/12/2018

41. Accessing Keck software and data from the mainland
Telescope operator uses display workstation at summit.
Instrument scientist uses display in Waimea
Observers use display on mainland
11/12/2018

42. Remote observing differences: Waimea versus the mainland
System Management:
Keck summit and HQ share a common domain
Mainland sites are autonomous
Remote File Access:
Observers at Keck HQ access summit data via NFS
Observers on mainland access data via ssh/scp
Propagation Delays:
Summit to Waimea round trip time is about 1 ms.
Summit to mainland round trip time is about 100 ms.
11/12/2018

43. Increased propagation delay to mainland presents challenges
Initial painting of windows is much slower
But once created, window updates fast enough
All Keck applications display to Waimea OK
A few applications display too slowly to mainland
System and application tuning very important
TCP window-size parameter (Web100 Initiative)
X server memory and backing store
Minimize operations requiring round trip transactions
11/12/2018

44. Tradeoffs from this approach to remote observing
Disadvantages:
X protocol does not make optimal use of bandwidth
Long propagation delays require considerable tuning
Advantages:
Minimizes staffing requirements at mainland sites
Only “vanilla” hardware and software needed there
Simplifies sparing and swapping of equipment
Simplifies system maintenance at mainland sites
11/12/2018

45. End-to-end reliability is critical to successful remote operation
Keck Telescope time is valued at $1 per second
Each observer gets only a few nights each year
Observers won’t use facility if not reliable
What happens if network link to mainland fails?
Path from Mauna Kea to mainland is long & complex
At least 14 hops crossing 7 different network domains
While outages are rare, consequences are severe
Even brief outages cause session collapse & panic
Observing time loss can extend beyond outage
11/12/2018

46. Mitigation plan: install end-to-end ISDN-based fallback path
Install ISDN lines and routers at:
Each mainland remote observing site
Keck 1 and Keck 2 control rooms
Fail-over and fallback are rapid and automatic
Toll charges incurred only during network outage
Lower ISDN bandwidth reduces efficiency, but:
Observer retains control of observations
Sessions remain connected and restarts avoided
Prevents observer panic
11/12/2018

47. Summary of ISDN-based fallback path
Install 3 ISDN (6 B channels) between sites
Install Cisco 2600-series routers at each end
Dual auto-sensing Ethernet interfaces
Quad BRI interfaces
Inverse multiplexing
Dial-on-demand (bandwidth-on-demand)
Caller ID (reject connections from unrecognized callers)
Multilink PPP with CHAP authentication
Uses GRE tunnels and OSPF routing
No manual intervention needed at either end
11/12/2018

48. Summary of progress to date
Prototype mainland facility assembled at UCSC
Multipoint videoconferencing to summit and HQ
Efficiently displays telescope & instrument status
Two Keck instruments operated remotely:
High Resolution Spectrometer (HIRES) on Keck 1
Echellette Spectrogram & Imager (ESI) on Keck 2
Primary use has been for remote engineering
On-sky remote observing performed with ESI
Efficient, automated transfer of image files
11/12/2018

49. Future Plans Conduct ongoing trials using prototype at UCSC
Work out operational details with Keck staff
Add capability to operate additional instruments
Aim for fully operational status by late 2001
Extend to other sites once debugged at UCSC
11/12/2018

50. Items Currently in Progress
Real time display of guide camera images
Installation of ISDN lines at the Keck summit
Installation of ISDN routers at the Keck summit
Adjustments to telescope scheduling procedures
Firewall issues at summit and mainland sites
11/12/2018

51. Summary Internet-2 makes mainland operation feasible
Proposed model should be affordable:
Mainland sites operate as satellites of Waimea
Leverage investment in existing facilities and staff
Leverage investment in existing software
Share existing resources wherever feasible
Maintain Waimea as focal point for remote operation
Avoid wasteful duplication of resources
Avoid expensive and inefficient travel for short runs
11/12/2018

52. Mainland remote observing gives observers a choice
Yields significant advantages for short runs
Provides alternatives when travel is difficult
Increases options for student participation
Increases options for large observing teams
Provides option of multi-site collaborations
Waimea remains available for longer runs
11/12/2018

53. Acknowledgments U.S. National Science Foundation
U.S. Department of Defense
University of Hawaii
Gemini Telescope Consortium
University Corp. for Advanced Internet Development (UCAID)
Corporation for Education Network Initiatives in California (CENIC)
11/12/2018

54. Author Information Robert Kibrick, UCO/Lick Observatory
University of California, Santa Cruz
California 95064, U.S.A.
WWW:
Phone:
FAX:
11/12/2018