1. Remote observing with the Keck Telescopes from multiple sites in California
Robert Kibrick, Brian Hayes, Steve Allen
University of California Observatories / Lick Observatory
Al Conrad
W.M. Keck Observatory
Advanced Global Communications Technologies for Astronomy II
9/17/2018

2. Overview of Presentation
Background
The Keck Telescopes and telescope scheduling
Remote observing with the Keck Telescopes
From remote control room at summit (30 meters)
From Keck Headquarters in Waimea, Hawaii (32 km)
From Santa Cruz, California via Internet2 (3200 km)
Network Reliability Concerns
Providing a backup data path
Recent operational experience
Extending the model to multiple sites
9/17/2018

3. The Keck Telescopes
9/17/2018

4. Keck Telescopes use Classical Scheduling
Kecks not designed for queue scheduling
Schedules cover a semester (6 months)
Approved proposals get 1 or more runs
Each run is between 0.5 to 5 nights long
Gaps between runs vary from days to months
Half of all runs are either 0.5 night or 1 night long
9/17/2018

5. From 1993 to 1995, all Keck observing was done at the summit
Observers at the summit work from control rooms located adjacent to the telescope domes
9/17/2018

6. Conducting observations involves coordinated effort by 3 groups
Telescope operator (observing assistant)
Responsible for telescope safety & operation
Keck employee; normally works at summit
Instrument scientist
Expert in operation of specific instruments
Keck employee; works at summit or Waimea
Observers
Select objects and conduct observations
Employed by Caltech, UC, NASA, UH, or other
9/17/2018

7. Keck 2 Control Room at the Mauna Kea Summit
Telescope operator, instrument scientist, and observers work side by side, each at their own computer.
9/17/2018

8. Observing at the Mauna Kea summit is both difficult and risky
Oxygen is only 60% of that at sea level
Lack of oxygen reduces alertness
Observing efficiency significantly impaired
Altitude sickness afflicts some observers
Some are not even permitted on summit:
Pregnant women
Those with heart or lung problems
9/17/2018

9. Initiative to support remote observing from Keck Headquarters
1995: Remote control rooms built at Keck HQ
Initial tests via 1.5 Mbps (T1) link to the summit
1996: Videoconferencing connects both sites
Remote observing with Keck 1 begins
1997: >50% of Keck 1 observing done remotely
Link to the summit upgraded to 45 Mbps (DS3)
1999: remote observing >90% for Keck 1 and 2
2000: remote observing now the default mode
9/17/2018

10. The Remote Observing Facility at Keck Headquarters in Waimea
Elevation of Waimea is 800 meters
Adequate oxygen for alertness
Waimea is 32 km NW of Mauna Kea
45 Mbps fiber optic link connects 2 sites
A remote control room for each telescope
Videoconferencing for each telescope
On-site dormitories for daytime sleeping
9/17/2018

11. Keck 2 Remote Control Room at the Keck Headquarters in Waimea
Observer and instrument scientist in Waimea use video conferencing system to interact with telescope operator at the summit
9/17/2018

12. Keck 2 Remote Observing Room as seen from the Keck 2 summit
Telescope operators at the summit converse with astronomer at Keck HQ in Waimea via the videoconferencing system.
9/17/2018

13. Videoconferencing has proved vital for remote observing from Waimea
Visual cues (body language) important!
Improved audio quality extremely valuable
A picture is often worth a thousand words
Troubleshooting: live oscilloscope images
“Cheap” desktop sharing (LCD screens)
Chose dedicated versus PC-based units:
Original (1996) system was PictureTel 2000
Upgrading to Polycom Viewstations
9/17/2018

14. Interaction between video-conferencing and type of monitors
Compression techniques motion sensitive
“Moving” scene requires more bandwidth
CRT monitors cause “flicker” in VC image
Beating of frequencies: camera .vs. CRT
CRT phosphor intensity peaking, persistence
CRT monitor “flicker” causes problems:
Wastes bandwidth and degrades resolution
Visually annoying / nausea inducing
Use LCD monitors to avoid this problem
9/17/2018

15. The Keck Headquarters in Waimea
Most Keck technical staff live and work in Waimea. Allows direct contact between observers and staff. Visiting Scientist’s Quarters (VSQ) located in same complex.
9/17/2018

16. Limitations of Remote Observing from Keck HQ in Waimea
Most Keck observers live on the mainland.
Mainland observers fly > 3,200 km to get to Waimea
Collective direct travel costs exceed $400,000 U.S. / year
9/17/2018

17. Remote Observing from Waimea is not cost effective for short runs
Round trip travel time is 2 days
Travel costs > $1,000 U.S. per observer
About 50% of runs are for 1 night or less
Cost / run is very high for such short runs
Such costs limit student participation
9/17/2018

18. Motivations for Remote Observing from the U.S. Mainland
Travel time and costs greatly reduced
Travel restrictions accommodated
Sinus infections and ruptured ear drums
Late stages of pregnancy
Increased options for:
Student participation in observing runs
Large observing teams with small budgets
Capability for remote engineering support
9/17/2018

19. Mainland remote observing goals and implementation strategy
Target mainland facility to short duration runs
Avoid duplicating expensive Waimea resources
Avoid overloading Waimea support staff
Strategy:
No mainland dormitories; observers sleep at home
Access existing Waimea support staff remotely
Restrict mainland facility to experienced observers
Restrict to mature, fully-debugged instruments
9/17/2018

20. Mainland remote observing facility is an extension of Keck HQ facility
Only modest hardware investment needed:
Workstations for mainland remote observers
Network-based videoconferencing system
Routers and firewalls
Backup power (UPS) – especially in California!!!
Backup network path to Mauna Kea and Waimea
Avoids expensive duplication of resources
Share existing resources wherever possible
Internet-2 link to the mainland
Keck support staff and operational software
9/17/2018

21. Keck software is accessed the same regardless of observer’s location
The control computers at the summit:
Each telescope and instrument has its own computer
All operational software runs only on these computers
All observing data written to directly-attached disks
Users access data disks remotely via NFS or ssh/scp
The display workstations
Telescopes and instruments controlled via X GUIs
All users access these X GUIs via remote X displays
X Client software runs on summit control computers
Displays to X server on remote display workstation
9/17/2018

22. Overall Topology Type your question here, and then click Search.
9/17/2018

23. View Empty
9/17/2018

24. Santa Cruz Remote Observing Facility
9/17/2018

25. Santa Cruz Remote Observing Video Conferencing
9/17/2018

26. The Weather in Waimea
9/17/2018

27. Why did we choose this approach?
Operational Simplicity
Operational control software runs only at the summit
All users run identical software on same computer
Simplifies management between independent sites
Allowed us to focus on commonality
Different sites / teams developed instrument software
Large variety of languages and protocols were used
BUT: all instruments used X-based GUIs
9/17/2018

28. Remote observing differences: Waimea versus the mainland
System Management:
Keck summit and HQ share a common domain
Mainland sites are autonomous
Remote File Access:
Observers at Keck HQ access summit data via NFS
Observers on mainland access data via ssh/scp
Propagation Delays:
Summit to Waimea round trip time is about 1 ms.
Summit to mainland round trip time is about 100 ms.
9/17/2018

29. Increased propagation delay to mainland presents challenges
Initial painting of windows is much slower
But once created, window updates fast enough
All Keck applications display to Waimea OK
A few applications display too slowly to mainland
System and application tuning very important
TCP window-size parameter (Web100 Initiative)
X server memory and backing store
Minimize operations requiring round trip transactions
9/17/2018

30. Simulating long propagations delays in the lab
Instruments are designed and built on mainland
Software is debugged on local area network
Testing on LAN does not reveal delay problems
Must measure delay effects before deployment
Simulate WAN delays using NIST simulator
Requires Linux PC with dual Ethernet interfaces
Can select specific packets delays, losses, jitters
9/17/2018

31. Shared access and control of instruments
Most software for Keck optical instruments provides native multi-user/multi-site control
All users have consistent view of status and data
Instrument control can be shared between sites
Multipoint video conferencing key to coordination
Some single-user applications can be shared via X-based application sharing environments:
XMX
VNC
9/17/2018

32. Tradeoffs from this approach to remote observing
Disadvantages:
X protocol does not make optimal use of bandwidth
Long propagation delays require considerable tuning
Advantages:
Minimizes staffing requirements at mainland sites
Only “vanilla” hardware and software needed there
Simplifies sparing and swapping of equipment
Simplifies system maintenance at mainland sites
Simplifies authentication/access control
9/17/2018

33. Fast and reliable network needed for mainland remote observing
1997: 1.5 Mbps Hawaii -> Oahu -> mainland
1998: 10 Mbps from Oahu to mainland
1999: First phase of Internet-2 upgrades:
45 Mbps commodity link Oahu -> mainland
45 Mbps Internet-2 link Oahu -> mainland
2000: Second phase upgrade:
35 Mbps Internet-2 link from Hawaii -> Oahu
Now 35 Mbps peak from Mauna Kea to mainland
2002: 155 Mbs from Oahu to mainland
9/17/2018

34. End-to-end reliability is critical to successful remote operation
Keck Telescope time is valued at $1 per second
Observers won’t use facility if not reliable
Each observer gets only a few nights each year
What happens if network link to mainland fails?
Path from Mauna Kea to mainland is long & complex
At least 14 hops crossing 6 different network domains
While outages are rare, consequences are severe
Even brief outages cause session collapse & panic
Observing time loss can extend beyond outage
The real question is, what happens if the link to the mainland fails?
9/17/2018

35. Keck Observatory policy on mainland remote observing
If no backup data path is available from mainland site, at least one member of observing team must be in Waimea
Backup data path must be proven to work before mainland remote observing is permitted without no team members in Waimea
9/17/2018

36. Mitigation plan: install end-to-end ISDN-based fall-back path
Install ISDN lines and routers at:
Each mainland remote observing site
Keck 1 and Keck 2 control rooms
Fail-over and fall-back are rapid and automatic
Toll charges incurred only during network outage
Lower ISDN bandwidth reduces efficiency, but:
Observer retains control of observations
Sessions remain connected and restarts avoided
Prevents observer panic
9/17/2018

37. Summary of ISDN-based fallback path
Install 3 ISDN lines (6 B channels) at each site
Install Cisco 2600-series routers at each end
Quad BRI interfaces
Inverse multiplexing
Caller ID (reject connections from unrecognized callers)
Multilink PPP with CHAP authentication
Dial-on-demand (bandwidth-on-demand)
No manual intervention needed at either end
Fail-over occurs automatically within 40 seconds
Uses GRE tunnels, static routes, OSPF routing
9/17/2018

38. Running OSPF routing over a GRE tunnel
On each router, we configure 3 mechanisms:
A GRE tunnel to the other endpoint
A floating static route that routes all traffic to the other endpoint via the ISDN dialer interface
A private OSPF domain that runs over the tunnel
OSPF maintains its route through the tunnel only if the tunnel is “up”
OSPF dynamic routes take precedence over floating static route
9/17/2018

39. Fail-over to ISDN backup data path
If the Internet-2 path is “up”, OSPF “hello” packets flow across the tunnel between routers
As long as “hello” packets flow, OSPF maintains the dynamic route, so traffic flows through tunnel
If Internet-2 path is “down”, OSPF “hello” packets stop flowing, and OSPF deletes dynamic route
With dynamic route gone, floating static route is enabled, so traffic flows through ISDN lines
9/17/2018

40. Fall-back to the normal Internet-2 path
OSPF keeps trying to send “hello” packets through the tunnel, even with Internet is down
As long as Internet-2 path remains down the “hello” packets can’t get through
Once the Internet-2 path is restored, “hello” packets flow between routers
OSPF re-instates dynamic route through tunnel
All current traffic gets routed through the tunnel
All ISDN calls are terminated
9/17/2018

41. Operational costs of ISDN backup data path
Fixed leased cost is $70 per line per month
Three lines at each site -> $2,500 per site/year
Both sites -> $5,000/year
Long distance cost (incurrent only when active)
$0.07 per B-channel per minute
If all 3 lines in use:
$0.42 per minute
$25.20 per hour
9/17/2018

42. Recent operational experience
Remote observing science from Santa Cruz:
Low Resolution Imaging Spectrograph (LRIS)
Echellete Spectrograph and Imager (ESI)
Remote engineering and instrument support
ESI
High Resolution Echelle Spectrometer (HIRES)
Remote Commissioning Support
DEIMOS (see SPIE paper & )
9/17/2018

43. Unplanned use of the facility during week of Sept. 11, 2001
All U.S. commercial air traffic grounded
Caltech astronomers have a 5-day LRIS run on Keck-I Telescope starting September 13
No flights available
Caltech team leaves Pasadena morning of 9/13
Drives to Santa Cruz, arriving late afternoon
Online with LRIS well before sunset in Hawaii
9/17/2018

44. The hardest problem was the lodging!
LRIS operated from Santa Cruz all 5 nights
ISDN backup path activated several times
Observing efficiency comparable to Waimea
Lodging was the biggest problem
Motel check-in/check-out times incompatible
Required booking two motels for the same night
Motels are not a quiet place for daytime sleep
9/17/2018

45. Extending mainland remote observing to other sites
Other sites motivated by Santa Cruz success
Caltech remote facility is nearly operational
Equipment acquired
ISDN lines and router installed
Will be operational once routers are configured
U.C. San Diego facility being assembled
Equipment specified and orders in progress
Other U.C. campuses considering plans
9/17/2018

46. Administrative challenges: scheduling shared facilities
Currently only one ISDN router at Mauna Kea
Limits mainland operation to one site per night
Interim administrative solution
Longer term solution may require:
Installation of additional ISDN lines at Mauna Kea
Installation of an additional router at Mauna Kea
9/17/2018

47. Remaining challenges TCP/IP tuning of end-point machines
Needed to achieve optimal performance
Conflicts with using “off-the-shelf” workstations
Conflict between optimal TCP/IP parameters for the normal Internet-2 path .vs. the ISDN fall-back path
Hoping for vendor-supplied auto-tuning
Following research efforts of Web100 Project
Administrative challenges
Mainland sites are currently autonomous
Need to develop coordination with Keck
9/17/2018

48. Summary Model is being extended to multiple sites
Internet-2 makes mainland operation feasible
Backup data path protects against interruptions
Keck HQ is the central hub for remote operation
Mainland remote observing model is affordable:
Mainland sites operate as satellites of Keck HQ
Leverage investment in existing facilities and staff
Leverage investment in existing software
Share existing resources wherever feasible
Avoid expensive and inefficient travel for short runs
Model is being extended to multiple sites
9/17/2018

49. Acknowledgments U.S. National Science Foundation
U.S. Department of Defense
University of Hawaii
Gemini Telescope Consortium
University Corp. for Advanced Internet Development (UCAID)
Corporation for Education Network Initiatives in California (CENIC)
9/17/2018

50. Author Information Robert Kibrick, UCO/Lick Observatory
University of California, Santa Cruz
California 95064, U.S.A.
WWW:
Phone:
FAX:
9/17/2018