Presentation is loading. Please wait.

Presentation is loading. Please wait.

Robert Kibrick, Brian Hayes, Steve Allen

Similar presentations


Presentation on theme: "Robert Kibrick, Brian Hayes, Steve Allen"— Presentation transcript:

1 Remote observing with the Keck Telescopes from multiple sites in California
Robert Kibrick, Brian Hayes, Steve Allen University of California Observatories / Lick Observatory Al Conrad W.M. Keck Observatory Advanced Global Communications Technologies for Astronomy II 9/17/2018

2 Overview of Presentation
Background The Keck Telescopes and telescope scheduling Remote observing with the Keck Telescopes From remote control room at summit (30 meters) From Keck Headquarters in Waimea, Hawaii (32 km) From Santa Cruz, California via Internet2 (3200 km) Network Reliability Concerns Providing a backup data path Recent operational experience Extending the model to multiple sites 9/17/2018

3 The Keck Telescopes 9/17/2018

4 Keck Telescopes use Classical Scheduling
Kecks not designed for queue scheduling Schedules cover a semester (6 months) Approved proposals get 1 or more runs Each run is between 0.5 to 5 nights long Gaps between runs vary from days to months Half of all runs are either 0.5 night or 1 night long 9/17/2018

5 From 1993 to 1995, all Keck observing was done at the summit
Observers at the summit work from control rooms located adjacent to the telescope domes 9/17/2018

6 Conducting observations involves coordinated effort by 3 groups
Telescope operator (observing assistant) Responsible for telescope safety & operation Keck employee; normally works at summit Instrument scientist Expert in operation of specific instruments Keck employee; works at summit or Waimea Observers Select objects and conduct observations Employed by Caltech, UC, NASA, UH, or other 9/17/2018

7 Keck 2 Control Room at the Mauna Kea Summit
Telescope operator, instrument scientist, and observers work side by side, each at their own computer. 9/17/2018

8 Observing at the Mauna Kea summit is both difficult and risky
Oxygen is only 60% of that at sea level Lack of oxygen reduces alertness Observing efficiency significantly impaired Altitude sickness afflicts some observers Some are not even permitted on summit: Pregnant women Those with heart or lung problems 9/17/2018

9 Initiative to support remote observing from Keck Headquarters
1995: Remote control rooms built at Keck HQ Initial tests via 1.5 Mbps (T1) link to the summit 1996: Videoconferencing connects both sites Remote observing with Keck 1 begins 1997: >50% of Keck 1 observing done remotely Link to the summit upgraded to 45 Mbps (DS3) 1999: remote observing >90% for Keck 1 and 2 2000: remote observing now the default mode 9/17/2018

10 The Remote Observing Facility at Keck Headquarters in Waimea
Elevation of Waimea is 800 meters Adequate oxygen for alertness Waimea is 32 km NW of Mauna Kea 45 Mbps fiber optic link connects 2 sites A remote control room for each telescope Videoconferencing for each telescope On-site dormitories for daytime sleeping 9/17/2018

11 Keck 2 Remote Control Room at the Keck Headquarters in Waimea
Observer and instrument scientist in Waimea use video conferencing system to interact with telescope operator at the summit 9/17/2018

12 Keck 2 Remote Observing Room as seen from the Keck 2 summit
Telescope operators at the summit converse with astronomer at Keck HQ in Waimea via the videoconferencing system. 9/17/2018

13 Videoconferencing has proved vital for remote observing from Waimea
Visual cues (body language) important! Improved audio quality extremely valuable A picture is often worth a thousand words Troubleshooting: live oscilloscope images “Cheap” desktop sharing (LCD screens) Chose dedicated versus PC-based units: Original (1996) system was PictureTel 2000 Upgrading to Polycom Viewstations 9/17/2018

14 Interaction between video-conferencing and type of monitors
Compression techniques motion sensitive “Moving” scene requires more bandwidth CRT monitors cause “flicker” in VC image Beating of frequencies: camera .vs. CRT CRT phosphor intensity peaking, persistence CRT monitor “flicker” causes problems: Wastes bandwidth and degrades resolution Visually annoying / nausea inducing Use LCD monitors to avoid this problem 9/17/2018

15 The Keck Headquarters in Waimea
Most Keck technical staff live and work in Waimea. Allows direct contact between observers and staff. Visiting Scientist’s Quarters (VSQ) located in same complex. 9/17/2018

16 Limitations of Remote Observing from Keck HQ in Waimea
Most Keck observers live on the mainland. Mainland observers fly > 3,200 km to get to Waimea Collective direct travel costs exceed $400,000 U.S. / year 9/17/2018

17 Remote Observing from Waimea is not cost effective for short runs
Round trip travel time is 2 days Travel costs > $1,000 U.S. per observer About 50% of runs are for 1 night or less Cost / run is very high for such short runs Such costs limit student participation 9/17/2018

18 Motivations for Remote Observing from the U.S. Mainland
Travel time and costs greatly reduced Travel restrictions accommodated Sinus infections and ruptured ear drums Late stages of pregnancy Increased options for: Student participation in observing runs Large observing teams with small budgets Capability for remote engineering support 9/17/2018

19 Mainland remote observing goals and implementation strategy
Target mainland facility to short duration runs Avoid duplicating expensive Waimea resources Avoid overloading Waimea support staff Strategy: No mainland dormitories; observers sleep at home Access existing Waimea support staff remotely Restrict mainland facility to experienced observers Restrict to mature, fully-debugged instruments 9/17/2018

20 Mainland remote observing facility is an extension of Keck HQ facility
Only modest hardware investment needed: Workstations for mainland remote observers Network-based videoconferencing system Routers and firewalls Backup power (UPS) – especially in California!!! Backup network path to Mauna Kea and Waimea Avoids expensive duplication of resources Share existing resources wherever possible Internet-2 link to the mainland Keck support staff and operational software 9/17/2018

21 Keck software is accessed the same regardless of observer’s location
The control computers at the summit: Each telescope and instrument has its own computer All operational software runs only on these computers All observing data written to directly-attached disks Users access data disks remotely via NFS or ssh/scp The display workstations Telescopes and instruments controlled via X GUIs All users access these X GUIs via remote X displays X Client software runs on summit control computers Displays to X server on remote display workstation 9/17/2018

22 Overall Topology Type your question here, and then click Search.
9/17/2018

23 View Empty 9/17/2018

24 Santa Cruz Remote Observing Facility
9/17/2018

25 Santa Cruz Remote Observing Video Conferencing
9/17/2018

26 The Weather in Waimea 9/17/2018

27 Why did we choose this approach?
Operational Simplicity Operational control software runs only at the summit All users run identical software on same computer Simplifies management between independent sites Allowed us to focus on commonality Different sites / teams developed instrument software Large variety of languages and protocols were used BUT: all instruments used X-based GUIs 9/17/2018

28 Remote observing differences: Waimea versus the mainland
System Management: Keck summit and HQ share a common domain Mainland sites are autonomous Remote File Access: Observers at Keck HQ access summit data via NFS Observers on mainland access data via ssh/scp Propagation Delays: Summit to Waimea round trip time is about 1 ms. Summit to mainland round trip time is about 100 ms. 9/17/2018

29 Increased propagation delay to mainland presents challenges
Initial painting of windows is much slower But once created, window updates fast enough All Keck applications display to Waimea OK A few applications display too slowly to mainland System and application tuning very important TCP window-size parameter (Web100 Initiative) X server memory and backing store Minimize operations requiring round trip transactions 9/17/2018

30 Simulating long propagations delays in the lab
Instruments are designed and built on mainland Software is debugged on local area network Testing on LAN does not reveal delay problems Must measure delay effects before deployment Simulate WAN delays using NIST simulator Requires Linux PC with dual Ethernet interfaces Can select specific packets delays, losses, jitters 9/17/2018

31 Shared access and control of instruments
Most software for Keck optical instruments provides native multi-user/multi-site control All users have consistent view of status and data Instrument control can be shared between sites Multipoint video conferencing key to coordination Some single-user applications can be shared via X-based application sharing environments: XMX VNC 9/17/2018

32 Tradeoffs from this approach to remote observing
Disadvantages: X protocol does not make optimal use of bandwidth Long propagation delays require considerable tuning Advantages: Minimizes staffing requirements at mainland sites Only “vanilla” hardware and software needed there Simplifies sparing and swapping of equipment Simplifies system maintenance at mainland sites Simplifies authentication/access control 9/17/2018

33 Fast and reliable network needed for mainland remote observing
1997: 1.5 Mbps Hawaii -> Oahu -> mainland 1998: 10 Mbps from Oahu to mainland 1999: First phase of Internet-2 upgrades: 45 Mbps commodity link Oahu -> mainland 45 Mbps Internet-2 link Oahu -> mainland 2000: Second phase upgrade: 35 Mbps Internet-2 link from Hawaii -> Oahu Now 35 Mbps peak from Mauna Kea to mainland 2002: 155 Mbs from Oahu to mainland 9/17/2018

34 End-to-end reliability is critical to successful remote operation
Keck Telescope time is valued at $1 per second Observers won’t use facility if not reliable Each observer gets only a few nights each year What happens if network link to mainland fails? Path from Mauna Kea to mainland is long & complex At least 14 hops crossing 6 different network domains While outages are rare, consequences are severe Even brief outages cause session collapse & panic Observing time loss can extend beyond outage The real question is, what happens if the link to the mainland fails? 9/17/2018

35 Keck Observatory policy on mainland remote observing
If no backup data path is available from mainland site, at least one member of observing team must be in Waimea Backup data path must be proven to work before mainland remote observing is permitted without no team members in Waimea 9/17/2018

36 Mitigation plan: install end-to-end ISDN-based fall-back path
Install ISDN lines and routers at: Each mainland remote observing site Keck 1 and Keck 2 control rooms Fail-over and fall-back are rapid and automatic Toll charges incurred only during network outage Lower ISDN bandwidth reduces efficiency, but: Observer retains control of observations Sessions remain connected and restarts avoided Prevents observer panic 9/17/2018

37 Summary of ISDN-based fallback path
Install 3 ISDN lines (6 B channels) at each site Install Cisco 2600-series routers at each end Quad BRI interfaces Inverse multiplexing Caller ID (reject connections from unrecognized callers) Multilink PPP with CHAP authentication Dial-on-demand (bandwidth-on-demand) No manual intervention needed at either end Fail-over occurs automatically within 40 seconds Uses GRE tunnels, static routes, OSPF routing 9/17/2018

38 Running OSPF routing over a GRE tunnel
On each router, we configure 3 mechanisms: A GRE tunnel to the other endpoint A floating static route that routes all traffic to the other endpoint via the ISDN dialer interface A private OSPF domain that runs over the tunnel OSPF maintains its route through the tunnel only if the tunnel is “up” OSPF dynamic routes take precedence over floating static route 9/17/2018

39 Fail-over to ISDN backup data path
If the Internet-2 path is “up”, OSPF “hello” packets flow across the tunnel between routers As long as “hello” packets flow, OSPF maintains the dynamic route, so traffic flows through tunnel If Internet-2 path is “down”, OSPF “hello” packets stop flowing, and OSPF deletes dynamic route With dynamic route gone, floating static route is enabled, so traffic flows through ISDN lines 9/17/2018

40 Fall-back to the normal Internet-2 path
OSPF keeps trying to send “hello” packets through the tunnel, even with Internet is down As long as Internet-2 path remains down the “hello” packets can’t get through Once the Internet-2 path is restored, “hello” packets flow between routers OSPF re-instates dynamic route through tunnel All current traffic gets routed through the tunnel All ISDN calls are terminated 9/17/2018

41 Operational costs of ISDN backup data path
Fixed leased cost is $70 per line per month Three lines at each site -> $2,500 per site/year Both sites -> $5,000/year Long distance cost (incurrent only when active) $0.07 per B-channel per minute If all 3 lines in use: $0.42 per minute $25.20 per hour 9/17/2018

42 Recent operational experience
Remote observing science from Santa Cruz: Low Resolution Imaging Spectrograph (LRIS) Echellete Spectrograph and Imager (ESI) Remote engineering and instrument support ESI High Resolution Echelle Spectrometer (HIRES) Remote Commissioning Support DEIMOS (see SPIE paper & ) 9/17/2018

43 Unplanned use of the facility during week of Sept. 11, 2001
All U.S. commercial air traffic grounded Caltech astronomers have a 5-day LRIS run on Keck-I Telescope starting September 13 No flights available Caltech team leaves Pasadena morning of 9/13 Drives to Santa Cruz, arriving late afternoon Online with LRIS well before sunset in Hawaii 9/17/2018

44 The hardest problem was the lodging!
LRIS operated from Santa Cruz all 5 nights ISDN backup path activated several times Observing efficiency comparable to Waimea Lodging was the biggest problem Motel check-in/check-out times incompatible Required booking two motels for the same night Motels are not a quiet place for daytime sleep 9/17/2018

45 Extending mainland remote observing to other sites
Other sites motivated by Santa Cruz success Caltech remote facility is nearly operational Equipment acquired ISDN lines and router installed Will be operational once routers are configured U.C. San Diego facility being assembled Equipment specified and orders in progress Other U.C. campuses considering plans 9/17/2018

46 Administrative challenges: scheduling shared facilities
Currently only one ISDN router at Mauna Kea Limits mainland operation to one site per night Interim administrative solution Longer term solution may require: Installation of additional ISDN lines at Mauna Kea Installation of an additional router at Mauna Kea 9/17/2018

47 Remaining challenges TCP/IP tuning of end-point machines
Needed to achieve optimal performance Conflicts with using “off-the-shelf” workstations Conflict between optimal TCP/IP parameters for the normal Internet-2 path .vs. the ISDN fall-back path Hoping for vendor-supplied auto-tuning Following research efforts of Web100 Project Administrative challenges Mainland sites are currently autonomous Need to develop coordination with Keck 9/17/2018

48 Summary Model is being extended to multiple sites
Internet-2 makes mainland operation feasible Backup data path protects against interruptions Keck HQ is the central hub for remote operation Mainland remote observing model is affordable: Mainland sites operate as satellites of Keck HQ Leverage investment in existing facilities and staff Leverage investment in existing software Share existing resources wherever feasible Avoid expensive and inefficient travel for short runs Model is being extended to multiple sites 9/17/2018

49 Acknowledgments U.S. National Science Foundation
U.S. Department of Defense University of Hawaii Gemini Telescope Consortium University Corp. for Advanced Internet Development (UCAID) Corporation for Education Network Initiatives in California (CENIC) 9/17/2018

50 Author Information Robert Kibrick, UCO/Lick Observatory
University of California, Santa Cruz California 95064, U.S.A. WWW: Phone: FAX: 9/17/2018


Download ppt "Robert Kibrick, Brian Hayes, Steve Allen"

Similar presentations


Ads by Google