Observing with Modern Observatories (the data flow) NEON School 2004 Observing with Modern Observatories (the data flow)

Lecture Layout Proposal Submission Phase I Phase II Observation Observing modes The proposal submission The OPC selection Long term scheduling “Web Letters” Phase II P2PP, OBs, etc. Observation Short-term scheduling Data acquisition

Lecture Layout Archival Connecting meta data to pixels Data shipment (or replication) to Garching Quality Control Preparation of master calibration files Preparation of reduced products for PIs of service mode programs Archive use Data distribution to PIs Data distribution to archive users

The data flow: Another View

The Data Flow User Support Group The ESO Data Flow Model and its Supporting Organizational Structure Paranal/La Silla Observatories Phase II proposal submission observing blocks medium term schedule Operations Technical Support Data Archive & Supporting Databases storage and re-distribution PI packages distribution Phase I proposal submission program selection time allocation Observation short term schedule archival of data Quality Control Master Cal files prod. Instrument trending PI products preparation Data Flow Operations Group

Phase I: Proposal Submission Provide information: Identification of PI/CoIs Facilities to be used Split program in “runs” per facility/group of targets Observing preferences Scientific Justification OPC selection: Peer review of proposal Below or above threshold? Long term scheduling “Web Letters” Did I get time or not? Message sent to all PI with link to explanations program run

Phase I: Observing Modes Two observing modes: “Visitor” mode Keep contact with telescope and instrumentation (observing should not become a black box/magic activity) Need to use a specific observing mode Need to be able to assess quality and react immediately “Service” mode Program is to use normal, supported instrument modes Program requires multiple visits of the targets Program has lots of targets spread in r.a. space

Phase II: Send Details Provide detailed information: About each RUN in the PROGRAM About each target in the proposal About instrument settings to be used Define observing sequence Define exposure times, filters, constraints (e.g., on the seeing) ==> Observing block (OBs) program run OB

Observation Steps in the observation sequence: OBs sent to La Silla/Paranal Short term scheduling to match required target visibility, moon brightness, seeing, … with prevailing conditions OB execution ==> exposures Two types of exposures: calibration and science program run OB exposure

Archival Connecting meta-data to pixels Collecting telescope, environment, instrument and user provided details Building a header with this information according to a pre-defined dictionary. Assembling header and pixels into a FITS file. Sending file to the archive system Storing the file onto the archive medium (DVD, Mag disk) Decomposing header and storing info into database Replicating database to Garching Sending archive media to Garching

Data Quality Control Building calibration database Assembling master flats, darks, biases Sending those to the archive for future Preparing calibrated science products For each run, build a structure with raw, calibration and calibrated data for the PIs of service mode runs. Send a “pack & send” signal to the archive

The Archive Collects data: Information about PIs, programs, runs, OBs, exposures Pixel data (raw science and calibration files) Distributes data: Proprietary data to PIs Public data to anyone

The Data Flow User Support Group The ESO Data Flow Model and its Supporting Organizational Structure Paranal/La Silla Observatories Phase II proposal submission observing blocks medium term schedule Operations Technical Support Data Archive & Supporting Databases storage and re-distribution PI packages distribution Phase I proposal submission program selection time allocation Observation short term schedule archival of data Quality Control Master Cal files prod. Instrument trending PI products preparation Data Flow Operations Group

The Archive: Content Contains ESO and HST raw data Started operations in 1991 Growth never stopped Need to constantly adapt to new storage technology to keep costs low Need to frequently adapt to current software user interfaces to provide users with better search tools (more and more data) Need to look for “advanced” products to empower users with doing science

The Archive: Content

The Archive: Output Data retrieval model is that of the Web Shop: Browse catalog Mark articles of interests Go to “cashier” Select delivery options Confirm submission

Request Submission: 1) Browse

Archive: Selection of Frames of interest

The Archive: Selecting Request Options

The Archive: Output

Technical vs. Science Archive Two types of Archives Technical vs. Science Archive Current system more tuned to serve the operational needs of the Observatory (data collection and distribution to internal/external users) Science users still mostly left out (emphasis on raw data) Discrepancy understood early by the VO “founding fathers” Need to provide access to well-described, instrumental signature-free data Need to provide hassle-free access to different archives without affecting their operation critical mission Science users will get better products and access through the VO

Archives for the people! The three layers Separate the mission critical support role from the delivery of products Leave the science service provision to the VO Provide interfaces to the top layers (e.g., image and spectra access protocols, high-level data processing grids) Virtual Observatory VO tools and facilities Instrumental signature-free data “Real” Observatory Data In/Out

Technical vs. Science Archive Two type of archives Technical vs. Science Archive Archive downloads continue to increase, but usage increasingly technical: Multiple data deliveries to PIs/CoIs Easy retrieval of own data Streamlining of internal data channels Science usage still limited: ESO Archive still has too few instrumental signature-free products to offer Astronomers will produce archive-based science if they are paid and/or supported! (e.g., Hubble’s archive proposals, ASTROVIRTEL)

On-going efforts in the ESO archive News from the “technical” archive: Efforts on-going to put more reduced products available Better web-based search screens under development Data available world-wide probably in 2005 (ESO Council to decide) News from the “VO” side @ ESO: On-going efforts to provide a multi-archive query system (Querator) Image and spectral VO access implemented as soon as standards approved Extra ESO resources arriving soon to help with all of the above