Presentation is loading. Please wait.

Presentation is loading. Please wait.

NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives.

Published byCatherine McCormick Modified over 9 years ago

Similar presentations

Presentation on theme: "NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives."— Presentation transcript:

1

2 NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives

3 NEON School NEON School 15 July 2004 Layout  What’s the role of the Data Provider  Three levels of Archive complexity –Raw Data (ESO Archive) –Reduced Products (The importance of Service Observing) –Enhanced Products (HST/WFPC2 Associations)  Enhanced Interfaces (Querator & VO)  Publications and Observing Programs

4 NEON School Operations Technical Support The ESO Data Flow Model and its Supporting Organizational Structure Data Archive & Supporting Databases storage and re- distribution PI packages distribution Phase I proposal submission program selection time allocation Phase II proposal submission observing blocks medium term schedule Observation short term schedule archival of data Quality Control Master Cal files prod. Instrument trending PI products preparation User Support Group Paranal/La Silla Observatories Data Flow Operations Group Data Flow, what you were told

5 NEON School Data Flow, the joyless Truth

6 NEON School Space-based observatories: The archive is a core activity from the start. Ground-based observatories: Priority given to Observers, not to Community With the difference that Observers can go observer themselves. ESO has changed such paradigm, by introducing Service Mode. But it takes time for the community to accept it, and effort within the Observatory to implement it The Archive: last in the chain?

7 NEON School http://www.eso.org/org/dmd/usg/philosched.html Service Mode

8 NEON School Real Archive vs Virtual Observatory Real Archives Three types of Archives –Level 0: Data Repository –Level 1: Ability to offer reduced products –Level 2: Ability to offer enhanced products 1 Instrumental signature-free data 2 Enhanced Products (e.g., mosaics) 0 Data Repository “Real” Archives Boring Handy

9 NEON School Level 0: Data Repository Data Repository: 1. Acquisition 2. Data Management 3. Metadata 4. User Interface

10 NEON School Level 0: Data Repository (1) Acquisition: · Observing run · The Control S/W acquires the Data · Stored in the on-line data archive · Transmitted to Science Archive

11 NEON School Level 0: Data Repository (2a) Data management: Integrity checks · Did I get all I should have gotten? · Check for corrupted data · Take action if corrupted

12 NEON School Level 0: Data Repository (2b) Data management: Storing on medium of choice Optical disks, CDs, DVDs, Magnetic Disks Bookkeeping Which file was stored where? When? In which format? Which compression? Store all this info in a Database (RDBMS)

13 NEON School Level 0: Data Repository Data Repository: 1. Acquisition 2. Data Management 3. Metadata 4. User Interface

14 NEON School Level 0: Data Repository ( 3 ) Metadata: · Where was the telescope pointing? (RA, Dec) · Which telescope/instrument was used? (telescope, instrument) · When did the observation occur? (start time, end time, exptime) · In which band? (filter or dispersive element used) · By whom? (Principal Investigator [PI]) · For which reason? (Observing Program Identifier) · Plus many other details about the observing mode, etc. · All this information is usually extracted from the Header of the FITS files containing the Data. · Metadata are typically stored in a Database

15 NEON School Level 0: Data Repository Data Repository: 1. Acquisition 2. Data Management 3. Metadata 4. User Interface

16 NEON School Level 0: Data Repository ( 4a ) User Interface Finally, an astronomer wants to see whether some information about photons regarding a certain celestial object are already stored in our archive. Metadata (Database) -> Internet (Web)

17 NEON School Level 0: Data Repository (4 Archive) User Interface

18 NEON School Level 0: Data Repository (4 Form) User Interface Finally, an astronomer wants to see whether some information about photons regarding a certain celestial object are already stored in our archive. Metadata (Database) -> Internet (Web)

19 NEON School Level 0: Data Repository ( 4b ) User Interface Name Resolvers: SIMBAD (CDS), NED (IPAC) Search by Box/Cone (RA, Dec, size/radius) Search by Night (Observing run, calibration obs.) Observing Blocks, Data Product Types: very ESO specific Exptime, Filter, Grism, Grating, Slit: very instrument specific Release Date: Are the data public? Proprietary period implements Observatory policy

20 NEON School Level 0: Data Repository ( 4 Query ) User Interface Finally, an astronomer wants to see whether some information about photons regarding a certain celestial object are already stored in our archive. Metadata (Database) -> Internet (Web)

21 NEON School Level 0: Data Repository ( 4 Results ) User Interface Finally, an astronomer wants to see whether some information about photons regarding a certain celestial object are already stored in our archive. Metadata (Database) -> Internet (Web)

22 NEON School Level 0: Data Repository ( 4 Submission ) User Interface Finally, an astronomer wants to see whether some information about photons regarding a certain celestial object are already stored in our archive. Metadata (Database) -> Internet (Web)

23 NEON School Level 0: Data Repository ( 5 ) Request Handler A system which takes good care of the submitted archive requests. The RH consists of a set of procedures, interfaces, C programs and libraries, and relational tables; It allows to: submit archive requests, keep track of their handling, inform (via email or via a web interface) the operators of any intervening problem, notify the archive users about the status of their requests up to successful closure.

24 NEON School Level 0: Users’ point of view A Level 0 archive only offers RAW Data. Removing the instrumental signature from the RAW files is left to the astronomer: Identify suitable BIAS, DARK and FLAT frames Produce Master frames (e.g., mean averaging, etc.) Run reduction software (e.g., bias&dark, flat fielding) Couldn’t the Observatory provide that service?

25 NEON School Level 1: Ability to offer reduced products (1) Data Repository & Processing: 1. Actual Storage 2. Data Management 3. Metadata 4. User Interface 5. Calibration DB + Rules 6. Reduction Pipeline

26 NEON School Level 1: Ability to offer reduced products (2) Calibration DB + Rules · Instrument Scientists (ESO: Quality Control group [QC]) build the “Master Calibration Files” (or “Reference Files” in HST terminology) · Reference files are stored in the Archive · They associate to each observation a set of reference files via a set of rules. · Rules are applied and results stored in the Calibration Database

27 NEON School Level 1: Ability to offer reduced products (3) Calibration DB + Rules Simple-minded example · Rules and recipe to build a master DARK file: · Retrieve 5 DARK images · the ones closer to the date of the observation, · With the same read-out mode and exposure time; · De-bias each DARK image; · Average them while removing cosmic rays; · Normalise to “dark current per second”; · Flag dubious pixels (Data Quality flag)

28 NEON School Level 1: Ability to offer reduced products (4) Reduction Software · Instrument Scientists and Software Group must implement the Reduction Software: a collection of processes necessary to remove the instrumental signature from a given observation. If such software can run unattended, we call it Reduction Pipeline.

29 NEON School Level 1: Ability to offer reduced products (5) Reduction Products · Once the Calibration DB and the Reduction Pipeline are in place, the Science Archive Request Handler can make use of it to offer Reduced Products to the relieved community.

30 NEON School Level 1: HST reduced products (form) HST

31 NEON School Level 1: HST reduced products (results) HST

32 NEON School Preview PREVIEW A highly compressed view of an observation, to provide a quick-look product for quality assessment. It is generated via a specialised pipeline takes in input the reduced (or otherwise raw) observation and processes it (resizing, cropping, reducing dynamical range, applying wavelets, etc.). Look at it, do not use it for your science!

33 NEON School Level 1: HST reduced products (preview) HST

34 NEON School Level 1: HST reduced products (submission) HST

35 NEON School Level 2: Enhanced Products & Interfaces How to improve the archive beyond the on-the-fly recalibration? 1. Deeper and cleaner products 2. Better User Interfaces

36 NEON School Level 2: Deeper and Cleaner: Associations HST/WFPC2 (Wide Field Planetary Camera 2) An observation in space is affected by Cosmic Rays No detector is perfect: bad columns, hot pixels, etc. Special observing strategy to cope with these facts: CR-SPLIT: Splitting one single long exposure into many short ones; POS-TARG: Dithering is the technique of displacing the telescope between observations either on integral pixel scales (to assist in removing chip blemishes such as hot pixels) or on sub-pixel scales (to improve sampling and thus produce a higher-quality final image).

37 NEON School Level 2: Deeper and Cleaner: Associations HST/WFPC2 (Wide Field Planetary Camera 2) CR-SPLIT and POS-TARG have to do with the Data Flow System of the HST observatory, not with the scientific exploitation of the data. The associations, i.e., grouping together those observations which cover the same sky region, provide a way to isolate the archive user from those technical details.

38 NEON School Level 2: WFPC2 Associations (form)

39 NEON School Level 2: WFPC2 Associations (asn)

40 NEON School Level 2: WFPC2 Associations (members)

41 NEON School Level 2: WFPC2 Associations (comparison)

42 NEON School Level 2: Advanced Interfaces: Querator  Astronomers requires Scientific interfaces  They don’t want to know Observatory details  Hence, not only instrumental signature but also Data Flow signature must be removed Querator goes in this direction. Querator in two words: A multi-archive search engine specialised on multi-colour imaging data.

43 NEON School Querator (form)

44 NEON School Querator (results)

45 NEON School Querator: (Filters)  Filters

46 NEON School Last but not list: Publications

47 NEON School Last but not list: Publications

48 NEON School Last but not list: Publications

49 NEON School Last but not list: Publications

50 NEON School Astrophysical Virtual Observatory

51 NEON School Recap You have seen the exciting day to day life of a Data Provider. URLs: Service mode: Archive Home Page: http://archive.eso.org/ Querator: http://archive.eso.org/querator/ Recap You have seen the exciting day to day life of a Data Provider. URLs: Service mode: http://www.eso.org/org/dmd/usg/philosched.html Archive Home Page: http://archive.eso.org/ Querator: http://archive.eso.org/querator/ http://archive.eso.org/ http://www.eso.org/org/dmd/usg/philosched.html http://archive.eso.org/

Download ppt "NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives."

Similar presentations

A.Micol IVOA Registry REGISTRY WG 19-20 Mar 2003 A Science Case (and 1000 Questions) for the IVOA Registry.

A.Micol IVOA Registry REGISTRY WG Mar 2003 A Science Case (and 1000 Questions) for the IVOA Registry.
MAST-VizieR/NED cross correlation tutorial 1. Introduction Figure 1: Screenshot of the MAST VizieR Catalog Search Form. or enter here as object class:

MAST-VizieR/NED cross correlation tutorial 1. Introduction Figure 1: Screenshot of the MAST VizieR Catalog Search Form. or enter here as object class:
Query Evaluation. An SQL query and its RA equiv. Employees (sin INT, ename VARCHAR(20), rating INT, age REAL) Maintenances (sin INT, planeId INT, day.

Query Evaluation. An SQL query and its RA equiv. Employees (sin INT, ename VARCHAR(20), rating INT, age REAL) Maintenances (sin INT, planeId INT, day.
NOAO/Gemini Data workshop – Tucson, 2010-07-20.  Hosted by CADC in Victoria, Canada.  Released September 2004  Gemini North data from May 2000  Gemini.

NOAO/Gemini Data workshop – Tucson,  Hosted by CADC in Victoria, Canada.  Released September 2004  Gemini North data from May 2000  Gemini.
HAWCPol / SuperHAWC Software & Operations J. Dotson July 28, 2007.

HAWCPol / SuperHAWC Software & Operations J. Dotson July 28, 2007.
Introduction to Spitzer and some applications Data products Pipelines Preliminary work K. Nilsson, J.M. Castro Cerón, J.P.U. Fynbo, D.J. Watson, J. Hjorth.

Introduction to Spitzer and some applications Data products Pipelines Preliminary work K. Nilsson, J.M. Castro Cerón, J.P.U. Fynbo, D.J. Watson, J. Hjorth.
18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.

18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.
Data provenance in astronomy Bob Mann Wide-Field Astronomy Unit University of Edinburgh

Data provenance in astronomy Bob Mann Wide-Field Astronomy Unit University of Edinburgh
Astronomical GRID Applications at ESAC Science Archives and Computer Engineering Unit Science Operations Department ESA/ESAC.

Astronomical GRID Applications at ESAC Science Archives and Computer Engineering Unit Science Operations Department ESA/ESAC.
Hubble Legacy Archive Lee Quick - TIPS meeting July 19, 2012 Goals Data History Current Work Demo.

Hubble Legacy Archive Lee Quick - TIPS meeting July 19, 2012 Goals Data History Current Work Demo.
Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)

Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)
GAUDI Ground-based Asteroseismology Uniform Database Interface E. Solano Bases de données en spectroscopie stellaire. Paris.

GAUDI Ground-based Asteroseismology Uniform Database Interface E. Solano Bases de données en spectroscopie stellaire. Paris.
HST Archival Studies of HH 30 WFPC2 Data Retrieval and Reduction Liu, Jimmy Chun-Fan Supervisor: Dr. Shang, Hsien August 23, 2002 Summer Student Program.

HST Archival Studies of HH 30 WFPC2 Data Retrieval and Reduction Liu, Jimmy Chun-Fan Supervisor: Dr. Shang, Hsien August 23, 2002 Summer Student Program.
Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.

Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.
Data Management Subsystem: Data Processing, Calibration and Archive Systems for JWST with implications for HST Gretchen Greene & Perry Greenfield.

Data Management Subsystem: Data Processing, Calibration and Archive Systems for JWST with implications for HST Gretchen Greene & Perry Greenfield.
SEMINAR ON :. ORGANISATION Organizations are formal social units devoted to attainment of specific goals. Organizations use certain resources to produce.

SEMINAR ON :. ORGANISATION Organizations are formal social units devoted to attainment of specific goals. Organizations use certain resources to produce.
MASSACHUSETTS INSTITUTE OF TECHNOLOGY NASA GODDARD SPACE FLIGHT CENTER ORBITAL SCIENCES CORPORATION NASA AMES RESEARCH CENTER SPACE TELESCOPE SCIENCE INSTITUTE.

MASSACHUSETTS INSTITUTE OF TECHNOLOGY NASA GODDARD SPACE FLIGHT CENTER ORBITAL SCIENCES CORPORATION NASA AMES RESEARCH CENTER SPACE TELESCOPE SCIENCE INSTITUTE.
50mm Telescope ACS Course Garching, 15 th to 19 th January 2007 January 2007Garching.

50mm Telescope ACS Course Garching, 15 th to 19 th January 2007 January 2007Garching.
STIS Closeout Plan Paul Goudfrooij 2005 HST Calibration Workshop, 10/26/2005.

STIS Closeout Plan Paul Goudfrooij 2005 HST Calibration Workshop, 10/26/2005.
ACCESS for VALIDITY ACCESS for INNOVATION. Starting January 2011 for NEW proposals Not voluntary – “integral part” of proposal and FastLane Required for.

ACCESS for VALIDITY ACCESS for INNOVATION. Starting January 2011 for NEW proposals Not voluntary – “integral part” of proposal and FastLane Required for.

Similar presentations

Ads by Google