Rocio Guerra European Space Astronomy Centre 1 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 ESAC and the Gaia Catalogue

Slides:



Advertisements
Similar presentations
During spacecraft-level environmental testing, after all instruments were integrated, the government assembled a data clerk team to support test data collection.
Advertisements

GENIUS kick-off - November 2013 GENIUS kick-off meeting The Gaia context: DPAC & CU9 X. Luri.
Questions from the European colleagues & Answers (only) from the Chinese engineers present for Y.-W. Zhang and S.-G. Yuan Chinese Academy of Space Technology.
PHEOS PSRR Concept of Operations. 2 Scope The purpose of the PCW-UVI Concept of Operations (ConOps) is to communicate how mission systems will operate,
GRID Activities at ESAC Science Archives and Computer Engineering Unit Science Operations Department ESA/ESAC – Madrid, Spain.
Folie 1 Service Oriented Architecture - Prototyping study - DLR/GSOC Author: S.Gully.
Network Management Overview IACT 918 July 2004 Gene Awyzio SITACS University of Wollongong.
23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.
Mission Operations and Data Flow Overview Koji Mukai Astro-E2 Guest Observer Facility.
GLAST LAT ProjectISOC Peer Review - March 2, 2004 Document: LAT-PR Section 2.1 Requirements 1 Gamma-ray Large Area Space Telescope GLAST Large.
Section 15-1GLAST Ground System Design Review August 18&19, 2004 ISOC Organization ISOC Manager R Cameron Commanding, H&S Timeline Planning Command Generation.
GLAST LAT ProjectISOC CDR, 4 August 2004 Document: LAT-PR-04500Section 4.11 GLAST Large Area Telescope: Instrument Science Operations Center CDR Section.
30 March 2006Birmingham workshop1 The Gaia Mission A stereoscopic census of our Galaxy.
Astronomical GRID Applications at ESAC Science Archives and Computer Engineering Unit Science Operations Department ESA/ESAC.
Data Processing and User Software Ken Ebisawa (Astro-E2 GOF) presentation and demonstration.
Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
ELISA Data Processing Centre Volker Beckmann APC, Francois Arago Centre A. Petiteau, E. Porter, G. Auger, E. Plagnol, P. Binétruy.
The Gaia mission Data reduction activities in the UK Floor van Leeuwen, IoA.
GAUDI Ground-based Asteroseismology Uniform Database Interface E. Solano Bases de données en spectroscopie stellaire. Paris.
Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.
Gaia, next frontier in Astronomy Jose Hernandez Gaia Data and Calibration Engineer European Space Astronomy Centre (ESAC) Madrid, Spain.
Data processing group. General study of data processing architecture: - overall definition of the data processing functions - share of tasks between on-board.
ITEC224 Database Programming
MASSACHUSETTS INSTITUTE OF TECHNOLOGY NASA GODDARD SPACE FLIGHT CENTER ORBITAL SCIENCES CORPORATION NASA AMES RESEARCH CENTER SPACE TELESCOPE SCIENCE INSTITUTE.
SGS: Activities and Requests Helen Middleton. Hermean Environment Working Group Meeting | SGS Team | Key Largo | 16/05/2013 | Slide 2 Contents 1.ESAC.
Data Management Subsystem Jeff Valenti (STScI). DMS Context PRDS - Project Reference Database PPS - Proposal and Planning OSS - Operations Scripts FOS.
CAPACITY Operational Atmospheric Chemistry Monitoring Missions CAPACITY Final Meeting - WP Ground Segment synthesis Final Meeting ESTEC02/06/05.
How to Adapt existing Archives to VO: the ISO and XMM-Newton cases Research and Scientific Support Department Science Operations.
Astronomical Spectroscopy and the Virtual Observatory ESAC, March 2007 VO tools and cross-calibration Pedro García-Lario European Space Astronomy.
Data Management Challenges in Gaia
All rights reserved © Altec ExoMars 2018 Rover Operations Control Centre Planned Organization of ROCC Operations I. Musso.
Planetary Science Archive PSA User Group Meeting #1 PSA UG #1  July 2 - 3, 2013  ESAC PSA Archiving Standards.
ASI-Eumetsat Meeting Matera, 4-5 Feb CNM Context Matera, February 4-5, 20092ASI-Eumetsat Meeting.
Consortium Meeting La Palma October PV-Phase & Calibration Plans Sarah Leeks 1 SPIRE Consortium Meeting La Palma, Oct. 1 – PV Phase and.
LSST: Preparing for the Data Avalanche through Partitioning, Parallelization, and Provenance Kirk Borne (Perot Systems Corporation / NASA GSFC and George.
MWA Operations Management Plan (OMP, v. 3) Ron Remillard (MIT); MWA Project Meeting, June 6, 2011.
Developing software and hardware in parallel Vladimir Rubanov ISP RAS.
RVS Calibration Workshop, Paris RVS RVS Calibration RVS Calibration & First Look Workshop, Paris Mark Cropper.
Research Networks and Astronomy Richard Schilizzi Joint Institute for VLBI in Europe
PLATO Data Center: Purpose and Structure Laurent Gizon (PDPM) Hamed Moradi (PDC Project Office)
1 Computing Challenges for the Square Kilometre Array Mathai Joseph & Harrick Vin Tata Research Development & Design Centre Pune, India CHEP Mumbai 16.
Space Operations as a Guide for a Real-World Scheduling Competition Eduardo Romero Marcelo Oglietti
Swift HUG April Swift data archive Lorella Angelini HEASARC.
Data Management at Gaia Data Processing Centers GREAT Workshop on Astrostatistics and Data Mining in Astrnomical Databases La Palma, Spain May 30 - June.
All rights reserved © Altec ExoMars 2018 Rover Operations Control Centre Science instruments data pipeline G. Martucci.
Science ESAC Cluster Final Archive Pedro Osuna Head of the Science Archives and VO Team Science Operations Department CAA-CFA Review Meeting.
AstroGrid NAM 2001 Andy Lawrence Cambridge NAM 2001 Andy Lawrence Cambridge Belfast Cambridge Edinburgh Jodrell Leicester MSSL.
Aquarius Mission Simulation A realistic simulation is essential for mission readiness preparations This requires the ability to produce realistic data,
18 th GIST Meeting 14 th –16 th May 2003 Ground Segment (GGSPS) Report 1 GERB Ground Segment B.C.Stewart RAL.
The Copernicus Sentinel-3 Mission: Update on status Susanne Mecklenburg – ESA Sentinel-3 Mission Manager Sentinel-3 ESA development & operations teams.
1. 2 Purpose of This Presentation ◆ To explain how spacecraft can be virtualized by using a standard modeling method; ◆ To introduce the basic concept.
Page 1 PACS GRITS 17 June 2011 Herschel Data Analysis Guerilla Style: Keeping flexibility in a system with long development cycles Bernhard Schulz NASA.
GENIUS Kick-Off meeting December 4th 2013 WP-620 Simulated catalogue data Francesc Julbe UB – IEEC, Barcelona.
E. Solano. GAIA Meeting, Menorca, Oct 2009 GAIA and the Virtual Observatory Enrique Solano, LAEX/CAB (INTA-CSIC) Spanish VO Principal Investigator.
Requirements to WSO Ground Segment Lavochkin Association & ROSCOSMOS 30/06/2006.
SwCDR (Peer) Review 1 UCB MAVEN Particles and Fields Flight Software Critical Design Review Peter R. Harvey.
E-USOC operations in Increments José Miguel Ezquerro Navarro E-USOC/UPM Jul 2010POIWG#28 - Huntsville, AL.
Scenario use cases Szymon Mueller PSNC. Agenda 1.General description of experiment use case. 2.Detailed description of use cases: 1.Preparation for observation.
Statistical process model Workshop in Ukraine October 2015 Karin Blix Quality coordinator
WP18, High-speed data recording Krzysztof Wrona, European XFEL
Detlef Koschny Research and Scientific Support Department ESA/ESTEC
JWST Pipeline Overview
Implementation Plan Ronald Drimmel (Project Manager)
Tools of Software Development
Databases, Web Pages and Archives
Observing with Modern Observatories (the data flow)
Monitoring the INTEGRAL satellite
GLAST Large Area Telescope Instrument Science Operations Center
Reportnet 3.0 Database Feasibility Study – Approach
Presentation transcript:

Rocio Guerra European Space Astronomy Centre 1 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 ESAC and the Gaia Catalogue Rocio Guerra Gaia SOC Test Manager ESAC/ESA Many photos and slides from William O’Mullane SOC development Manager

Rocio Guerra European Space Astronomy Centre 2 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Launch spring 2012 from Kourou … but it is a long way from here

Rocio Guerra European Space Astronomy Centre 3 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Gaia Ground Segment Mission Operations Ground Segment –Operated by ESA –Responsible for the satellite operations –Composed of the MOC and the Ground Stations and communication networks Data Processing Ground Segment –Jointly operated by ESA and DPAC –Responsible for payload monitoring and the scientific data reduction and analysis –Comprises of the SOC and a set of the DPCs

Rocio Guerra European Space Astronomy Centre 4 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Ground Stations Antennas During the LEOP and during critical mission phases, the 15-m Kourou will be used. Maspalomas and Perth as backups During transfer and commissioning New Norcia will be used for contact with s/c. Kourou will be backup The 35m Cebreros (and 35m New Norcia) will be during all other mission phases.

Rocio Guerra European Space Astronomy Centre 5 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 MOC team Preparation and execution of all operational tasks in order to achieve the optimum performance of the s/c in orbit The Science Operations Centre in ESAC is responsible of: The Mission Operations Centre in ESOC (Darmstadt) is responsible for: Photo: David Milligan SCOM Spacecraft Operations Manager David Milligan

Rocio Guerra European Space Astronomy Centre 6 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 MOC Responsibilities. This includes: S/C and payload activities Orbit determination and control Attitude determination and control On-board software maintenance Making available the raw satellite data to the SOC

Rocio Guerra European Space Astronomy Centre 7 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC – the beginning 2006 instrument commanding (mode changes, calibration uploads, etc) Initial Data Treatment Getting data to DPAC (Science Data Processing) The Science Operations Centre in ESAC (Madrid) is responsible for:

Rocio Guerra European Space Astronomy Centre 8 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC team

Rocio Guerra European Space Astronomy Centre 9 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC team – 2019????

Rocio Guerra European Space Astronomy Centre 10 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC Responsibilities Part of ESA Ground Segment with MOC Development led by ESA Project Interface and contact point between MOC and DPAC Both an ESA SOC and DPAC Data Processing Centre Operate ‘MOC-facing’ tasks (MOC Interface Task and Payload Operations System) Operate IDT, First Look, AGIS, Main Database – DPAC-facing tasks Archive science data collected during the mission Disseminate sets of the database to the DPCs and re- integrate the results

Rocio Guerra European Space Astronomy Centre 11 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 DPAC The Science Operations Centre in ESAC is responsible of: The Gaia Data Processing is done by the Data Processing and Analysis Consortium (DPAC) That is a community effort (>300 people) The SOC will operate within DPAC and will rely heavily on its expertise in many areas The Project Scientist oversees all this (advised by GST) CU1: System Architecture CU2: Data Simulations CU3: Core Processing CU4: Object Processing CU5: Photometric Processing CU6: Spectroscopic Processing CU7: Variability Processing CU8: Astrophysical Parameters CU9: Catalogue Access

Rocio Guerra European Space Astronomy Centre 12 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Architecture The Science Operations Centre in ESAC is responsible of: Highly distributed Multiple independent DPCs and CUs Want/need decouple Reduce dependencies risk Hub and spokes Max flexibility for CUs and DPCs Minimum ICDs

Rocio Guerra European Space Astronomy Centre 13 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Daily Processing at SOC IDT (Initial Data Treatment) and FL (First Look) form together with MIT (MOC Interface Task) the daily processing pipeline which will run in the SOC IDT and FL are both being developed in CU3 Both IDT and FL are Schedule Critical

Rocio Guerra European Space Astronomy Centre 14 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 MIT MOC Interface Task Part of the daily processing pipeline (with IDT and FL) Schedule Critical Receives raw telemetry from the MOC (star packets, ASD data, HK telemetry) in file format and writes that data to the IDT/FL database for downlink software (IDT, FL) to pick up Reconstructs ’bad packets’ where possible Sends to MOC data from the POS

Rocio Guerra European Space Astronomy Centre 15 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 POS Payload Operations System Monitors spacecraft attitude data against model attitude and attitude derived from ground processing, for deviations. Generates (for the MOC) Scanning Law (model attitude) data Science Event Files and Avoidance periods Payload Operations Requests Optical data (observations of Gaia) Generates (for DPAC) Exposure Maps

Rocio Guerra European Space Astronomy Centre 16 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Interface with the MOC

Rocio Guerra European Space Astronomy Centre 17 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 IDT Purpose of Initial Data Treatment: Generate refined attitude data Compute basic image parameters from telemetry Match observations to known sources (cross matching)‏ Convert telemetry in raw observations IDT is being developed by UB in Barcelona IDT is run every cycle on large simulated data sets at ESAC. The results are used to test several other systems in DPAC.

Rocio Guerra European Space Astronomy Centre 18 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 FL Purpose of First Look: Check the instrument health on a daily basis The ‏FL system consist of: One day calibrations (ODAS, PODC, LODC, CODC, RODC)‏ Detailed FL (DFLM, DFLE) ‏ FL is being developed by ARI in Heidelberg with contributions from other institutes

Rocio Guerra European Space Astronomy Centre 19 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Further Daily processing After IDT and FL run data is sent to all DPCs Cambridge will run part of the CU5 photometric pipeline to look for interesting objects –Science Alerts may be issued. CNES will run a more detailed RVS calibration and First Look and provide feedback to SOC. CNES will also run CU4 software to detect interesting SSOs Torino will perform Basic Angle Monitor(BAM) data processing

Rocio Guerra European Space Astronomy Centre 20 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Data Reduction.. HUGE task Gaia data reduction is iterative Starts with a release and distribution of a version of the Gaia Main Database to the DPCs by SOC MDB release fed through the processing systems at each DPC New version of the Main Database assembled at SOC from the results of the data processing New data received during a data reduction cycle is included in next release of the MDB

Rocio Guerra European Space Astronomy Centre 21 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 The Science Operations Centre in ESAC is responsible of: Version each 6 Months ~20 TB inc. each version Hence ~200TB in final GTS pushes data to other DPCs Governed by ICD MainDB – the data hub

Rocio Guerra European Space Astronomy Centre 22 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 On-going activities Integration and testing of software to run at SOC Processing of simulation data to provide input data sets for CU tests Planning, design and implementation of the DPAC Integration Tests, to culminate in the 1+ year End-to-End test Leading role in defining operations Preparation of plans and procedure for running software, databases, transfer systems Configuration control and QA activities

Rocio Guerra European Space Astronomy Centre 23 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Gaia Catalogue Part of DPAC: CU9 Reporting, etc. similar to others CUs Some DPAC members will have to be in CU9 (no one understands the data and algorithms better) No real possibility to have initial catalogue before months after launch -> CU9 after launch

Rocio Guerra European Space Astronomy Centre 24 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 The ARCHIVE is not simply MDB MDB is CU oriented and contains redundancy Archive more cohesive and present single coherent dataset Mode of access is radically different: Arbitrary queries, data mining, download in various formats, etc Replicas ESA must have a copy Any institute involved should be allowed a replica is serious about it Portability is therefore an essential requirement Archive in context

Rocio Guerra European Space Astronomy Centre 25 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Archive in context CU9 may decide on another decomposition

Rocio Guerra European Space Astronomy Centre 26 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Interface / Ingestor COMMUNITY INTERFACE Should be Public Oriented Basics with no login Easy self-registration for advanced features Some sort of Sky-Browser (ala SDSS, GoogleSky) Multi-lingual of course INGESTOR MDB ICD already exists (contains all data) Lots of data Probably needs to be distributed Should not take too long – one week

Rocio Guerra European Space Astronomy Centre 27 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Interrogator INTERROGATOR Need fast engine to answer queries DBMS tuning is needed Should allow powerful queries to user (SQL and/or ADQL) Local space for registered users to upload data and store query results Extraction in multiple formats (csv, FITS, VOtable) VO access of course … TAP, SIAP … And must be … FAST!!!

Rocio Guerra European Space Astronomy Centre 28 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Advanced Applications ADVANCED APPLICATIONS They are “added value” applications 3D visualization for all for part of sky (e.g. globular cluster) Light curve tools … Interrogator should provide an API to build these apps on

Rocio Guerra European Space Astronomy Centre 29 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Science Alerts / Help Desk SCIENCE ALERTS Bit of a grey area for now Flux based alerts straight from CU5 NEO orbits from CU4 go direct to IMCCE Others will come up – processing first!!!! CU9 will provide a facility for DPAC to publish alerts However alerts sent should be archived by CU9 Follow up should be gathered HELP DESK Usual problem reporting, tracking, FAQ, etc. Some real people somewhere … Should be ready to politely answer all questions

Rocio Guerra European Space Astronomy Centre 30 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Documentation DOCUMENTATION Uniform documentation on all data MDB dictionary provides a lot on the data Algorithm doc from CUs Bunch of statistics (e.g. maps source, observation, etc) Derived plots such us HR, galactic-kinematics, Hess diagram,… Printed volume would be nice! At least Source Catalogue on media… Probably most important CU9 task!!!! Probably most important CU9 task!!!!

Rocio Guerra European Space Astronomy Centre 31 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Releases very tentative … Yes more than one – we wont make you wait till But all remains tentative 1. Probably one after 2-3 years Some Astrometry and some Photometry 2. One more after 5 years Full astrometry better photometry, spectra 3. Then the final one 8 years after launch Variable stars, astroparams …

Rocio Guerra European Space Astronomy Centre 32 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Open areas Living archive (Anthony Brown) Can we/should we allow additions to the Archive? e.g. improved solutions for binaries using follow observations Implications for maintenance, quality and security Cross matching other catalogues …

Rocio Guerra European Space Astronomy Centre 33 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Questions ?? Ariane V188 carrying Herschel and Planck (May )