Rocio Guerra European Space Astronomy Centre 1 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 ESAC and the Gaia Catalogue Rocio Guerra Gaia SOC Test Manager ESAC/ESA Many photos and slides from William O’Mullane SOC development Manager

Rocio Guerra European Space Astronomy Centre 2 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Launch spring 2012 from Kourou … but it is a long way from here

Rocio Guerra European Space Astronomy Centre 3 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Gaia Ground Segment Mission Operations Ground Segment –Operated by ESA –Responsible for the satellite operations –Composed of the MOC and the Ground Stations and communication networks Data Processing Ground Segment –Jointly operated by ESA and DPAC –Responsible for payload monitoring and the scientific data reduction and analysis –Comprises of the SOC and a set of the DPCs

Rocio Guerra European Space Astronomy Centre 4 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Ground Stations Antennas During the LEOP and during critical mission phases, the 15-m Kourou will be used. Maspalomas and Perth as backups During transfer and commissioning New Norcia will be used for contact with s/c. Kourou will be backup The 35m Cebreros (and 35m New Norcia) will be during all other mission phases.

Rocio Guerra European Space Astronomy Centre 5 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 MOC team Preparation and execution of all operational tasks in order to achieve the optimum performance of the s/c in orbit The Science Operations Centre in ESAC is responsible of: The Mission Operations Centre in ESOC (Darmstadt) is responsible for: Photo: David Milligan SCOM Spacecraft Operations Manager David Milligan

Rocio Guerra European Space Astronomy Centre 6 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 MOC Responsibilities. This includes: S/C and payload activities Orbit determination and control Attitude determination and control On-board software maintenance Making available the raw satellite data to the SOC

Rocio Guerra European Space Astronomy Centre 7 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC – the beginning 2006 instrument commanding (mode changes, calibration uploads, etc) Initial Data Treatment Getting data to DPAC (Science Data Processing) The Science Operations Centre in ESAC (Madrid) is responsible for:

Rocio Guerra European Space Astronomy Centre 8 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC team

Rocio Guerra European Space Astronomy Centre 9 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC team – 2019????

Rocio Guerra European Space Astronomy Centre 10 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 SOC Responsibilities Part of ESA Ground Segment with MOC Development led by ESA Project Interface and contact point between MOC and DPAC Both an ESA SOC and DPAC Data Processing Centre Operate ‘MOC-facing’ tasks (MOC Interface Task and Payload Operations System) Operate IDT, First Look, AGIS, Main Database – DPAC-facing tasks Archive science data collected during the mission Disseminate sets of the database to the DPCs and re- integrate the results

Rocio Guerra European Space Astronomy Centre 11 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 DPAC The Science Operations Centre in ESAC is responsible of: The Gaia Data Processing is done by the Data Processing and Analysis Consortium (DPAC) That is a community effort (>300 people) The SOC will operate within DPAC and will rely heavily on its expertise in many areas The Project Scientist oversees all this (advised by GST) CU1: System Architecture CU2: Data Simulations CU3: Core Processing CU4: Object Processing CU5: Photometric Processing CU6: Spectroscopic Processing CU7: Variability Processing CU8: Astrophysical Parameters CU9: Catalogue Access

Rocio Guerra European Space Astronomy Centre 12 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Architecture The Science Operations Centre in ESAC is responsible of: Highly distributed Multiple independent DPCs and CUs Want/need decouple Reduce dependencies risk Hub and spokes Max flexibility for CUs and DPCs Minimum ICDs

Rocio Guerra European Space Astronomy Centre 13 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Daily Processing at SOC IDT (Initial Data Treatment) and FL (First Look) form together with MIT (MOC Interface Task) the daily processing pipeline which will run in the SOC IDT and FL are both being developed in CU3 Both IDT and FL are Schedule Critical

Rocio Guerra European Space Astronomy Centre 14 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 MIT MOC Interface Task Part of the daily processing pipeline (with IDT and FL) Schedule Critical Receives raw telemetry from the MOC (star packets, ASD data, HK telemetry) in file format and writes that data to the IDT/FL database for downlink software (IDT, FL) to pick up Reconstructs ’bad packets’ where possible Sends to MOC data from the POS

Rocio Guerra European Space Astronomy Centre 15 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 POS Payload Operations System Monitors spacecraft attitude data against model attitude and attitude derived from ground processing, for deviations. Generates (for the MOC) Scanning Law (model attitude) data Science Event Files and Avoidance periods Payload Operations Requests Optical data (observations of Gaia) Generates (for DPAC) Exposure Maps

Rocio Guerra European Space Astronomy Centre 16 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Interface with the MOC

Rocio Guerra European Space Astronomy Centre 17 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 IDT Purpose of Initial Data Treatment: Generate refined attitude data Compute basic image parameters from telemetry Match observations to known sources (cross matching)‏ Convert telemetry in raw observations IDT is being developed by UB in Barcelona IDT is run every cycle on large simulated data sets at ESAC. The results are used to test several other systems in DPAC.

Rocio Guerra European Space Astronomy Centre 18 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 FL Purpose of First Look: Check the instrument health on a daily basis The ‏FL system consist of: One day calibrations (ODAS, PODC, LODC, CODC, RODC)‏ Detailed FL (DFLM, DFLE) ‏ FL is being developed by ARI in Heidelberg with contributions from other institutes

Rocio Guerra European Space Astronomy Centre 19 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Further Daily processing After IDT and FL run data is sent to all DPCs Cambridge will run part of the CU5 photometric pipeline to look for interesting objects –Science Alerts may be issued. CNES will run a more detailed RVS calibration and First Look and provide feedback to SOC. CNES will also run CU4 software to detect interesting SSOs Torino will perform Basic Angle Monitor(BAM) data processing

Rocio Guerra European Space Astronomy Centre 20 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Data Reduction.. HUGE task Gaia data reduction is iterative Starts with a release and distribution of a version of the Gaia Main Database to the DPCs by SOC MDB release fed through the processing systems at each DPC New version of the Main Database assembled at SOC from the results of the data processing New data received during a data reduction cycle is included in next release of the MDB

Rocio Guerra European Space Astronomy Centre 21 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 The Science Operations Centre in ESAC is responsible of: Version each 6 Months ~20 TB inc. each version Hence ~200TB in final GTS pushes data to other DPCs Governed by ICD MainDB – the data hub

Rocio Guerra European Space Astronomy Centre 22 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 On-going activities Integration and testing of software to run at SOC Processing of simulation data to provide input data sets for CU tests Planning, design and implementation of the DPAC Integration Tests, to culminate in the 1+ year End-to-End test Leading role in defining operations Preparation of plans and procedure for running software, databases, transfer systems Configuration control and QA activities

Rocio Guerra European Space Astronomy Centre 23 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Gaia Catalogue Part of DPAC: CU9 Reporting, etc. similar to others CUs Some DPAC members will have to be in CU9 (no one understands the data and algorithms better) No real possibility to have initial catalogue before months after launch -> CU9 after launch

Rocio Guerra European Space Astronomy Centre 24 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 The ARCHIVE is not simply MDB MDB is CU oriented and contains redundancy Archive more cohesive and present single coherent dataset Mode of access is radically different: Arbitrary queries, data mining, download in various formats, etc Replicas ESA must have a copy Any institute involved should be allowed a replica is serious about it Portability is therefore an essential requirement Archive in context

Rocio Guerra European Space Astronomy Centre 25 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Archive in context CU9 may decide on another decomposition

Rocio Guerra European Space Astronomy Centre 26 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Interface / Ingestor COMMUNITY INTERFACE Should be Public Oriented Basics with no login Easy self-registration for advanced features Some sort of Sky-Browser (ala SDSS, GoogleSky) Multi-lingual of course INGESTOR MDB ICD already exists (contains all data) Lots of data Probably needs to be distributed Should not take too long – one week

Rocio Guerra European Space Astronomy Centre 27 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Interrogator INTERROGATOR Need fast engine to answer queries DBMS tuning is needed Should allow powerful queries to user (SQL and/or ADQL) Local space for registered users to upload data and store query results Extraction in multiple formats (csv, FITS, VOtable) VO access of course … TAP, SIAP … And must be … FAST!!!

Rocio Guerra European Space Astronomy Centre 28 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Advanced Applications ADVANCED APPLICATIONS They are “added value” applications 3D visualization for all for part of sky (e.g. globular cluster) Light curve tools … Interrogator should provide an API to build these apps on

Rocio Guerra European Space Astronomy Centre 29 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Science Alerts / Help Desk SCIENCE ALERTS Bit of a grey area for now Flux based alerts straight from CU5 NEO orbits from CU4 go direct to IMCCE Others will come up – processing first!!!! CU9 will provide a facility for DPAC to publish alerts However alerts sent should be archived by CU9 Follow up should be gathered HELP DESK Usual problem reporting, tracking, FAQ, etc. Some real people somewhere … Should be ready to politely answer all questions

Rocio Guerra European Space Astronomy Centre 30 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Documentation DOCUMENTATION Uniform documentation on all data MDB dictionary provides a lot on the data Algorithm doc from CUs Bunch of statistics (e.g. maps source, observation, etc) Derived plots such us HR, galactic-kinematics, Hess diagram,… Printed volume would be nice! At least Source Catalogue on media… Probably most important CU9 task!!!! Probably most important CU9 task!!!!

Rocio Guerra European Space Astronomy Centre 31 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Releases very tentative … Yes more than one – we wont make you wait till But all remains tentative 1. Probably one after 2-3 years Some Astrometry and some Photometry 2. One more after 5 years Full astrometry better photometry, spectra 3. Then the final one 8 years after launch Variable stars, astroparams …

Rocio Guerra European Space Astronomy Centre 32 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Open areas Living archive (Anthony Brown) Can we/should we allow additions to the Archive? e.g. improved solutions for binaries using follow observations Implications for maintenance, quality and security Cross matching other catalogues …

Rocio Guerra European Space Astronomy Centre 33 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 Questions ?? Ariane V188 carrying Herschel and Planck (May )