Data Management at Gaia Data Processing Centers GREAT Workshop on Astrostatistics and Data Mining in Astrnomical Databases La Palma, Spain May 30 - June.

Slides:

Advertisements

Similar presentations

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Jesús Salgado SLAP Implementations Sep 2006, Moscow, Russia Simple Line.

Advertisements

SLAP: Simple Line Access Protocol v0.5

Andrew Hanushevsky7-Feb Andrew Hanushevsky Stanford Linear Accelerator Center Produced under contract DE-AC03-76SF00515 between Stanford University.

CASDA Virtual Observatory CSIRO ASTRONOMY AND SPACE SCIENCE Arkadi Kosmynin 11 March 2014.

GENIUS kick-off - November 2013 GENIUS kick-off meeting The Gaia context: DPAC & CU9 X. Luri.

NEAT: Very high precision astrometry to detect nearby planetary systems down to one Earth mass F. Malbet, A. Crouzier, M. Shao, A. Léger and the NEAT collaboration.

Multi-Data-Center Hadoop in a Snap Dr. Konstantin Boudnik Vice President, Open Source Development.

Rocio Guerra European Space Astronomy Centre 1 Gaia: la Galaxia en un Petabyte Mao- Menorca – 2 nd October 2009 ESAC and the Gaia Catalogue

Grid S.G. Ansari 15 June June June 2015 VSWG – Observatoire de Genève Variability detection, period search with GaiaGrid S. Ansari, L. Eyer,

SWOT mission Current and ongoing activities at CNES.

Compilation of stellar fundamental parameters from literature : high quality observations + primary methods Calibration stars for astrophysical parametrization.

30 March 2006Birmingham workshop1 The Gaia Mission A stereoscopic census of our Galaxy.

Next Generation of Apache Hadoop MapReduce Arun C. Murthy - Hortonworks Founder and Architect Formerly Architect, MapReduce.

Die Vermessung der Milchstraße: Hipparcos, Gaia, SIM Vorlesung von Ulrich Bastian ARI, Heidelberg Sommersemester 2004.

U.S. Department of the Interior U.S. Geological Survey David V. Hill, Information Dynamics, Contractor to USGS/EROS 12/08/2011 Satellite Image Processing.

2/10/2000 CHEP2000 Padova Italy The BaBar Online Databases George Zioulas SLAC For the BaBar Computing Group.

The Gaia mission Data reduction activities in the UK Floor van Leeuwen, IoA.

Gaia, next frontier in Astronomy Jose Hernandez Gaia Data and Calibration Engineer European Space Astronomy Centre (ESAC) Madrid, Spain.

CS525: Big Data Analytics Machine Learning on Hadoop Fall 2013 Elke A. Rundensteiner 1.

Ground based observations for Gaia 2001 : need to have reference stars to calibrate AP algorithms for Gaia i.e. stars with well-known APs that will observed.

GENIUS kick-off - November 2013 GENIUS kick-off meeting WP400 – Tools for data exploitation X. Luri.

Pilar de Teodoro Report on ESA Gaia activities & VLDB 2010 Pilar de Teodoro Gaia Science Operations database administrator European Space Astronomy Centre.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Jesús Salgado SLAP Implementations May 2007, Beijing, China Simple Line.

Spectroscopy in VO, ESAC Mar Access to Spectroscopic Data In the VO Doug Tody (NRAO/US-NVO ) for the IVOA DAL working group I NTERNATIONAL.

1 Workshop First look, calibrations, reference sources IAP – 24 November CU6 structure 2.Aims of the workshop / open questions.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Isa Barbarisi VOSpec, new functionalities Madrid, 6-7Oct 2005 New functionalities.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Jesús Salgado Spectroscopic lines in the VO context Mar 2007, ESAC, Madrid,

Strasbourg astronomical Data Centre (DS) Françoise GENOVA.

Planetary Science Archive PSA User Group Meeting #1 PSA UG #1  July 2 - 3, 2013  ESAC PSA User Support.

E. Solano, R. Gutiérrez, B. Montesinos, C. Morales, J. García, L. Sanz LAEFF-INTA. P.O. Box 50727, Madrid (Spain) Development of a multi-mission.

The Three Dimensional Universe with GAIA Paris-Meudon, October 4-7, 2004 Gaia First Look: Description and Status Report Stefan Jordan, Uli Bastian, Helmut.

RVS Calibration Workshop, Paris RVS RVS Calibration RVS Calibration & First Look Workshop, Paris Mark Cropper.

PLATO Data Center: Purpose and Structure Laurent Gizon (PDPM) Hamed Moradi (PDC Project Office)

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Pedro OSUNA ESAC ADT Team VO-tech Cambridge 2004 VOSpec: A Tool to Handle.

Source catalog generation Aim: Build the LAT source catalog (1, 3, 5 years) Jean Ballet, CEA SaclayGSFC, 29 June 2005 Four main functions: Find unknown.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Applications May 2006, Victoria, Canada VOQuest A tool.

Scanning sky monitor (SSM) Technical Physics Division, ISAC & Astrophysics Group, RRI.

Early science on exoplanets with Gaia A. Mora 1, L.M. Sarro 2, S. Els 3, R. Kohley 1 1 ESA-ESAC Gaia SOC. Madrid. Spain 2 UNED. Artificial Intelligence.

DDM Kirk. LSST-VAO discussion: Distributed Data Mining (DDM) Kirk Borne George Mason University March 24, 2011.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Jesús Salgado AIDA Tech. Meeting Strasbourg, March 2009 (1/8) WP7 Task.

ESAVO/European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Pedro Osuna ASVOWS Mar 2007 ESAC Astronomical Spectroscopy.

Computing Systems: Next Call for Proposals Dr. Panagiotis Tsarchopoulos Computing Systems ICT Programme European Commission.

GENIUS Kick-Off meeting December 4th 2013 WP-620 Simulated catalogue data Francesc Julbe UB – IEEC, Barcelona.

E. Solano. GAIA Meeting, Menorca, Oct 2009 GAIA and the Virtual Observatory Enrique Solano, LAEX/CAB (INTA-CSIC) Spanish VO Principal Investigator.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Pedro Osuna VOSpec Kyoto May 2005 VOSpec: A Tool to Handle VO-Compatible.

FoV: 0.7 deg x 0.7 deg, pixel (10 µm x 30 µm): 0.059”(AL) x 0.177”(AC) 106 CCD 4500x1966 px (TDI) ~4.4 sec 0.93m 0.42m Skymapper.

1 HBASE – THE SCALABLE DATA STORE An Introduction to HBase XLDB Europe Workshop 2013: CERN, Geneva James Kinley EMEA Solutions Architect, Cloudera.

Next Generation of Apache Hadoop MapReduce Owen

BIG DATA/ Hadoop Interview Questions.

Module 6: Configuring and Managing Windows SharePoint Services 3.0.

Gaia Data Processing Coryn A.L. Bailer-Jones Max-Planck-Institut für Astronomie, Heidelberg on behalf of the Data Processing and Analysis Consortium (DPAC)

Source catalog generation Aim: Build the LAT source catalog (1, 3, 5 years) Jean Ballet, CEA SaclaySLAC, 23 May 2005 Four main functions: Find unknown.

DU15: Internal Photometric Calibration Dafydd Wyn Evans, IoA

Energy efficient SCalable

Gaia DR2/3: Serving Time Series, Spectra, SSOs in the VO

Zhangxi Lin, The Rawls College,

Institute of Cosmos Sciences - University of Barcelona

Vibration in turbine blades must be prevented

ESA Gaia Archive: Architecture & key points

Introduction to Software Process

Gaia impact on asteroidal occultations

Gaia impact on asteroidal occultations

Overview of big data tools

Data analytics with Hadoop In the Microsoft Azure cloud

Big Data Young Lee BUS 550.

NAF Product Training.

GLAST Large Area Telescope Instrument Science Operations Center

GENIUS CSIC contribution Enrique Solano

Presentation transcript:

Data Management at Gaia Data Processing Centers GREAT Workshop on Astrostatistics and Data Mining in Astrnomical Databases La Palma, Spain May 30 - June 3, 2011 Pilar de Teodoro Idiago Gaia Database Administrator European Space Astronomy Center (ESAC) ‏ Madrid Spain

Data Processing Centres *DPCE (ESAC) ‏ * DPCB (Barcelona) ‏ * DPCC (CNES) ‏ * DPCG (Obs. Geneva / ISDC) ‏ * DPCI (IoA, Cambridge) ‏ * DPCT (Torino) ‏ All contributed to this talk Data Processing Centers

Photometry Treatment Calibrate flux scale give magnitudes Spectral Treatment Calibrate and disentangle provide s spectra Astrometric Treatment Fix geometrical calibration Adjust Attitude Fix source positions Variability Astrophysical Parameters Non Single Systems Solar System Many iterations Catalogue Many iterations Processing Overview (simplified) ‏ Initial Data Treatment Turn CCD transits into source observations on sky Should be linear transform CU3 CU3/SOC CU5 CU6 CU4 CU7 CU8 CU4

DPCE

DPCB

DPCC (CNES) CU4 (Objects Processing), CU6 (Spectroscopic processing) CU8 (Astrophysical Parameters) Solutions based on: performance scalability of the solution data safety impacts on the existing software impacts on the hardware architecture cost of the solution during the whole mission durability of the solution administration and monitoring tools

DPCG Detection and characterization of variable sources observed by Gaia (CU7) Analytical queries must be done over sources or processing results (attributes) to support unknown research requirements. Timeseries reconstruction while importing MDB data Parameter analysis for simulations and configurations changes on historical database. ETL-like support must be done for external data. At present Apache OpenJPA. Postgress used as well. Other alternatives : Hadoop, SciDB, VoltDB and Extensions to PG.

DPCI Given the use case: bulk-processing of a large data set data volume increases with time (DPAC-wide iterations) We can state that: Random data access is expensive and less efficient than sequential access. Hub-and-Spoke architecture is prone to bottlenecks and therefore does not scale very well with the number of clients. Hadoop adopted in 2009 HDFS:distributed filesystem Map/Reduce jobs to minimize synchronization DAL much simpler

DPCT CU3 AVU IGSL support Persistent data management