1. RRFWG. Dresden June 11th 2003 1 GDAAS II Algorithm Implementation Review

2. RRFWG. Dresden June 11th 2003 2 GAIA data analysis understood to be a complex task - Data volume: 20-50 TB data over 5 yrs, - 10 20 flop - Complexity: Data ‘mixed’ in time and space due to scanning motion,... Hipparcos approach (flat files/sequential processing) inappropriate Major software engineering infrastructure required Introduction GAIA Data Analysis and Access System

3. RRFWG. Dresden June 11th 2003 3 GDAAS Phase I Objectives Define an efficient, scalable, maintainable and usable system for populating the GAIA mission database from the satellite data stream: –The GAIA Data Access and Analysis System should allow the reduction processes to be optimally run. –Data must be accessed readily in the time and spatial domain. Challenge: Establish the technical baseline concepts for the system on realistic basis and prove the feasibility of the approach chosen for the reduction of the mission.

4. RRFWG. Dresden June 11th 2003 4 Objectives Investigate feasibility of an Object-Oriented Approach to Data Manipulation Analyse an appropriate Data Storage, Processing and Access Concept Simulate an end-to-end data processing environment for the Operational Phase Devise and implement a mechanism to incorporate critical algorithms produced by the GAIA Science Community Investigate Performance Issues of the System

5. RRFWG. Dresden June 11th 2003 5 Consortium GMV: Management and Software development UB: Scientific support and customisation of the GAIA simulator CESCA: Hardware infraestructure, processing power and on-line support. Project team organisation GMV TeamUB TeamCESCA Team GMV Prime Contractor ESA/ESTEC

6. RRFWG. Dresden June 11th 2003 6 The Context Objectivity was chosen as the Database Management System for the Study Only the ASTRO Instruments were considered A Data Model was developed, based on GAIA’s specifications Some critical Algorithms were identified and implemented

7. RRFWG. Dresden June 11th 2003 7 Definition of a satellite model Definition of the DB data model Processing requirements Interface requirements System specification

8. RRFWG. Dresden June 11th 2003 8 Prototype Development Design –Design of the GAIA database Data Model Refinement Data Manipulation Layer –Design of the Processing Framework Implementation –Database Data Model and Database Manipulation Layer –Processing algorithms –Processing Framework Testing –Integration and Validation at CESCA premises

9. RRFWG. Dresden June 11th 2003 9 System Evaluation Based on three main processes: Ingestion, Cross Matching (M. Lattanzi) and GIS (L. Lindegren) Data storage: sizing, overheads and access time Distributed processing performances Resources requirements (memory/disk/CPU); Precision of the processed data.

10. RRFWG. Dresden June 11th 2003 10 Data processing structure: prototype GASS simulator Telemetry Stream Ingestion & Initial data treatment Attitude Updating Astrometric updating Calibration Global - Raw Observations - Obs2Elem. - Centroiding Sources Cross-Matching Global Iterative Solution The GAIA Database

11. RRFWG. Dresden June 11th 2003 11 GDAAS Phase I Conclusions The approach chosen has proved quite succesfully O-O + UML tools demonstrated its advantatges in the implementation of this complex system Java has demonstrated to be ideal for the problems posed by the system The choice of the DBMS has shown to be a key element To get good concurrent performance on ingestion and CM was an expensive task GIS complexity increased by the use of wrappers Some Test results

12. RRFWG. Dresden June 11th 2003 12 GDAAS Phase II

13. RRFWG. Dresden June 11th 2003 13 Objectives The objective of the Phase II study is to provide complete confidence in the overall GAIA data processing approach, identifying interfaces with all foreseen data reduction steps, implementing and testing an agreed package of algorithms provided by the wider GAIA community, and demonstrating scalability to a final data processing system. Aims to deliver to ESA a processing environment capable of providing the basis for the data analysis system applicable to the GAIA mission data.

14. RRFWG. Dresden June 11th 2003 14 Tasks (I) Selection and recomendation of the processing hadware and software environment Recommend Programming language rules for critical components Incorporate the changes in the baseline payload Develop suitable data performance monitoring tools Demonstration of scalability to the final data processing system Definition of the overall data processing structure: “what algorithms go where”

15. RRFWG. Dresden June 11th 2003 15 Tasks (II) Establish priorities for algorithmic development - Representative algorithms should be targetted in all major areas - Simplified modules may be defined, but with clear goals and outputs - Selected algorithms should be representative of algorithm and interface complexities - Identified algorithms should lead to confidence in number of flop - Identification of the necessary algorithms should help demonstrate the feasibility of the GDAAS approach

16. RRFWG. Dresden June 11th 2003 16 Tasks (III) Propose and implement an overall configuration control strategy Definition of a comprehensive test plan. Definition and procurement of the storage and processing hardware

17. RRFWG. Dresden June 11th 2003 17

18. RRFWG. Dresden June 11th 2003 18 The Role of the DPWG 1. Categorise each Algorithm in terms of whether it is a Core/Shell/Quick Look 2. Validate the context within which an algorithm is being proposed to fit the GAIA specifications. 3. Assess the scientific validity of the algorithm. 4. Ensure that all the information being provided is complete. 5. Check the completeness of each test case and the expected results

19. RRFWG. Dresden June 11th 2003 19

20. RRFWG. Dresden June 11th 2003 20

21. RRFWG. Dresden June 11th 2003 21

22. RRFWG. Dresden June 11th 2003 22 The role of the Configuration Control Board Review all algorithms proposed by the DPWG in light of their implementation into GDAAS. Classify the algorithms according to their criticality. Schedule the implementation of a particular algorithm Verify the implementation and use of the required version of the identified GAIA Simulator components. Select and implement a Configuration Control Tool for the GDAAS Review the source code rules Manage the GDAAS Library: a set of routines, readily available for use by the community. Review the Interface Control Document for coding practices within GDAAS. Monitor and review on a regular basis all algorithms being implemented Monitor all the test procedures accordingly. Run a Risk Assessment dealing with GDAAS implementation issues.

23. RRFWG. Dresden June 11th 2003 23 Algorithms to be delivered by July 2003 Improved object matching ASM handling Fundamental algorithms Improved attitude modelling (preliminary) Improved instrument model Chromaticity calibration (preliminary) Astrometric binary star analysis Planets Photometric raw data treatment Radial velocity cross correlation Minor planet detection Discrete source classifier (preliminary) Single star parametrizer (preliminary)

24. RRFWG. Dresden June 11th 2003 24 Algorithms to be delivered by Sept. 2003 Visual double star analysis (preliminary) Ingestion level classifier (preliminary)

25. RRFWG. Dresden June 11th 2003 25 Algorithms to be delivered by April 2004 Improved attitude modelling (update) Detailed geometrical calibration Chromaticity calibration (update) CCD calibration (preliminary) Additional global parameters Visual double star analysis (update) Science alerts Radial velocity source detection Radial velocity wavelength calibration Multiple source detector (preliminary) Single quasar parametrizer (preliminary)

26. RRFWG. Dresden June 11th 2003 26 All these algorithms require simulations of a matching level of detail to be tested. The algorithm providers should in principle provide the appropriate modules, but it is the responsibility of the SWG to provide the basis for these simulations (GASS & GIBIS) and to ensure its implementation

27. RRFWG. Dresden June 11th 2003 27 Algorithm SUBMISSION Follows the following steps: 1.Proposal by author/reviewer 2.DPWG Evaluation 3.CCB Evaluation 4.Implementation and Testing

28. RRFWG. Dresden June 11th 2003 28 Title: Name of the algorithm Author: Name of author providing the algorithm Reviewer: Name of the reviewer (normally a Working Group leader) Method: Details of how the algorithm is to be used frequency of execution impact it may have on data (large/small modifications) requirement of temporary disk space Function: Explanation of what the algorithm accomplishes from a scientific standpoint and a brief high level description of how it achieves it. Assumptions: Scientific assumptions made during the design and implementation of the algorithm, including those concerning the values of constants

29. RRFWG. Dresden June 11th 2003 29 Input Data: For each input for each step of an algorithm Correspondence – From the GDAAS Data Model or the necessary pre-processing described in pseudo-code. If the input is passed from a previous step it should be stated explicitly Type – The type used in the algorithm and if necessary the format. Units – The units used in the algorithm. Range – When only a certain range of values is accepted Conditions –special conditions, e.g. observations must be linked to a source (i.e. cross-matched) before their values can be used. Output Data: Analogous information to that provided for inputs. Details of the meaning of any output error codes should be provided in order to assist the implementation team during testing.

30. RRFWG. Dresden June 11th 2003 30 Source Code: The actual file(s) containing the source code. All delivered code should be compliant with the appropriate language standards Test Data : Input and output data to verify that the implementation was succesful More Details: Algorithm Interface Control Document GDAAS Algorithm Preparation Guidelines GATT: http://gaia.esa.int/algorithms/

31. RRFWG. Dresden June 11th 2003 31 How to access the database A standard set of routines takes care of retrieving and updating the DB The GDAAS team will take care of the wrappers needed to interface to the DB To restrict the observation: by time interval sky region (HTM) instrument or range of CCDs

32. RRFWG. Dresden June 11th 2003 32 Simulations: The GAIA simulations are responsability of the GAIA scientific community The telemetry stream for GDAAS is provided by GASS (GAIA System Simulator) The providers must be sure that the complexity of his algorithm is matched by the simulations. If not they should provide appropriate code to the SWG

33. RRFWG. Dresden June 11th 2003 33 Present status of GDAAS Test on GAIA 1 design + Oracle is going on Fully distributed HW: 1Tb, Oracle 9i RAC, Results: Ingestion up to 2.5 years already done CM is running for 6 months telemetry GIS: re-designned, running on 12h telemetry

34. RRFWG. Dresden June 11th 2003 34 Present status of GDAAS Design of telemetry according to the new GAIA design New Data Model in progress according to the GAIA design and the existing algorithm information System: design is going on preparing for the implementation of algorithms

35. RRFWG. Dresden June 11th 2003 35 Ing. - CM.- GIS (Improved) + Other Oracle 9iNewGDAAS 2 Phase 2 Ing. - CM.- GIS (First approx.) Oracle 9iOld. Red Book Only Astro GDAAS 2 Phase 1 Ing. - CM.- GIS (First approx.) ObjectivityOld. Red Book Only Astro GDAAS 1 AlgorithmsDB engineGAIA design

36. RRFWG. Dresden June 11th 2003 36 SIMULATOR: SATELLITE MODEL GAIA barycentric ephemeris: satellite orbit around L2 point (provided by F. Mignard) GAIA scan law: nominal scan law without perturbations (provided by L. Lindegren) GAIA parameters: compilation of satellite and instrument parameters (UB-SWG-012). Only Astro instrument is considered.

37. RRFWG. Dresden June 11th 2003 37 UNIVERSE MODEL: GALAXY MODEL Torra et al. (Universitat de Barcelona) Single stars Double star (provided by F. Arenou) Variable stars: no actual light curves and distribution period implemented. Extintion model: F. Arenou The JAVA structure allows in a easy way to implement other galaxy model -> a implementation of the more realistic Besançon Galaxy Model (A. Robin et al.) is on going.

38. RRFWG. Dresden June 11th 2003 38 Algorithm Implementation deadlines: July, 1st July, 7th CCB meeting December, 03 First set of algorithms implemented December, 04 Completion of GDAAS2