1. Lehrstuhl Informatik III: Datenbanksysteme Astrometric Matching - E-Science Workflow 1 Lehrstuhl Informatik III: 1 Datenbanksysteme 1 Fakultät für Informatik 1 Technische Universität München Max-Planck-Institut 2 für Astrophysik Max-Planck-Institut 3 für extraterrestrische 3 Physik Full paper at the: 2nd IEEE International Conference on e-Science and Grid Computing, Dec. 4-6, 2006, Amsterdam

2. Lehrstuhl Informatik III: Datenbanksysteme 2 Astrometric Matching - E-Science Workflow The SED Scenario 1.Catalog query 2.Astrometric (spatial) matching 3.Assembly of raw photometry 4.Photometric transformation 5.SED classification

3. Lehrstuhl Informatik III: Datenbanksysteme 3 Astrometric Matching - E-Science Workflow Astrometric (Spatial) Matching Current solutions … … load all data into main memory Uses a lot of memory Infeasible if memory size is insufficient … process all data at once and deliver the complete result at the end Inefficient No results until all processing has completed

4. Lehrstuhl Informatik III: Datenbanksysteme 4 Astrometric Matching - E-Science Workflow Our Contributions In-network processing Early filtering Parallelization Pipelining (data streaming) Load-balancing Mobile user-defined operators Dynamic integration Extensible framework Legacy applications

5. Lehrstuhl Informatik III: Datenbanksysteme 5 Astrometric Matching - E-Science Workflow The StarGlobe Architecture

6. Lehrstuhl Informatik III: Datenbanksysteme 6 Astrometric Matching - E-Science Workflow Mobile User-Defined Operators Infrastructure provided by StarGlobe encapsulated operators provided by community Load user-defined operators from function provider servers in the network Common interface for integrating external operators Flexibility

7. Lehrstuhl Informatik III: Datenbanksysteme 7 Astrometric Matching - E-Science Workflow Communication between Stream Processor and Stream Iterator

8. Lehrstuhl Informatik III: Datenbanksysteme 8 Astrometric Matching - E-Science Workflow Astrophysical Example Workflow

9. Lehrstuhl Informatik III: Datenbanksysteme 9 Astrometric Matching - E-Science Workflow Astrophysical Example Workflow (Setup)

10. Lehrstuhl Informatik III: Datenbanksysteme 10 Astrometric Matching - E-Science Workflow Distributed Query Evaluation Plan

11. Lehrstuhl Informatik III: Datenbanksysteme 11 Astrometric Matching - E-Science Workflow Conclusion StarGlobe Prototype Handling large data volumes efficiently Early filtering, parallelization, pipelining Returning first results early on Pipelining (data streaming) Flexible support of domain-specific application logic Mobile user-defined operators Results also applicable to other domains