04/02/2019 The Use of Grid Technology in Large Scale Data Processing Collaboration Environments S.G. Ansari S.G. Ansari 04/02/2019

The Human Eye Resolution: 2 car headlights 3 km away 04/02/2019 The Human Eye Resolution: 2 car headlights 3 km away 1 Eye pixel has an angular resolution of 3’ 3600/3x3 = 400 pixels/deg2 Angle of Eyesight = 45° 6000 deg2  2.5 x 106 pixels At a shutter of 20 Hz and 3x8 bit/picture element Raw data rate of 1 Gb/s 150 Mbytes/sec ! U. Bastian Univ. Heidelberg S.G. Ansari 04/02/2019

04/02/2019 S.G. Ansari 04/02/2019

The ESA Astronomical Data Volume 04/02/2019 The ESA Astronomical Data Volume C. Arviset (ESAC) ESA Science Data involve a large number of Data Centres distributed all across Europe. Access to these data is also widely distributed S.G. Ansari 04/02/2019

The Gaia Challenge An example of a Collaboration Environment 04/02/2019 The Gaia Challenge An example of a Collaboration Environment Managing 1 Petabyte of data Astrometry of 1 billion stars over 5-year time span 100 positional determinations per star to yield micro-arcsecond accuracy two photometric systems and 1 Radial Velocity spectrometer calibration data S.G. Ansari 04/02/2019

Collaborative Tasks Scientific: 04/02/2019 Collaborative Tasks Scientific: Shell Tasks involve the whole Gaia community Shell Tasks may be developed by autonomous groups, independent of a core team Shell Tasks deliver “derived” data Shell Tasks can be collaborative tools Shell Tasks are building blocks for data analysis. They may be combined to address more complex processing tasks Technical: Shell Tasks can be modular Shell Tasks access the Gaia Database to work on a subset of data Shell Task results can be independently validated. Less interaction with the core data. Shell Tasks could be developed in multiple set programming languages S.G. Ansari 04/02/2019

The Gaia Virtual Organisation 04/02/2019 Core Tasks RVS Quick Looks Photometry Astrometry ABS Minor Planets Variable Stars Fundamental Algos The Gaia Virtual Organisation Some 20 institutes collaborate on establishing a relevant set of tasks for the Gaia Data Processing S.G. Ansari 04/02/2019

The Grid The Grid is: It is ideal for the Shell Tasks 04/02/2019 The Grid The Grid is: A resource sharing concept Used to augment computational resources whenever and wherever needed Ideal to build a collaborative environment, where users can share algorithms and analyse data It is ideal for the Shell Tasks S.G. Ansari 04/02/2019

04/02/2019 The Grid Architecture The best current example of a Grid implementation is Google! Applications Middleware Infrastructure S.G. Ansari 04/02/2019

Abandon the geographical distribution 04/02/2019 Where do we go from here? The Grid exercise is relevant to very huge amounts of number crunching Network latency adds unnecessary overheads to the problem CPU is cheap Abandon the geographical distribution HOWEVER S.G. Ansari 04/02/2019

The Virtual Collaboration Aspect 04/02/2019 The Virtual Collaboration Aspect Grid infrastructure is ideal for large collaboration environments: 10 sites or more. Data can be distributed with a single central “master copy”. Virtual Organisations are the answer for future Collaborations S.G. Ansari 04/02/2019

04/02/2019 The Future As the quantity of data increases, so must the quality of its organisation and analysis Our scientific tools must reflect the changing ways with which we do conventional science Our interaction with the data must evolve. Analysis tools must become more human-friendly and intuitive S.G. Ansari 04/02/2019