1. 04/02/2019
The Use of Grid Technology in Large Scale Data Processing Collaboration Environments
S.G. Ansari
S.G. Ansari
04/02/2019

2. The Human Eye Resolution: 2 car headlights 3 km away
04/02/2019
The Human Eye
Resolution: 2 car headlights 3 km away
1 Eye pixel has an angular resolution of 3’
3600/3x3 = 400 pixels/deg2
Angle of Eyesight = 45°
6000 deg2  2.5 x 106 pixels
At a shutter of 20 Hz and 3x8 bit/picture element
Raw data rate of 1 Gb/s
150 Mbytes/sec !
U. Bastian
Univ. Heidelberg
S.G. Ansari
04/02/2019

3. 04/02/2019
S.G. Ansari
04/02/2019

4. The ESA Astronomical Data Volume
04/02/2019
The ESA Astronomical Data Volume
C. Arviset (ESAC)
ESA Science Data involve a large number of Data Centres distributed all across Europe. Access to these data is also widely distributed
S.G. Ansari
04/02/2019

5. The Gaia Challenge An example of a Collaboration Environment
04/02/2019
The Gaia Challenge An example of a Collaboration Environment
Managing 1 Petabyte of data
Astrometry of 1 billion stars over 5-year time span
100 positional determinations per star to yield micro-arcsecond accuracy
two photometric systems and 1 Radial Velocity spectrometer
calibration data
S.G. Ansari
04/02/2019

6. Collaborative Tasks Scientific:
04/02/2019
Collaborative Tasks
Scientific:
Shell Tasks involve the whole Gaia community
Shell Tasks may be developed by autonomous groups, independent of a core team
Shell Tasks deliver “derived” data
Shell Tasks can be collaborative tools
Shell Tasks are building blocks for data analysis. They may be combined to address more complex processing tasks
Technical:
Shell Tasks can be modular
Shell Tasks access the Gaia Database to work on a subset of data
Shell Task results can be independently validated. Less interaction with the core data.
Shell Tasks could be developed in multiple set programming languages
S.G. Ansari
04/02/2019

7. The Gaia Virtual Organisation
04/02/2019
Core Tasks
RVS
Quick Looks
Photometry
Astrometry
ABS
Minor Planets
Variable Stars
Fundamental Algos
The Gaia Virtual Organisation
Some 20 institutes collaborate on establishing a relevant set of tasks for the Gaia Data Processing
S.G. Ansari
04/02/2019

8. The Grid The Grid is: It is ideal for the Shell Tasks
04/02/2019
The Grid
The Grid is:
A resource sharing concept
Used to augment computational resources whenever and wherever needed
Ideal to build a collaborative environment, where users can share algorithms and analyse data
It is ideal for the Shell Tasks
S.G. Ansari
04/02/2019

9. 04/02/2019
The Grid Architecture
The best current example of a Grid implementation is Google!
Applications
Middleware
Infrastructure
S.G. Ansari
04/02/2019

10. Abandon the geographical distribution
04/02/2019
Where do we go from here?
The Grid exercise is relevant to very huge amounts of number crunching
Network latency adds unnecessary overheads to the problem
CPU is cheap
Abandon the geographical distribution
HOWEVER
S.G. Ansari
04/02/2019

11. The Virtual Collaboration Aspect
04/02/2019
The Virtual Collaboration Aspect
Grid infrastructure is ideal for large collaboration environments: 10 sites or more.
Data can be distributed with a single central “master copy”.
Virtual Organisations are the answer for future Collaborations
S.G. Ansari
04/02/2019

12. 04/02/2019
The Future
As the quantity of data increases, so must the quality of its organisation and analysis
Our scientific tools must reflect the changing ways with which we do conventional science
Our interaction with the data must evolve. Analysis tools must become more human-friendly and intuitive
S.G. Ansari
04/02/2019