1. Big Data - Efficient SW Processing
April 3rd 2017

2. Agenda What is Big Data? What makes Big Data a Thing?
Does SKA Qualifies as a Big Data System?
Overview of the Slow Transients Pipeline
Resources Available & Implementation Challenges
Very Simple Example on Parallel Computing
Limits to Parallelisation
Is Big Data the same as High-Performance Computing?

3. What is Big Data?
A relatively new buzzword arising from the need to extract knowledge from the immense data volumes created by the ever increasing digitization of business and social life.
Big Data can be seen as three Vs:
Volume
In the order of terabytes, petabytes or more
Velocity
Real-time or near real-time processing
Variety
Social networks, e-business logs, sensor information, etc.

4. What makes Big Data a Thing?
Network Capacity
Over the 90s and 2000s, expanded at an incredible pace; companies raced to install land and submarine communication cables delivering a huge excess of bandwidth enabling interconnectivity as never seen before.
Storage Capacity
Storage capacity followed the pace of bandwidth; new discoveries such as giant magneto-resistance increased data density by orders of magnitude while data centres kept growing equally fast
Processing Capacity
The stand-alone CPU reached a point where it could scale no more as before, but the multi-core and ever faster networks made the processing power capable to match data size evolution.

5. Does SKA Qualifies as a Big Data System?
Exabytes/year from telescope array
2x300 petabyte/year through submarine links
Real-time closed-loop control
Complex events processed every second or less

6. Overview of the Slow Transients Pipeline
Slow transients last for hours, days or more than a week, but “slow” is misleading
Samples must be processed fast – e.g. within a second

7. Resources Available & Implementation Challenges
The essential features of high-performance computing, even if at small scale, at are the hand of almost everyone
Multi-CPU and/or multi-core machines
Parallel computing frameworks
Optimised math libraries
There are important implementation challenges however:
Partitioning of data sets
Work distribution and orchestration
Error recovery
Scalability and management
Usability of parallel computing technologies

8. Technology used in the Slow Transients Pipeline
Parallelisation Technology
Threading Building Blocks (TBB)
Programming Languages
C++11
Python
Mathematical Libraries
Armadillo
LAPACK
OpenBLAS
FFTW

9. Very Simple Example on Parallel Computing
Given a random data set of size 12 we can calculate the average of
The entire data set or else, the average of the average of non-overlapping subsets of sizes: 2, 3, 4 and 6
The algorithm can be parallelised respective among: 1, 6, 4, 3 or 2 computational cores, in a single machine or a cluster
On the other hand, this is not feasible for the standard deviation
STDEV of the sub-averages is the same as STDEV of the initial set
Algorithms depending from the full data set cannot be parallelised
Dependency from previous iteration state also breaks parallelisation

10. Limits to Parallelisation
Parallelisation is good but gains are all but linear, even when they seem linear as in the previous case
Time is required to partition the data set, distribute it and collect the results from the fractions – e.g. MapReduce time in Big Data
We may also be limited about how many partitions we can allocate at once – e.g. 1, 2 and 4 cores are feasible but 3 are not
The most important aspect to keep always in mind is that “all computers wait at the same speed”!

11. Is Big Data the same as High-Performance Computing?
Big Data and HPC are more alike than different, but have different roots – Big Data is more trendy now!
Big Data is an offspring of the Internet while HPC comes from the supercomputer world
Big Data uses technologies such a Hadoop and Spark, while HPC heavily relies in MPI
Big Data is more about detecting patterns out of user data while HPC is more about scientific computing
The basic mean however are the same: map partitions, distribute workloads, collect partial results, calculate aggregated result.

12. Concluding Thoughts
There is a fundamental difference between "as fast as possible" and "in real-time".
A multicore machine will not execute a SW any faster unless the SW is programmed to use multi-core.
The challenge is not in dealing with 2 or 2,000,000 cores, it is in dealing with one or more than one.
A parallel computing framework will not break the problem for you, it will only allow you do that.

13. SKA Pushing New Boundaries of the Known and Uncovering the Unknown

14. SKA Ecossystem SKA Radio-telescope Complementary Space Observatory
Centre for Processing & Transit (PT)
SKA Radio-telescope
First Level Processing Centre (ZA)
Complementary Ground Observatory
Complementary Space Observatory
Processing Centre (other location)
Telescope Control System (ZA)
Virtual Observatory Astronomy Events Data Base
Primary Data Flow (exabytes/year)
Secondary Data Flow (2x300PBytes Year Total) (10-15PBytes Year para PT)
Transient’s Coordinates (feedback for control)
Transient’s Coordinates
(feedback for complementary observation means)
Monitoring & Control
Pipeline I/O Data

15. Slow Transients Pipeline
Subtract
Global Sky Model
Visibilities
Residual Visibilities
Gridding
FFT
Find Sources
Image
Transient Sources
Kernels
SKA Radio-telescope
First Level Processing Centre (ZA)

16. Extra thoughts
Recursion is hard to grasp because in to understand recursion one first needs to understand recursion.
There is a fundamental difference between "as fast as possible" and "in real-time".
A multicore machine will not execute a SW any faster unless the SW is programmed to use multi-core.
The challenge is not in dealing with 2 or 2,000,000 cores, it is in dealing with one or more than one.
A parallel computing framework will not break the problem for you, it will only allow you do that.