1. Big Data EUDAT 2012 – Training Day Adam Carter, EPCC EUDAT Training Task Leader

2. Data

3. Collect Organise ProcessInterpret Store Share Knowledge Paper

4. Dat a Collect Organise Process Interpret Store Share Knowledge Paper iRODS PIDs Replication SimpleStore Data Staging

5. Dat a Collect Organise Process Interpret Store Share Knowledge Paper iRODS PIDs Replication SimpleStore Data Staging Workflows Dataflows

6. Data Bytes km I/O ≫ Compute

7. Dat a Data

8. BIG DATA - PROBLEMS

9. 1. Many Bytes Data takes a lot of space to store Data takes a long time to move Data takes a long time to process Data takes a long time to organise Data too large to be manually interpreted

10. 2. Distributed Data You have to bring (some of) the data together to process it You have to bring (some of) the data together to interpret it Often implies: –Different ownership –Different formats –Different quality

11. 3. Data Intensive Actually a feature of the application and the data size (and, arguably, the hardware) Difficult to compute efficiently Often implies big (in bytes) Different definitions: –A lot of data on which to compute –A lot more I/O required than compute

12. BIG DATA - REWARDS Everyone can now do big science…

13. Datascopes Data Intensive Research TheoryExperiment Simulation Datascope “Observe” a large amount of data and look for patterns

14. Interdisciplinary Research Using others’ data brings more to your problem than you could have collected yourself May already have been pre-processed, possibly with input from experts in another field

15. More data Better statistics More likely to be data collected that can be used to test a theory

16. Of course, bigger doesn’t mean better As always, you should use the right tools for the job

17. BIG DATA – (PARTIAL) SOLUTIONS

18. Data Infrastructure Share your data more easily Use others’ data more easily Store more data Store data more reliably Move data more quickly Process data more efficiently Interpret data more easily

19. Move the Compute to the Data Virtualisation Queries to relational databases Distributed query processing If you’ve got a large amount of “baseline” data that is accessed repeatedly –put compute power close to the data –consider mechanisms to allow others to perform arbitrary compute against your data at your site

20. Data Intensive Computing Build and use computers built for data-intensive research –Slower, low-power processors –High performance I/O systems –“Amdahl-Balanced” –In some cases these machines might (locally) move the compute to the data

21. Organise Your Data Metadata PIDs Catalogues, Indexing Make your data addressable –Allows for caching, or using nearby replicas –Allows for retrieving only necessary data Consider why you’re storing your data –Think big!

22. Conclusions We can do a lot more with our data than we currently do A problem shared is a problem halved –Use infrastructure and services provided by others Consider your data as important as the paper –Infrastructure likely to help with this –Also need change of mindset re attribution, etc. Big data offers opportunities –Think about what you could do with data or techniques from other disciplines Think big!