1. Analyzing Yellowstone’s Network with a Raspberry Pi Cluster Lauren Patterson

2. Using a low cost Raspberry Pi cluster to find the interconnect path between two nodes on Yellowstone in order to analyze the performance of jobs. Objective of the Project

4. Assembling the Raspberry Pi cluster

6. Yellowstone Interconnect Credit: Siddhartha Ghosh

7. Files Used job1_nodes.txt – Gives the job ID and nodes used ibnetdiscover.log (Discover File) – Lists connections between switches LFTS.txt – Routing table for each switch

8. What is Hadoop? HDFS MapReduce

9. HDFS  

10. Input Data Map Phase Shuffle phase Reduce phase Outpu t Data MapReduce

11. Pig Apache Pig Pig Latin Grunt

12. Pig Latin Script Created Pig Latin Script to find the path between two nodes in Yellowstone http://www.edureka.in/blog/pig-programming-create-your-first-apache-pig-script/

13. JOIN Operations in PIG Default, Inner Join returns intersection of A and B Set B Set A A B U Full, Right and Left Outer Joins return A and B with different parts nulled out (white) Full Right Left Join

14. Path Finder Code Flow

15. Results ±3 ±82 ±19 ±15 ±3±4

16. Python Single Path Python Parallel Python – Mpi4py 1.3.1

17. ±0.02 ±0.07 ±0.006 ±0.11 ±0.004 ±0.11

18. ±18 ±4 ±20 ±2 ±7±4 ±1 ±2 ±0.5

19. What Do All Of These Have In Common? Raspberry Pi Hadoop Pig Python

20. Acknowledgments Richard Loft Karina Hauser Stephanie Barr Bruce Chittenden Amogh Simha Raghu Raj Prasanna Kumar

21. Questions?