1. Primers CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook Some content adapted from Dr. Kalpakis’s CMSC 621 slides

2. Agenda Distributed Computing – Evolution of Computing Infrastructure – Networking Infrastructure – Properties of Distributed Systems – Example System Architectures Java Linux

3. EVOLUTION OF COMPUTING INFRASTRUCTURE

4. Mainframe – 50s to 70s Custom hardware Custom low-level specialized code Very expensive solutions

5. Client/Server – 80s to 00s IT-led architectures More portable solutions Scalable solutions based on demand Reign of the Enterprise Data Warehouse

6. Cloud – 00s to Today Consumer-grade infrastructure Growing IaaS and PaaS markets Data revolution Focus on applications and not infrastructure

7. Where does Hadoop fit? A piece of your data infrastructure – Can crunch data for analytics – Can expose data for web applications Exploration of raw data Augments today’s infrastructure IMO, a big toolbox that can do a bit of everything

8. NETWORKING INFRASTRUCTURE

9. Single Server HDD CPU RAM NIC Server Scale Up Scale Out Faster CPUs Bigger Storage More Servers

10. Local-Area Network (LAN) Rack HDD CPU RAM NIC Server HDD CPU RAM NIC Server HDD CPU RAM NIC Server HDD CPU RAM NIC Server Rack HDD CPU RAM NIC Server HDD CPU RAM NIC Server HDD CPU RAM NIC Server HDD CPU RAM NIC Server WAN Gateway

11. Wide Area Network (WAN) London, England Beijing, China New York, NY

12. PROPERTIES OF DISTRIBUTED SYSTEMS

13. Distributed Systems The development of low-cost powerful microprocessors, together with the invention of high speed networks, enable us to construct computer systems by connecting a large number of computers A distributed system is a collection of independent computers that appears to its users as a single coherent system.

14. Properties of Distributed Systems Reliability Scalability Availability Efficiency CAP Theorem

15. Reliability Can the system deliver services in face of several component failures?

16. Scalability Can the system scale to support a growing number of tasks?

17. Availability How much latency is imposed on the system when a failure occurs?

18. Efficiency How efficient is the system, in terms of latency and throughput?

19. CAP Theorem Consistent Available Partition Tolerant Trade-off between Consistency and Availability

20. Stateful vs. Stateless Whether or not a distributed system saves their state on an attached device for recovery

21. EXAMPLE SYSTEM ARCHITECTURES

22. Simple Client/Server

23. Multi-Tiered Client/Server

24. Round-Robin Client/Server

26. Java Object-oriented class-based programming language designed for code reuse and portability Programs compile to bytecode that can run on any Java Virtual Machine (JVM) Memory is managed for you and automatically cleaned up by the JVM’s garbage collector Syntax is similar to C++

27. public class Animal { // Member Variables protected int age = 0; protected String species = null; public Animal() { } public Animal(int a, String s) { setAge(a); setSpecies(s); } public String getSpecies() { return species; } public void setSpecies(String s) { this.species = s; } public int getAge() { return age; } public void setAge(int a) { this.age = a; } public String toString() { return age + " " + species } ; }

28. // Inherits all the public/protected items from Animal public class Human extends Animal { // Additional member Variables private String name = null public Human(String name, int age) { super(age, "Human”); setName(name); } public String getName() { return name; } public void setName(String n) { this.name = n; } public String toString() { return name + " " + super.toString() } ; }

29. // Main class to be executed public class Main { public static void main(String[] args) { Animal a = new Animal(); a.setAge(10); a.setSpecies("Hiphopopotamus”); System.out.println(a); a = new Human("Adam", 85); System.out.println(a); } } 10 Hiphopopotamus Adam 85 Human

30. // Templated class public class Pair { public FIRST first; public SECOND second; public String toString() { return first + ":" + second; } } public class Main { public static void main(String[] args) { Pair p1 = new Pair (); p1.first = 10; p1.second = "Rhymenocerous"; System.out.println(p1); Pair p2 = new Pair (); p2.first = new Human("Adam", 85); p2.second = "Hiphopopotamus"; System.out.println(p2); } } 10:Rhymenocerous Adam 85 Human:Hiphopopotamus

31. 101’d Simply scratched the surface of Java Includes interfaces, abstract classes, lots of libraries for data structures, networking, multi-threading, etc. We will be using Eclipse and Maven in this class

32. Let’s look at Maven sorry

33. LINUX

34. Linux Reference A free and open source operating system In this course, we live in Eclipse and the command line Mastery of 'vi' gets you +4 charisma http://www.ibm.com/developerworks/library/l- lpic1-v3-103-1/ http://www.linuxdevcenter.com/excerpt/LinuxPG_q uickref/linux.pdf

35. References http://webdam.inria.fr/Jorge/html/wdmch15.html Google Images http://www.ibm.com/developerworks/library/l-lpic1- v3-103-1/ http://www.ibm.com/developerworks/library/l-lpic1- v3-103-1/ http://www.linuxdevcenter.com/excerpt/LinuxPG_qu ickref/linux.pdf http://www.linuxdevcenter.com/excerpt/LinuxPG_qu ickref/linux.pdf