1. NETWORK PROGRAMMING INTRO TO DISTRIBUTED SYSTEMS FALL 2015 L3-socket Dongsu Han Some material taken from publicly available lecture slides including Srini Seshan’s and David Anderson’s

2. Today’s Lecture  Lecture  Transport protocols: UDP and TCP  Version control using SVN  TA-led programming session  Editor, socket programming  Debugging with gdb  Announcement  Programming Assignment 1 is out. (Due in two weeks-- 추석 )

3. Review of Previous Lectures  What is a distributed system? (L1)  Definition  Example  What are some of the important decisions made in the Internet architecture? (L2)  Narrow waist: Network provides minimal functionality (best-effort) Why?: to connect existing networks that are heterogeneous  Stateless architecture: no per-flow state inside network Why? For survivability

4. Principles  P1: IP over all network (narrow waist)  P2: Layering  Principle 3: Fate Sharing  Principle 4: Soft-state  Principle 5: End-to-End Argument  Next lecture  Principle 6: network layer provides one simple service: best effort datagram (packet) delivery

5. Principle 4: Soft-state 5  Soft-state (as oppose to hard-state)  Announce state  Refresh state  Timeout state  Penalty for timeout – poor performance  Robust way to identify communication flows (firewall)  Possible mechanism to provide non-best effort service  Helps survivability

6. Goal 2: Types of Service 6  Principle 6: network layer provides one simple service: best effort datagram (packet) delivery  All packets are treated the same  Network layer (IP) provides “best effort” service.  Relatively simple core network elements  Other functions are implemented at the end-host at the transport layer (TCP/UDP)  Building block from which other services (such as reliable data stream) can be built  Contributes to scalability of network  Issues: No QoS support assumed from below  Hard to implement without network support  QoS is an ongoing debate…

7. Summary  Successes: IP on everything!  Drawbacks… but perhaps they’re totally worth it in the context of the original Internet. Might not have worked without them! “This set of goals might seem to be nothing more than a checklist of all the desirable network features. It is important to understand that these goals are in order of importance, and an entirely different network architecture would result if the order were changed.” 7

8. 8 Transport Protocols  Lowest level end-to-end protocol.  Header generated by sender is interpreted only by the destination  Routers view transport header as part of the payload; they are not supposed to see the transport header. 7 7 6 6 5 5 7 7 6 6 5 5 Transport IP Datalink Physical Transport IP Datalink Physical IP router 2 2 2 2 1 1 1 1

9. What does the transport layer do?  No per-flow (connection) state in the network  Transport layer maintains the connection state. Transport layer identifies an end-to-end connection.  How does the transport layer identify an end-point (application)?

10. De-multiplexing: Port numbers  Network layer gets you to the host.  A port identifies the sending or receiving socket (application).  A socket is one endpoint of a two-way communication between two programs running on the network. A socket is bound to a port number so that the transport layer can identify the application that data is destined to be sent.  Identifying two end-points: An endpoint =  Source endpoint =  destination endponit =

11. 11 Protocol and Port Demultiplexing  Multiple choices at each layer  Applications open a socket (identified by the port number) Port 80Port 22 Port 80 TCPUDP IP NET 1 NET 2 NET n … TCP/UDPIP Port Number = 80 Network Protocol Field = TCP Type Field =IP

12. 12 Server and Client TCP/UDP IP Ethernet Adapter Server TCP/UDP IP Ethernet Adapter Clients Server and Client exchange messages over the network through a common Socket API Socket API hardware kernel space user space ports

13. Transport protocols  UDP  Provides only integrity and de-multiplexing  TCP adds…

14. 14 UDP: User Datagram Protocol [RFC 768]  “bare bones” Internet transport protocol, “Best effort” service  UDP segments may be:  Lost, duplicated (bug, spurious retransmission a t L2)  Delivered out of order to app  Connectionless:  No handshaking between UDP sender, receiver  Each UDP segment handled independently Why is there a UDP? No connection establishment (which can add delay) Simple: sender, receiver stat e is minimal Small header No congestion control: UDP can blast away as fast as de sired

15. 15 UDP, cont.  Often used for streaming multimedi a apps (e.g., Skype)  Loss tolerant  Rate sensitive  Other UDP uses (why?):  DNS  Reliable transfer over UDP  Must be at application layer  Application-specific error recovery Source port #Dest port # 32 bits Application data (message) UDP segment format Length Checksum Length, in bytes of UDP segment, including header

16. Transport protocols  UDP  Provides only integrity and de-multiplexing  TCP adds…  Connection-oriented  Reliable (error control)  Ordered byte stream (in-order delivery)  Bi-directional byte-stream  Flow and congestion controlled

17. TCP  Protocol implemented entirely at the ends  Fate sharing  Abstraction provided  Connection oriented protocol (hand-shake or connection establishment)  In-order byte-stream  Implemented functionality (next lecture)  Flow control  Reliability (error control)  Reassembly (in-order delivery)  Congestion control

18. /* get the server host entry */ hp = gethostbyname(argv[1]); if (hp == NULL) { perror("gethostbyname"); return -1; } /* create stream socket */ sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); if (sockfd < 0) { perror("socket"); return -1; } /* set the destination address */ daddr.sin_family = AF_INET; memcpy(&daddr.sin_addr.s_addr, hp->h_addr, hp->h_length); daddr.sin_port = htons(PORT); TCP Client Example 18

19. /* connect to the destination */ if (connect(sockfd, (struct sockaddr *)&daddr, sizeof(struct sockaddr_in))) { close(sockfd); perror("connect"); return -1; } inet_ntop(AF_INET, &daddr.sin_addr, s, sizeof(s)); printf("client: connecting to %s\n", s); /* receive message sent from the server */ if ((numbytes = recv(sockfd, buf, MAXDATASIZE-1, 0)) == -1) { perror("recv"); exit(1); } /* print the received message */ buf[numbytes] = '\0'; printf("client: received '%s'\n", buf); close(sockfd); TCP Client Example 19

20. Difference in abstraction  UDP  Recv(len) call gives you a packet (the payload part)  TCP  Connection establishment (hand-shake)  Will see in the TA led session.  Recv(len) call give you a sequence of bytes UDP segment (packet) Byte stream UDP segment (packet) len Incoming stream Application Recv() len Byte stream Application send()

21. TCP– What goes on underneath Byte stream Packet (1~7) Packet (8~10) Receive buffer Packet (9~11) Packet (8~10) I need seq# 8.

22. 22 User Datagram Protocol(UDP): An Analogy Example UDP applications Multimedia, voice over IP UDP Single socket to receive messages No guarantee of delivery Not necessarily in-order delivery Datagram – independent packets Must address each packet Postal Mail Single mailbox to receive letters Unreliable Not necessarily in-order delivery Letters sent independently Must address each reply

23. Transmission Control Protocol (TCP): An Analogy TCP  Reliable – guarantee delivery  Byte stream – in-order delivery  Connection-oriented – single socket per connection  Setup connection followed by data transfer Telephone Call Guaranteed delivery In-order delivery Connection-oriented Setup connection followed by conversation Example TCP applications Web, Email, Telnet

24. REVISION CONTROL & MAKE

25. Overview  Revision control systems  Motivation  Features  Subversion primer  Make  Simple gcc  Make basics and variables  Testing  Useful Unix commands

26. Dealing with large codebases  Complications  Code synchronization  Backups  Concurrent modifications  Solution: Revision Control System (RCS)  Store all of your code in a repository on a server  Metadata to track changes, revisions, etc…  Also useful for writing books, papers, etc…

27. Basic RCS features (1/2)  Infinite undo  go back to previous revisions  Automatic backups  all your code ever, forever saved  Changes tracking  see differences between revisions

28. Basic RCS features (2/2)  Concurrency  Multiple people working at the same time  Snapshots  Save stable versions on the side (i.e. handin)  Branching  Create diverging code paths for experimentation

29. Typical RCS workflow 1. Create repository on RCS server 2. Checkout the repository to your local machine 3. Work locally: create, delete and modify files 4. Update the repository to check for changes other people might ha ve made 5. Resolve any existing conflicts 6. Commit your changes to the repository

30. RCS implementations  Concurrent Versions System (CVS)  Concurrency, versioning of single files  Subversion (SVN)  File and directory moving and renaming, atomic commits  Git  Modern, de-facto standard

31. Concurrent edits (1/2) John David File v1 Both check out file Edits (lines 1-20) File v2 Edits (lines 30-40) commit Update Merged automatically!

32. Concurrent edits (2/2) John David File v1 Both check out file Edits (lines 1-20) File v2 Edits (lines 10-20) commit Update Conflict needing manual inte rvention!

33. Resolving conflicts  When changes can’t be merged automatically  SVN gives you 3 files: file.mine : your file file.rx : the remote file file : original file with marked conflicts  You can  Discard your changes  Discard others’ changes  Go over each conflict and arbitrate as appropriate

34. Interacting with SVN  Command line  Readily available on your lab machines  Graphical tools  Easier to use  Subclipse for Eclipse gives IDE/SVN integration  tortoiseSVN

35. 실습 server  IP: 143.248.141.40, 143.248.141.42~143.248.141.44  ID: ee (e.g., ee20110111)  Password: ee324EE#@$

36. Svn password  20100456 63900 20100667 26834 20100967 48013 20110198 810 20110227 39617 20110319 16411 20110468 57700 20110548 64302 20110968 30860 20120184 19424 20120277 12976 20120508 17278 20120677 40029 20120973 61034  20121092 42821 20130098 42807 20130185 30791 20130234 13448 20130363 14421 20130394 20796 20130451 13008 20130477 53705 20130478 45122 20130505 4805 20130842 742 20140055 20389 20140244 45336 20140544 23793 20153347 5044

37. Command line SVN example (1/2) $ svn co –-username= > https://ina.kaist.ac.kr/svn/test $ cd test $ vim >.txt $ vim >. $ svn add >.txt $ svn commit -m 'adding a text file with my student id‘ $ svn update $ cd../ $ svn cp trunk tags/ >_submission $ svn commit -m 'making a tag of the trunk for submission!‘ $ svn up $ echo -e ” >" >> student.txt $ svn commit –m “added my student id”

38. Command line SVN example (2/2) Revision control lets you note (and then see) what you changed: > svn log student.txt r986 | ntolia | 2006-08-01 17:13:38 -0400 (Tue, 01 Aug 2006) | 6 lines This allows the sp to get rid of chunks early before a transfer is complete. Useful when a file is requested in-order and the file size > mem cache size > svn diff -r 1:2 student.txt Index: file =================================================================== --- file (revision 1) +++ file (revision 2) @@-1,2+1,3@@ This isatestfile -It startedwithtwolines +It nolongerhastwolines +it hasthree

39. General SVN tips  Update, make, test, only then commit  Merge often (svn up)  Comment commits  Avoid commit races  Modular design avoids conflicts

40. Know more  subversion.tigris.org for SVN software & info  svnbook.red-bean.com for SVN book

41. Makefile  You are on your own.

42. Make  Utility for executable building automation  Saves you time and frustration  Helps you test more and better

43. Simple gcc If we have files: prog.c: The main program file lib.c: Library.c file lib.h: Library header file % gcc -c prog.c -o prog.o % gcc -c lib.c -o lib.o % gcc lib.o prog.o -o binary

44. gcc flags Useful flags 1.-g: debugging hook 2.-Wall: show all warnings 3.-Werror: treat warning as errors 4.-O0, -O1, -O2, -O3: optimization level 5.-DDEBUG: macro for DEBUG (#define DEBUG) Avoid using dangerous optimizations that could affe ct correctness

45. More gcc % gcc -g -Wall -Werror -c prog.c -o prog.o % gcc -g -Wall -Werror -c lib.c -o lib.o % gcc -g -Wall -Werror lib.o prog.o -o binary This gets boring, fast!

46. Makefile basics Build targets target: dependency1 dependency2... unix command (start line with TAB)‏ unix command

47. binary: lib.o prog.o gcc -g -Wall lib.o prog.o -o binary lib.o: lib.c gcc -g -Wall -c lib.c -o lib.o prog.o: prog.c gcc -g -Wall -c prog.c -o prog.o clean: rm *.o binary Makefile example

48. Makefile variables (1/7) Variables CC = gcc CFLAGS = -g -Wall -Werror OUTPUT = binary

49. binary: lib.o prog.o gcc -g -Wall lib.o prog.o -o binary lib.o: lib.c gcc -g -Wall -c lib.c -o lib.o prog.o: prog.c gcc -g -Wall -c prog.c -o prog.o clean: rm *.o binary Makefile variables (2/7)

50. CC = gcc CFLAGS = -g -Wall OUTPUT = binary $(OUTPUT): lib.o prog.o $(CC) $(CFLAGS) lib.o prog.o -o binary lib.o: lib.c $(CC) $(CFLAGS) -c lib.c -o lib.o prog.o: prog.c $(CC) $(CFLAGS) -c prog.c -o prog.o clean: rm *.o $(OUTPUT)‏ Makefile variables (3/7)

51. CC = gcc CFLAGS = -g -Wall OUTPUT = binary $(OUTPUT): lib.o prog.o $(CC) $(CFLAGS) lib.o prog.o -o binary lib.o: lib.c $(CC) $(CFLAGS) -c lib.c -o lib.o prog.o: prog.c $(CC) $(CFLAGS) -c prog.c -o prog.o clean: rm *.o $(OUTPUT)‏ Makefile variables (4/7)

52. CC = gcc CFLAGS = -g -Wall OUTPUT = binary OBJFILES = lib.o prog.o $(OUTPUT): $(OBJFILES)‏ $(CC) $(CFLAGS) $(OBJFILES) -o binary lib.o: lib.c $(CC) $(CFLAGS) -c lib.c -o lib.o prog.o: prog.c $(CC) $(CFLAGS) -c prog.c -o prog.o clean: rm *.o $(OUTPUT)‏ Makefile variables (5/7)

53. CC = gcc CFLAGS = -g -Wall OUTPUT = binary OBJFILES = lib.o prog.o $(OUTPUT): $(OBJFILES)‏ $(CC) $(CFLAGS) $(OBJFILES) -o binary lib.o: lib.c $(CC) $(CFLAGS) -c lib.c -o lib.o prog.o: prog.c $(CC) $(CFLAGS) -c prog.c -o prog.o clean: rm *.o $(OUTPUT)‏ Makefile variables (6/7)

54. CC = gcc CFLAGS = -g -Wall OUTPUT = binary OBJFILES = lib.o prog.o $(OUTPUT): $(OBJFILES)‏ $(CC) $(CFLAGS) $(OBJFILES) -o binary %.o: %.c # $<: dependency (%.c)‏ # $@: target (%.o)‏ $(CC) $(CFLAGS) -c $< -o $@ clean: rm *.o $(OUTPUT)‏ Makefile variables (7/7)

55. Simple Test Script (1/2) %./server 6667 & % cat testfile.01 |./testscript.py % cat testfile.02 |./testscript.py % killall -9 server

56. Simple Test Script (2/2) #/bin/sh echo “Starting server on port 6667.”./server 6667 & SERVERPID = $! echo “Running test files.” cat testfile.01 |./testscript.py cat testfile.02 |./testscript.py echo “Killing server process.” kill $(SERVERPID)‏

57. CC = gcc CFLAGS = -g -Wall OUTPUT = binary OBJFILES = lib.o prog.o all: $(OUTPUT)‏ $(OUTPUT): $(OBJFILES)‏ $(CC) $(CFLAGS) $(OBJFILES) -o binary %.o: %.c # $<: dependencies (%.c)‏ # $@: target (%.o)‏ $(CC) $(CFLAGS) -c $< -o $@ clean: rm *.o $(OUTPUT)‏ Augmenting the Makefile for testing (1/2)

58. CC = gcc CFLAGS = -g -Wall OUTPUT = binary OBJFILES = lib.o prog.o all: $(OUTPUT) test $(OUTPUT): $(OBJFILES)‏ $(CC) $(CFLAGS) $(OBJFILES) -o binary %.o: %.c # $<: dependencies (%.c)‏ # $@: target (%.o)‏ $(CC) $(CFLAGS) -c $< -o $@ test: $(OUTPUT)‏ sh./testscript.sh clean: rm *.o $(OUTPUT)‏ Augmenting the Makefile for testing (2/2)

59. Using the Makefile % make % make test % make clean Know more Google –“makefile example” –“makefile template” –“make tutorial” –Chapter 3 of Dave Andersen’s notes

60. Extra: useful Unix commands (1/2) find “func_name” in files % grep -r func_name. replace “bad_func_name” to “good_func_n ame” % sed -e “s/bad_func_name/good_func_name/g”\ prog.c > prog.c.new

61. Extra: useful Unix commands (2/2) find a file named “prog.c” % find -name prog.c download files from the Internet % wget http://address/to/file.tar.gz untar and unzip a file % tar xzvf file.tar.gz

62. Assignment  You have access to svn repository at https://ina.kaist.ac.kr/svn/ https://ina.kaist.ac.kr/svn/<student  However, we will change your password (to some unknown integer).  Luckily, we made a server that gives you a hint.  When your client asks whether the username and the password match, the server can tell you whether the password provided is less than, equal to, or greater than your real password.

63. Assignment I  Protocol is defined in the document. (See course web page). Query Response Server (provided by the instructor) Client  Part I: Assume the network is reliable. (due 9/14)  Part II: The server may drop a packet. (due 9/24)