1. Copyright © 1998 Alex Chaffee Building a Robust Multithreaded Server in Java Alexander Day Chaffee jGuru Training by the MageLang Institute alex@jguru.com

2. Copyright © 1998 Alex Chaffee Abstract Mark Andreesen says, ”Client-side Java is dead." Fortunately, server-side Java is alive and kicking! This session will quickly introduce you to the concepts of threading and networking in Java. We then build a robust, multithreaded server (using a simple chat as our application). We examine several different threading models, specifically comparing the one-thread-per-connection model with the queuing model.

3. Copyright © 1998 Alex Chaffee Introduction jGuru Training by the Magelang Institute –http://www.jguru.com –Java Training and Consulting Alex Chaffee –Creator of Gamelan –Cool Java Dude

4. Copyright © 1998 Alex Chaffee Background Java language basics Java networking Java threading

5. Copyright © 1998 Alex Chaffee Overview Start slow, end up really really fast Part I: Networking and I/O in Java Part II: Threads and concurrency Part III: Multithreaded server design Part IV: Implementations –Design decisions –Object models –Problems and solutions

6. Copyright © 1998 Alex Chaffee Part I: Networking and I/O in Java

7. Copyright © 1998 Alex Chaffee Java and Networking Built into language One of the 11 buzzwords Network classloader Java.Net API Based on TCP/IP, the internet protocol

8. Copyright © 1998 Alex Chaffee Client-server Difference between client and server is semantic It’s all just peers talking to each other Protocol - roles, vocabulary, rules for communication

9. Copyright © 1998 Alex Chaffee TCP/IP: The Internet Protocol Physical Network Transport Layer (TCP, UDP) Internet Layer (IP) Application Layer (HTTP, FTP, SMTP)

10. Copyright © 1998 Alex Chaffee Sockets and Ports Port: a meeting place on a host –One service per port –1-1023 = well-known services –1024+ = experimental services, temporary Socket: a two-way connection

11. Copyright © 1998 Alex Chaffee Client Sockets and Ports (Diagram) port 13 port 80 Time Service Web Service Socket Server Socket

12. Copyright © 1997 Alex Chaffee The Socket Class Socket(String host, int port) InputStream getInputStream() OutputStream getOutputStream() void close()

13. Copyright © 1997 Alex Chaffee Socket Code Socket s = new Socket(“www.sigs.com”, 80); Simple, huh? Creates the connection We still need to get the data there and back again

14. Copyright © 1997 Alex Chaffee Streams A stream is a sequence of bytes I/O from disk, network, memory, etc. is handled in exactly the same way

15. Copyright © 1997 Alex Chaffee InputStream and OutputStream InputStream: abstract int read() OutputStream: abstract void write(int b) Common: abstract void close()

16. Copyright © 1997 Alex Chaffee Filter Streams Like photographic filters Change the input (or output) on the way through E.g.: BufferedInputStream, PrintStream, LineNumberInputStream

17. Copyright © 1997 Alex Chaffee Filter Streams (Diagram) Abc FileInputStreamAllCapsInputStream Abc ABC NoVowelInputStream BC DataInputStream readLine() “BC” DataInputStream in = new DataInputStream( new NoVowelInputStream( new AllCapsInputStream( new FileInputStream(“input.txt”)))); String line = in.readLine();

18. Copyright © 1997 Alex Chaffee PrintStream print() –outputs an object, primitive, or String println() –same, but adds a newline System.out is a PrintStream PrintStream ps = new PrintStream(outputstream)

19. Copyright © 1997 Alex Chaffee DataInputStream int readInt() type readType() String readLine() DataInputStream in = new DataInputStream(inputstream)

20. Copyright © 1997 Alex Chaffee Browser.java (source) import java.net.*; import java.io.*; public class Browser { public static void main(String[] args) { String host = "www.stinky.com"; int port = 80; String file = "/index.html";

21. Copyright © 1997 Alex Chaffee Browser.java (source) Socket s = new Socket(host, port); InputStream strIn = s.getInputStream(); OutputStream strOut = s.getOutputStream(); PrintStream out = new PrintStream(strOut); DataInputStream in = new DataInputStream(strIn);

22. Copyright © 1997 Alex Chaffee Browser.java (source) out.println("GET " + file + " HTTP/1.0"); out.println(); String line; while ((line = in.readLine()) != null) { System.out.println(line); } in.close(); out.close();

23. Copyright © 1998 Alex Chaffee Java Servers A server listens on a port and accepts connections Must be applications (or signed applets) Only one service per port for the entire host machine –Java throws an exception if you try to open more than one

24. Copyright © 1997 Alex Chaffee ServerSocket ServerSocket ss = new ServerSocket(1234); Socket socket = ss.accept(); accept() returns only after a client makes the connection

25. Copyright © 1997 Alex Chaffee ServerSocket Now we have a socket, so we can call... InputStream in = socket.getInputStream(); OutputStream out = socket.getOutputStream();

26. Copyright © 1997 Alex Chaffee EchoServer.java ServerSocket ss = new ServerSocket(1234); Socket s = ss.accept(); InputStream in = s.getInputStream(); OutputStream out = s.getOutputStream();

27. Copyright © 1997 Alex Chaffee EchoServer.java int ch; while ((ch = in.read()) != -1) { out.write(ch); } out.close(); in.close();

28. Copyright © 1998 Alex Chaffee EchoServer Demo Connect Type some stuff Disconnect Problem: Multiple simultaneous connections –Connections are made, but server doesn’t respond –Even if we loop, can still handle only one at a time

29. Copyright © 1998 Alex Chaffee Multiple Connections Solution: –Java threads!

30. Copyright © 1998 Alex Chaffee Part II: Threads

31. Copyright © 1998 Alex Chaffee Java Threads A thread is not an object A thread is a flow of control A thread is a series of executed statements A thread is a nested sequence of method calls

32. Copyright © 1998 Alex Chaffee The Thread Object A thread is not an object A Thread is an object void start() –Creates a new thread and makes it runnable void run() –The new thread begins its life inside this method

33. Copyright © 1998 Alex Chaffee Thread Creation Diagram Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Object AObject BThread (extends Thread)

34. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); Thread Creation Diagram Object A BThread() { } void start() { // create thread } void run() { doSomething(); } Object BThread (extends Thread)

35. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

36. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

37. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

38. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

39. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

40. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

41. Copyright © 1998 Alex Chaffee Thread t = new BThread(); t.start(); doMoreStuff(); BThread() { } void start() { // create thread } void run() { doSomething(); } Thread Creation Diagram Object AObject BThread (extends Thread)

42. Copyright © 1998 Alex Chaffee Runnable Interface A helper to the thread object The Thread object’s run() method calls the Runnable object’s run() method Allows threads to run inside any object, regardless of inheritance

43. Copyright © 1998 Alex Chaffee Runnable Example Talker talker = new Talker(); Thread t = new Thread(talker); t.start(); --- class Talker implements Runnable { public void run() { while (true) { System.out.println(“yakitty yak”); }

44. Copyright © 1998 Alex Chaffee Blocking Threads When reading from a stream, if input is not available, the thread will block Thread is suspended (“blocked”) until I/O is available Allows other threads to automatically activate When I/O available, thread wakes back up again –Becomes “runnable” –Not to be confused with the Runnable interface

45. Copyright © 1998 Alex Chaffee Thread Scheduling In general, the runnable thread with the highest priority is active (running) Java is priority-preemptive –If a high-priority thread wakes up, and a low-priority thread is running –Then the high-priority thread gets to run immediately Allows on-demand processing –Efficient use of CPU

46. Copyright © 1998 Alex Chaffee Thread Starvation If a high priority thread never blocks Then all other threads will starve Must be clever about thread priority

47. Copyright © 1998 Alex Chaffee Thread Priorities: General Strategies Threads that have more to do should get lower priority Counterintuitive Cut to head of line for short tasks Give your I/O-bound threads high priority –Wake up, immediately process data, go back to waiting for I/O

48. Copyright © 1998 Alex Chaffee Race Conditions Two threads are simultaneously modifying a single object Both threads “race” to store their value In the end, the last one there “wins the race” (Actually, both lose)

49. Copyright © 1998 Alex Chaffee Race Condition Example class Account { int balance; public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; } Looks good, right?

50. Copyright © 1998 Alex Chaffee Race Condition Example class Account { int balance; public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; } What if we are swapped out right here?

51. Copyright © 1998 Alex Chaffee Thread Synchronization Language keyword: synchronized Takes out a monitor lock on an object –Exclusive lock for that thread If lock is currently unavailable, thread will block

52. Copyright © 1998 Alex Chaffee Thread Synchronization Protects access to code, not to data –Make data members private –Synchronize accessor methods Puts a “force field” around the locked object so no other threads can enter Actually, it only blocks access to other synchronizing threads A rogue thread can "sneak in" and modify a variable it has access to

53. Copyright © 1998 Alex Chaffee Synchronization Example class Account { private int balance; synchronized public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; } synchronized public void withdraw(int val) { int newBal; newBal = balance - val; balance = newBal; }

54. Copyright © 1998 Alex Chaffee Thread Deadlock If two threads are competing for more than one lock Example: bank transfer between two accounts –Thread A has a lock on account 1 and wants to lock account 2 –Thread B has a lock on account 2 and wants to lock account 1

55. Copyright © 1998 Alex Chaffee Avoiding Deadlock No universal solution Ordered lock acquisition Encapsulation (“forcing directionality”) Spawn new threads Check and back off Timeout Minimize or remove synchronization

56. Copyright © 1998 Alex Chaffee Wait and Notify Allows two threads to cooperate Based on a single shared lock object

57. Copyright © 1998 Alex Chaffee Wait and Notify: Code Consumer: synchronized (lock) { while (!resourceAvailable()) { lock.wait(); } consumeResource(); }

58. Copyright © 1998 Alex Chaffee Wait and Notify: Code Producer: produceResource(); synchronized (lock) { lock.notifyAll(); }

59. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

60. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

61. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

62. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

63. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

64. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

65. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

66. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

67. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

68. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

69. Copyright © 1998 Alex Chaffee Wait/Notify Sequence Lock Object Consumer Thread Producer Thread 1. synchronized(lock){ 2. lock.wait(); 3. produceResource() 4. synchronized(lock) { 5. lock.notify(); 6.} 7. Reacquire lock 8. Return from wait() 9. consumeResource(); 10. }

70. Copyright © 1998 Alex Chaffee Wait/Notify: Details Often the lock object is the resource itself Sometimes the lock object is the producer thread itself

71. Copyright © 1998 Alex Chaffee Wait/Notify: Details Must loop on wait(), in case another thread grabs the resource... –After you are notified –Before you acquire the lock and return from wait() Use lock.notifyAll() if there may be more than one waiting thread

72. Copyright © 1998 Alex Chaffee Wait/Notify Example: Blocking Queue class BlockingQueue extends Queue { public synchronized Object remove() { while (isEmpty()) { wait(); // really this.wait() } return super.remove(); } public synchronized void add(Object o) { super.add(o); notifyAll(); // this.notifyAll() }

73. Copyright © 1998 Alex Chaffee Part III: Multithreaded Server Design

74. Copyright © 1998 Alex Chaffee Different Models of Information Flow Pull model Push model: One thread per client Buffer model: One thread per task Single-thread model

75. Copyright © 1998 Alex Chaffee Pull-based Flow Consumer asks producer for new information Demand-driven problems –Examples Not appropriate for chat/message- passing server

76. Copyright © 1998 Alex Chaffee Push-based Flow Producer triggers sequence of events One thread per client Receives message, delivers it, waits

77. Copyright © 1998 Alex Chaffee Push-based: Pro Simple Efficient use of CPU –Events occur only when message available - no polling –No time wasted passing information from one stage to the next

78. Copyright © 1998 Alex Chaffee Push-based: Con Much thread swapping Slow receivers can stall upcoming messages from sender

79. Copyright © 1998 Alex Chaffee Buffer Model Independent threads Communicate via buffers Producer puts message in buffer, consumer takes message out

80. Copyright © 1998 Alex Chaffee Buffer Model: Pro Can separate tasks into separate threads Optimize and prioritize per task –e.g. Processing existing messages is more important than accepting new connections Small number of threads means fewer chances for deadlock –Easier to model the system and avoid any chance of deadlock

81. Copyright © 1998 Alex Chaffee Buffer Model: Con More complex code Must deal with deadlock and starvation

82. Copyright © 1998 Alex Chaffee The One-Thread Model All work done by a single thread Finishes processing one request before going to the next

83. Copyright © 1998 Alex Chaffee The One-Thread Model (Cont.) Pro: –Suitable for low-traffic scenarios –Quick to implement –Avoids any possibility of deadlock or race conditions Con –Unsuitable for high-traffic server –Hard to implement things like timers or callbacks

84. Copyright © 1998 Alex Chaffee Part IV: Implementations

85. Copyright © 1998 Alex Chaffee Implementation 1: Push Model Chat Server Listens on a port Accepts connections Spawns threads, one per connection Receives a message from one client Dispatches it to all clients

86. Copyright © 1998 Alex Chaffee Chat Server Objects ChatServer –Main ClientHandler –One per connection –Threaded Dispatcher –Vector

87. Copyright © 1998 Alex Chaffee Chat Server Objects (Diagram) ChatServer ClientHandler Dispatcher Socket

88. Copyright © 1997 Alex Chaffee ClientHandler Stores information for a single client Socket incoming; int id; Dispatcher dispatcher;

89. Copyright © 1997 Alex Chaffee ClientHandler.run() generate filter streams read a line send it to the dispatcher loop

90. Copyright © 1997 Alex Chaffee ClientHandler.send() sends a single message to the client called by the dispatcher uses the socket

91. Copyright © 1997 Alex Chaffee Dispatcher Keeps a list of all client handlers Uses java.util.Vector –synchronized list structure

92. Copyright © 1997 Alex Chaffee Dispatcher.dispatch() Sends a message to all clients Accepts a String and an identifying integer Enumerates down all client handlers and calls each one’s send() method

93. Copyright © 1997 Alex Chaffee ChatServer One method: main() Creates a new ServerSocket Creates a new Dispatcher Listens for connections When connection received, creates a new ClientHandler and starts a new thread running inside it Adds it to the Dispatcher

94. Copyright © 1998 Alex Chaffee Push model: Pro and Con Pro –Straightforward object, thread models –Efficient CPU usage Threads unblock only when message available Con –Must send message to all clients before receive new one –No upper limit on number of threads May thrash or starve

95. Copyright © 1998 Alex Chaffee Implementation 2: Buffer Model Message Server

96. Copyright © 1998 Alex Chaffee Queuing Model Producer and consumer communicate via a buffer Buffer is a queue First in, first out

97. Copyright © 1998 Alex Chaffee Blocking Queue If a thread tries to remove an item from an empty queue Then it blocks until an item is placed inside Uses wait/notify technique

98. Copyright © 1998 Alex Chaffee JDK 1.2 Collections Set List Map Queue –Built on top of LinkedList

99. Copyright © 1998 Alex Chaffee Basic Object Model Acceptor (thread) Clients (set) Receiver (thread) Incoming (queue) Processor (thread)

100. Copyright © 1998 Alex Chaffee Acceptor Receiver Clients Incoming Processor Thread Data write read write create Basic Object Model (Fig.)

101. Copyright © 1998 Alex Chaffee Acceptor Thread Listens for incoming connections Creates a client object Adds it to clients set

102. Copyright © 1998 Alex Chaffee Clients Set Set of all active clients Synchronized methods –Only one thread at a time can access

103. Copyright © 1998 Alex Chaffee Receiver Thread Walks down clients set For each client, checks if message is (fully) available Reads message and places it on incoming queue Uses non-blocking input streams

104. Copyright © 1998 Alex Chaffee Receiver Thread: Alternative Alternative: one receiver thread per client Each receiver thread blocks until input available Con: –Reach number of threads bottleneck –Scheduling not determined - could starve or do in weird order

105. Copyright © 1998 Alex Chaffee Incoming Queue Queue of messages Synchronized methods –Only one thread at a time can access Blocking queue –If a thread tries to get a message from an empty queue, it blocks until another thread adds a message

106. Copyright © 1998 Alex Chaffee Processor Thread While incoming queue is not empty Pops message off incoming queue Delivers it

107. Copyright © 1998 Alex Chaffee Full Object Model Checker (thread) Problems (list) Auditor (thread)

108. Copyright © 1998 Alex Chaffee Acceptor Receiver Clients Incoming Processor Checker Problems Auditor Thread Data Full Object Model (Fig.)

109. Copyright © 1998 Alex Chaffee Problems List List of clients who have problems Receiver can add a client to problems list if it gets a socket read error Processor can add a client to problems list if it gets a socket write error Checker thread periodically disconnects them all

110. Copyright © 1998 Alex Chaffee Checker Thread Wakes up every N seconds Walks list of clients Move to problems list –If N seconds has passed (“timeout”) –If socket has a problem Walk problems list and actively close each client

111. Copyright © 1998 Alex Chaffee Auditor Thread Every N seconds, wakes up Collects and prints statistics

112. Copyright © 1998 Alex Chaffee Non-Blocking Input Stream Needed for receiver to do its job If a full message is not available, it does NOT block Instead, it returns null Tricky to implement –Must use available(), mark() and reset() cleverly

113. Copyright © 1998 Alex Chaffee Thread Safety Policies All data objects synchronized All data objects independent Therefore no possibility of deadlock in the data objects themselves

114. Copyright © 1998 Alex Chaffee Thread Safety Policies Thread objects must be very careful to avoid deadlock Current model: no multi-object transactions Example –Checker does clients.remove(c) then problems.add(c) –NOT synchronized(clients) { synchronized(problems) { clients.remove(c); problems.add(c); }} –It's OK if it's interrupted between remove and add

115. Copyright © 1998 Alex Chaffee Priority: Why? In low-volume server, all threads are normally blocked When something needs to happen, it happens That's the beauty of threads! So priorities are irrelevant –If you don’t know what you’re doing, don’t mess with priorities

116. Copyright © 1998 Alex Chaffee Priority: Why? In high-volume server, many threads always have something to do Important to do things in the right order Deal with starvation issue

117. Copyright © 1998 Alex Chaffee Priorities: General Strategies Threads that have more to do get lower priority Counterintuitive Cut to head of line for short tasks

118. Copyright © 1998 Alex Chaffee Priorities: Queuing Strategies Most important: keep incoming queue empty So we don't get bogged down Minimize latency for received messages

119. Copyright © 1998 Alex Chaffee Priorities: Decisions Don't want to add new messages if we're already busy –Therefore processor > receiver Don't want to accept new connections if we're already busy –Therefore processor > acceptor Implication: –Receiver and acceptor may starve? –No: queue will eventually get cleared

120. Copyright © 1998 Alex Chaffee Priorities: Decisions Want to be able to accept new connections, even if a currently connected client has something to say –Therefore acceptor > receiver

121. Copyright © 1998 Alex Chaffee Priorities: Decisions Checker's job is brief but important Needs to wake up, check all clients, and go back to sleep Therefore checker > processor

122. Copyright © 1998 Alex Chaffee Priorities: Decisions Auditor's job is brief but important Need to get accurate statistics -- timing is crucial Can't wait around for other threads to finish Therefore auditor > checker

123. Copyright © 1998 Alex Chaffee Priorities: Conclusion Auditor - 10 Checker - 8 Processor - 6 Acceptor - 4 Receiver - 2 Goes by twos to accommodate Windows threading model

124. Copyright © 1998 Alex Chaffee Processor Thread Behavior Processor thread does: 1. Pop message off incoming queue 2. Deliver it 3. Loop Very independent

125. Copyright © 1998 Alex Chaffee Processor Thread Blocking Step 1 (pop message) may block If queue is empty This is good Allows other threads to wake up and fill queue

126. Copyright © 1998 Alex Chaffee Processor Thread Blocking Step 2 (delivery) may block –If network is busy –If OS socket buffer is full Bad, because queue is still full Increases latency (delivery time) for already-queued messages –While new messages are arriving, old messages should be delivered instead

127. Copyright © 1998 Alex Chaffee Multiple Processor Threads Solution: have many processor threads working simultaneously If one blocks, another picks up the next message

128. Copyright © 1998 Alex Chaffee Multiple Processors (Fig.) Acceptor Receiver Clients Incoming Processor Thread Data write read write create Processor

129. Copyright © 1998 Alex Chaffee How Many Threads? 1. Constant, tuned per application 2. Dynamic a/k/a thread pooling

130. Copyright © 1998 Alex Chaffee Thread Pooling Thread pool keeps a certain number of threads alive Other threads ask the thread pool to perform a task If an existing thread is available, that thread performs the task Else a new thread is created, or the task blocks until one is available –(depending on policy)

131. Copyright © 1998 Alex Chaffee Thread Pooling (Cont.) Removes overhead of creating threads Allows modular task strategy

132. Copyright © 1998 Alex Chaffee Information Flow Decoupling threads with buffers is a good idea –Maximizes concurrency –Smooths out bursty differences in rate Can cause problems with information flow –Producers outpace Consumers –Too many threads

133. Copyright © 1998 Alex Chaffee Limiting Flow Bounded Buffers –Stalls producers if they outpace consumers Bounded Thread Pools –Prevents thrashing Rate-adjustment / back-pressure –Consumer asks Producer to slow down

134. Copyright © 1998 Alex Chaffee Another Problem One message, multiple recipients, one processor thread If one recipient is in Finland, it will stall the remaining recipients for that message

135. Copyright © 1998 Alex Chaffee Another Problem: Solutions Solution 1: one message queue per client; thread pool applied to these delivery queues Solution 2: split multi-message into several single-recipient messages –Called a Splitter (or Fork or Multicaster) –Also possible: Routers, Mergers, Collectors, Combiners, Conduits -- see Lea’s Concurrent Programming in Java

136. Copyright © 1998 Alex Chaffee Debugging Techniques Make sure to trap all Throwables in your Thread object’s run() method –Otherwise a stray OutOfMemoryError or NullPointerException will kill your thread and it will look like deadlock Good logging procedure is vital –I use a global log method that outputs the name of the thread and the time in addition to the message –Name your threads at creation time

137. Copyright © 1998 Alex Chaffee Thread Dump Control-backslash in Solaris Control-break in Windows –Unfortunately, it scrolls –There’s no way to redirect standard error –Oops

138. Copyright © 1998 Alex Chaffee Thread Watcher Data object All tasks (threads) have a pointer to the master ThreadWatcher Put debug code in your tasks to inform the ThreadWatcher when they change state watcher.setIdle(); Message m = incoming.remove(); watcher.set(“Processing message “ + m);

139. Copyright © 1998 Alex Chaffee Thread Watcher (cont.) ThreadWatcher has its own thread –Periodically prints status of threads –Prints an error message if one thread is blocked for too long Alternative to fancy GUI thread debugger –Symantec Café –Compuware DevPartner –et al.

140. Copyright © 1998 Alex Chaffee Bots PingBot –periodically sends a message to itself, measures the latency QuoteBot –sends random Zippy quotes to mimic a chat room

141. Copyright © 1998 Alex Chaffee Measurements Number of bots before sharp latency increase Average latency of single PingBot for N ZippyBots

142. Copyright © 1998 Alex Chaffee Test Architecture Need at least three machines –Server –Zippy Bot Host –Ping Bot Host So they don’t interfere with one another Should also have some clients on a far away machine –If your server will be running on the Internet

143. Copyright © 1998 Alex Chaffee Conclusion

144. Copyright © 1998 Alex Chaffee Thanks to Eric Malmstrom Carl Muckenhoupt Gerry Seidman Greg Travis For helping prepare this talk

145. Copyright © 1998 Alex Chaffee Where to Get More Information: Network Programming Cornell & Horstmann, Core Java (Sunsoft press) Harold, Java Network Programming (O’Reilly) Hughes et al., Java Network Programming, (Manning, an imprint of Prentice-Hall)

146. Copyright © 1998 Alex Chaffee Where to Get More Information : Multithreading –Cornell & Horstmann, Core Java (sunsoft press) –Oaks and Wong, Java Threads (o’reilly) –Lea, Concurrent Programming in Java (Addison Wesley) –Travis, Using thread pools to increase threading efficiency, developer.com, http://www.Developer.Com/journal/techworkshop/0 60498_thread.html –Travis, Working with the blocking queue, http://www.Developer.Com/journal/techworkshop/0 91098_blockq.html

147. Copyright © 1998 Alex Chaffee Where to Get More Information: Web sites –http://www.jGuru.com/ (Java Training) –http://www.Developer.com/ (Gamelan) –http://www.Javaworld.com/ (magazine) –http://www.Purpletech.com/ (author’s site)