Distributed Trading SWE 622 Spring 2009

Group Members Dave Johnson Akhil Pathania Sohrab Rezvan Nick Ton

Overview System Design Experiments and Measurements

Peer to Peer Technology, Asynchronous Messaging System Design Peer to Peer Technology, Asynchronous Messaging

Message Types Registration Ping: heartbeat to check for connectivity Ping Acknowledge Time: used to calculate offset Buy Sell

Trader/Exchange Heartbeat Reason To check if Trader or Exchange goes down Also use to check if Trader or Exchange come back up Benefits Timely notification of when a Trader or Exchange goes down Consequence Ping messages are queued there; is currently no way to flush the queue of backlog messages More network traffic

/** * Sends Ping message to all exchange. This reduces thread overhead. */ private void sendPing() { StringBuffer ping = new StringBuffer(); ping.append(MessageType.PING); ping.append(","); ping.append(System.currentTimeMillis()); ping.append(this.traderIP); ping.append(this.traderPort); ping.append(this.traderName); Enumeration<String> en = connectionMap.keys(); while (en.hasMoreElements()) { ExchangeInfo exInfo = connectionMap.get(en.nextElement()); send(ping.toString(), exInfo.exchangeAddress, Integer .parseInt(exInfo.exchangePort)); } private void sendPingEcho(ExchangePingMessage epm){ ping.append(MessageType.PINGACK); send(ping.toString(), epm.getExchangeIP(), epm.getExchangePort());

Time Between Trade and Other Activity (Sell 21)

In JFrameTraderGUI.java public void actionPerformed(ActionEvent e) { final String request = e.getActionCommand(); . else if(request == IOperationsConsole.__SELL21){ String exchange = _exchange.getSelectedItem().toString(); String product = _product.getText(); String price = _price.getText(); String qt = _qt.getText(); LOGGER.info("<====EXPERIMENT:1 START>"); LOGGER.info("<====EXPERIMENT:1 capture trades 2 - 21 elapsed time>"); long elapsedTime = 0; long time = 0; long curTime = 0; for(int i=1; i<=21;i++){ _trader.sell(exchange, product, price, qt); curTime = System.currentTimeMillis(); elapsedTime = curTime - time; time = curTime; LOGGER.info("<====Experiment:1 Trade:" + i + ":ms since last trade. " + elapsedTime ); } LOGGER.info("<====EXPERIMENT:1 END>");

Exchange: Parse Incoming Messages

public void receive(String message) { String mType = MessageType.getMessageType(message); boolean stale = false; if (mType.equals(MessageType.REGISTRATION)) { RegistrationMessage rm = new RegistrationMessage(message); addRemoteEndpoint(rm.getTraderIP(), Integer.parseInt(rm.getTraderListenPort())); sendRegistrationAck(rm); } else if(mType.equals(MessageType.TRADE)){ TradeMessage tm = new TradeMessage(message); if (tm.getOperation().equals(MessageType.BUY)){ if (productList.contains(tm.getProduct()) == true){ gui.logOperation(tm.getTraderName(), tm.getEchoNumber(), tm.getOperation(), tm.getProduct(), tm.getPrice(), tm.getQt(), stale); sendTradeEcho(tm); }else{ tm.rollback(); //decrements the echo number counter }else if (tm.getOperation().equals(MessageType.SELL)){ if (!productList.contains(tm.getProduct())){ productList.add(tm.getProduct()); }else if (mType.equals(MessageType.PING)){ TraderPingMessage tpm = new TraderPingMessage(message); sendPingEcho(tpm); }else if(mType.equals(MessageType.PINGACK)){

}else if(mType.equals(MessageType.TIME)){ TimeAckMessage ttm = new TimeAckMessage(message); Calendar cal = Calendar.getInstance(); String startTime = Long.toString(cal.getTimeInMillis() + cal.getTimeZone().getOffset(cal.getTimeInMillis())); send(new String(ttm.getOperation() + "," + startTime + "," + (cal.getTimeInMillis() + cal.getTimeZone().getOffset(cal.getTimeInMillis())) + ",” + exchangeName + "," + ttm.getId()), ttm.getTraderIp(), Integer.parseInt(ttm.getTraderPort())); }

Exchange: Sending Error

public void errorSending(String message, String remoteHost, int remotePort, IOException exception) { Calendar cal = Calendar.getInstance(); System.out.println("Error sending " + message); System.out.println("Removing remote host " + remoteHost); removeRemoteEndpoint(remoteHost, remotePort); iTraderCount-=1; bUpdateTraderInfo = true; if(iTraderCount==0) gui.updateTraderInfo(iTraderCount, 0); LOGGER.info(remoteHost + " was removed at :" + (cal.getTimeInMillis() + cal.getTimeZone().getOffset(cal.getTimeInMillis()))); }

Exchange-Trader Offset Calculation The calculation is based on the method described in “Distributed Systems: Principles & Paradigms” by Andrew S. Tanenbaum. Figure 1, shows a similar Exchange-Trader arrangement, where: T1 is the time when the request leaves the Trader. T2 is the time when Exchange receives the request. T3 is the time when Exchange sends the response back to the Trader. T4 is the time when Trader receives Exchange response.

Figure 2 Figure 1

Prototype Requirements Traders receive information about all correctly processed orders, and only about those. All traders should display the orders processed by each exchange in the exact same (processing) order, regardless of the possible reordering of messages by the network. Traders should receive information on orders no later than 10 seconds after an exchange processes them, and the sooner, the better. Information received after 10 seconds needs to be marked as stale. Traders should be able to determine automatically how long ago an order was processed. Note: traders may not assume that the local timer is synchronized with the one on the exchange, unless your prototype takes definitive measures to make that happen. In case a trader crashes, all other components should be able to continue operation. Exchanges should recognize the failure and stop echoing processed orders to the trader. Upon restart, and re-register, the trader should be readmitted to the system. In case an exchange crashes, all other components should be able to continue operation. Traders should recognize the failure and send orders only to the surviving exchanges. Recovery mechanisms don’t need to be provided for exchange crashes.

Experiments and Measurements Dave

Experiment 1 Latency between starting to send an order and proceeding with other activities, averaged over at least 20 orders Average latency: 2.68 ms (3 sets of 21 trades) Maximum latency: 11 ms Created a button on the GUI to initiate a sequence of 21 sells Since our messages are sent from the trader asynchronously, we expected the latency to be very low Dave trader – initiate trade Nick trader Akhil exchange Sohrab exchange

Experiment 2 Latency between issuing an order and the last trader receiving it’s echo, averaged over at least 20 orders Average latency: 473 ms Tested with 1 exchange and 3 traders 1 trader sent all 21 trades Captured time message was sent and each message was received, factoring in offsets Dave trader – initiate trade Nick trader Akhil exchange Sohrab exchange – swap echos

Experiment 3 Latency between a trader failure and the last exchange stopping echoing orders, averaged over at least 5 failures Average latency: 18 secs Tested with 3 exchanges, 1 trader Time of trader closing is written to the log file When exchange detects failure (exception raised from message failure), time is recorded Dave trader Nick trader – close window Akhil exchange Sohrab exchange

Experiment 4a: Skew Average and max observed during operation Clock skew Average: 13.5 sec Max: 1 min 14 sec Each trader calculates the offset with every exchange to which it is connected by “time” messages sent at startup Configuration file includes parameter for number of hours to wait before calculating offset again

Experiment 4a: Offset

Experiment 4b: Latency Average and max observed during operation network latency Average: 189 ms (124 trades, 1 direction) Max: 470 ms (124 trades, 1 direction) On both exchange and trader, checkbox and spinner control allow for the simulation of network latency (on/off, 1 - 15 sec) “…assume the propagation delays from A to B is roughly the same as B to A…” Dave trader – initiates trades Nick trader Akhil exchange Sohrab exchange – network latency of 5 then 13 seconds

Experiment 5 Storage footprint on each exchange required to echo orders to traders, as a function of the number of traders (use empirical and/or analytical methods) Storage footprint of Exchange: 1.61 MB Storage footprint on Exchange per Trader: 59 KB x = memory utilized by an idle exchange y = memory used by exchange to communicate with each trader connected n = number of traders connected to exchange z = total memory used by exchange z = x + ny y = (z – x) / n

Experiment 5: Memory Usage

Development Measurements 1: Lines of Code

Development Measurements 2: Level of Effort

Measurement 3 Network traffic: 1 Exchange, 1 Trader, approximately 5 min 37 sec 1 Registration 1 RegistrationAck 10 Time 10 TimeAck 34 Ping (trader) 34 PingAck (exch) 23 Ping (exch) 23 PingAck (trader) = 136 Messages