Interconnect simulation

Different levels for Evaluating an architecture Numerical models – Mathematic formulations to obtain performance characteristics (close form or non-close form) Simulation – Simulating the hardware/software in a program Emulation Prototype Operational system

Simulation *Jim Kurose, University of Massachusets, Amherst system under study (has deterministic rules governing its behavior) exogenous inputs to system (the environment) system boundary observer “real” life computer program simulates deterministic rules governing behavior psuedo random inputs to system (models environment) program boundary observer “simulated” life

Why simulation? Real-system not available, is complex/costly or dangerous (eg: space simulations, flight simulations) Quickly evaluate design alternatives (eg: different system configurations) Evaluate complex functions for which closed form formulas or numerical techniques not available

Simulation advantages and limitations Advantages: – sometimes cheaper – find bugs (in design) in advance – generality: over analytic/numerical techniques – detail: can simulate system details at an arbitrary level Drawbacks: – caution: does the simulation reflect reality? – large scale systems: lots of resources to simulate (especially to accurately simulate) – may be slow (computationally expensive – 1 min real time could be hours of simulated time) – art: determining right level of model complexity – statistical uncertainty in results

Types of simulators Time driven simulation – Cycle/instruction based simulation – Stop and observe the system after a fixed time interval (N cycles, N instructions, etc) – Events are assumed to happen at the time interval boundary Event driven simulation – Move from one event to another – Not at a fixed interval – Always simulate the first event in the future – this event may trigger new events before the second event.

Types of simulators For network simulation, it is more common to use event driven simulation technique. Flow-level simulation (fluid model) – How each flow (message) is transmitted – Events: flow start, flow finish Both can affect the rates of the current flows. When there is no event, the system state does not change, flows just follow the fluid model, which is computed in the previous event. – Topology and routing issues

Types of simulators Packet-level simulation – How each packet is sent from the source to the destination – Messages are chopped into packets – The network is modeled as a network of queues; and packets move from queue to queue. Processing node: send queue, receive queue Routers: queues for each port (input queue and output queue). – Sample events: message arrival, message departure, packet arrival (departure) to(from) a queue. – Besides the topology and routing issues, packet level simulation can also be used to study more detailed packet level issues such as packet scheduling, congestion control mechanisms, switch latency, etc. Flit-level simulation – How each flit goes through the network – Similar to the packet-level simulation except the unit is flit: there are a lot more flits than packets. – Needed to study cut-through routing and wormhole routing.

Programming an event driven interconnect simulator Define a set of events – E.g. Message arrival, message departure, etc Each event will trigger the change of the state in the system The simulator simulates the system activities for all events – Change system states E.g. In a flow-level simulator, when a message starts communicating, the throughput of other messages may be affected because of sharing. – Adding future events if necessary A message arrival will trigger packets being generated.

Programming an event-driven simulator An event driven simulator must maintain the following: – simulated time: internal (to simulation program) variable that keeps track of simulated time – system “state”: variables maintained by simulation program define system “state” e.g., may track number (possibly order) of packets in queue, current value of retransmission timer – events: points in time when system changes state each event has associate event time – e.g., arrival of packet to queue, departure from queue – precisely at these points in time that simulation must take action (change state and may cause new future events) model for time between events (probabilistic) caused by external environment

Event driven simulator block diagram initialize event list get next (nearest future) event from event list Advance time to event time update statistics done? n process event (change state values, add/delete future events from event list), etc

An example: a flow level network simulator Input: – a list of messages as well as their arrival time src_id, dst_id, msg_size, arrival_time Output – the finishing time for each message Assumption: The topology and the routing scheme is given. – Topology: Link between switches and its bandwidth – Routing: given a pair of src_id and dst_id, routing givens the list of links along a path from source to destination.

An example flow level simulator Events – Message arrival and message departure Modeling network activities – Given a set of active flows, we must model the transmission rates for each active flow. Need to make assumptions One possible assumption: the flow rate is equal to the equal share of the link bandwidth for the most loaded link along its path (remember Method 1 in the modeling) Another possible assumption: MAX-MIN fairness.

An example flow level simulator Assumption 1: the flow rate is equal to the equal share of the link bandwidth for the most loaded link along its path. Assumption 2: MAX-MIN fairness Active flows: 1->5, 2->3, 3->5, 4->5, link bandwidth 1Mbps – Assumption 1: 1->5, 3->5, 4->5 333Mbps, 2->3, 500Mbps – Assumption 2: 1->5, 3->5, 4->5 333Mbps, 2->3, 667Mbps

An example flow level simulator Network state: – Active messages, sizes of remaining active messages, rate for each active message, may be the finishing time for each active message based on the current data rate. – When a new message arrives, the number of active messages increases: the network state (flow rates) needs to be recomputed – When a message finishes, the number of active message decreases: the network state (flow rates) needs to be recomputed.

An example flow level simulator initialize event list get next (nearest future) event from event list time = event time update statistics done? n process event (change state values, add/delete future events from event list) put all messages in a priority queue get next (nearest future) (first arrival time or finishing time) time = event time If message finish, output time/msg done? n Change the set of active flows Recompute the remaining message size, new rate, and new finishing time

An example flow level simulation How accurate if the simulator? – A lot of assumptions – this is common for all simulation type of work. – If the assumption or the model is off, simulation produces junk results. – Implementation details have significant impacts on the results (numbers reported) – unknown simulator results are less credible. How to offset this? – Use existing (well known) simulator, or simulation infrastructure. – Be very careful in developing a new simulator.

Another example: a packet level simulator Objective: to investigate different packet scheduling algorithms used in routers. Input: – a list of messages as well as their arrival time src_id, dst_id, msg_size, arrival_time – The messages can be generated following some distribution – to evaluate the performance under different traffic conditions. Output – the finishing time for each message

An example packet level simulator Events: – Message arrives a compute node, packet at a compute node, packet arrives at an input port, packet moves to an output port, etc. Model system activities – Modeling the compute node behavior – Modeling router behavior

An example packet level simulator Model system activities – Modeling the compute node behavior When a message arrives, packetize the message, insert packets into a host packet queue (with ready time) to be injected to the network. When a packet is received, if it is the last packet of a message, keep the statistics.

An example packet level simulator – Modeling router behavior Each input port can be modeled by an input queue Each output port can be modeled by an output queue Packet scheduling algorithm: when multiple packets are ready at the input queue, the algorithm determines which packet gets to be scheduled to the output queue. Modeling router latency: after a packet is removed from an input queue, insert the packet to the output queue with a time stamp equal to current_time + latency – Modeling link latency?

An example packet level simulator Network states: – Queues on routers and end hosts, message at end hosts The simulator basically figures out the next future event, and perform the event – A message arrives at end host: packetize the message – A packet is ready on end host: move the packet – A packet is ready on a queue: move the packet

An example packet level simulator initialize event list get next (nearest future) event from event list time = event time update statistics done? n process event (change state values, add/delete future events from event list)

Packet level.vs. flow level simulation Flow level simulation has higher level of abstraction – more details are lost. Is flow level simulation always more efficient? – Not necessary, rate calculation can be quite compute-intensive.

Traffic generation issue Random uniform traffic, traffic arrival follows a Poisson process. – In one time slot, each SD pair has equal probability to have a message. – Cycle level simulation: use a random number trying to generate traffic in every cycle. Each cycle each SD pair has 0.1% change of having a traffic. – Event driven simulation: Determine the time when a message is supposed to be generated. – On average 1 message in 1000 cycles. The next message time is exponentially distributed with an average of » Once the time is determined, generate a random SD pair.

Network simulation packages For network simulation, the driver is quite standard. – Difference in the modeling techniques Quite some network simulators are available: – NS2, OpNet, Mininet, Booksim, etc – Provide many common library funcationality – Allow build the network topology and routing schemes. When to use the existing simulators and when to write one from scratch? – Decide whether existing one can do what you want to do.