Host Side Dynamic Reconfiguration with InfiniBand TM By Wei Lin Guay*, Sven-Arne Reinemo*, Olav Lysne*, Tor Skeie*, Bjørn Dag Johnsen^ and Line Holen^ *Simula Research Laboratory ^Sun Microsystems

Introduction The quest for ever increasing computing power drives the state-of-art large scale clusters. In Top500 list, more than 20 sites have > 10k processors supercomputers. The increased cluster size is challenging the reliability of interconnects – InfiniBand.

Introduction What are the available fault tolerance mechanisms?  Check-point/restart: Halted and restarted from the last checkpoint. Disadvantages: non-application transparent.  Deadlock-free re-routing Application transparent. Disadvantages: Inflexible.  Network Dynamic reconfiguration is the trend!

Network Dynamic Reconfiguration Network dynamic reconfiguration.  Move from one routing function to another while system is up and running.  Application transparent.  More flexible. Challenges of network dynamic reconfiguration  Deadlock freedom in the transition phase.  Assume that the network interface attributes have not been changed.

Host Side Dynamic Reconfiguration Host Side Dynamic Reconfiguration.  Migrate the attributes of the connection (Queue Pair) from the old routing structure to the new one.  Fault tolerance mechanism.  Live Migration  Policy Changes – Cluster Maintenance. Challenges of Host Side Dynamic Reconfiguration.  Which component to trigger the changes of routing path during the fault happened?  Setup prior alternative paths?  Network manager responsible to find new path?

Challenges of Dynamic Reconfiguration RC connection established between A and B.

Challenges of Dynamic Reconfiguration RC connection established between A and B. During the transmission, a link fails!

Challenges of Dynamic Reconfiguration RC connection established between A and B. During the transmission, a link fails! SM regenerated a deadlock free routing table.

Challenges of Dynamic Reconfiguration RC connection established between A and B. During the transmission, a link fails! SM regenerated a deadlock free routing table. Predefined deadlock free and shortest path for every paths are very difficult!

Host Side Dynamic Reconfiguration

1 1

2 2

3 3

Host Reconfiguration Keep track active QPs created in each host stack

Host Reconfiguration Keep track active QPs created in each host stack Modify QP’s context in RTS state  Reset Queue Pair

Host Reconfiguration Keep track active QPs created in each host stack Modify QP’s context in RTS state  Reset Queue Pair  Send Queue Drain(SQD)

Host Reconfiguration Keep track active QPs created in each host stack Modify QP’s context in RTS state  Reset Queue Pair  Send Queue Drain(SQD)  Auto. Path Mig.(APM)

Performance Evaluation Synthetic Traffic Patterns.  6-3:5-2:4-1:3-6:2-5:1-4 Application traffic patterns  HPCC b_eff

Performance Evaluation Micro benchmark − Setup Phase: No additional overhead!

Performance Evaluation Synthetic traffic patterns

Performance Evaluation HPCC b_eff Without dynamic reconfiguration  Benchmark will not complete once the first fault happened.  Deadlock happened!

Conclusion Novel fault tolerance mechanism  Feedback from SM.  Application Transparent. Evaluation of Scalability.  Event notification. Live Migration of Virtualization. Future Work

Thanks!