Optimistic Load Balancing in a Distributed Virtual Environment Roman Chertov and Sonia Fahmy Department of Computer Science Purdue University {rchertov,

u2 Serv1 Serv2 Distributed Virtual Environment DVE Servers Internet ISP 2 ISP1 Serv1 Serv2 u1 u2 u3 u4 u5 u4 u1 u3

Massive Virtual Environments  Massive Distributed Virtual Environments (DVEs) are hard to create.  How to design a DVE architecture capable of: 1.High update rates 2.Large scale 3.Simple yet effective load balancing 4.Non-partitioned unified world

Approaches Peer-to-Peer + Scales Well + No single point of failure - No latency guarantees - Not very dynamic Distributed Servers + No single point of failure - Not very dynamic Server Cluster + Can be highly dynamic (hot spots) + Latency can be guaranteed - Single point of failure

Related Work: Architecture  Peer to Peer  S. Hu and G. Liao, “Scalable peer-to-peer networked virtual environment,” NetGames 2004  T. Iimura et al., “Zoned federation of game servers: A peer-to-peer approach to scalable multi-player online games,” NetGames 2004  Distributed Servers  A. Bharambe et al., “A Distributed Architecture for Interactive Multiplayer Games,” Tech Report 2005  J. Lui and M. F. Chan, “An Efficient Partitioning Algorithm for Distributed Virtual Environment Systems,” IEEE Transactions on Parallel Distributed Systems 2002  B. Ng et al., “Multiserver support for large-scale distributed virtual environments,” IEEE Transactions on Multimedia 2005  Server Cluster  T. Duong and S. Zhou, “A dynamic load sharing algorithm for massively multiplayer online games,” ICON 2003  D. Min et al., “A Load Balancing Algorithm for a Distributed Multimedia Game Server Architecture,” ICMCS 1999  M. Hori et al., “Scalability issues of dynamic space management for multiple-server networked virtual environments,” PACRIM 2001  D. Bauer et al, “Network infrastructure for massively distributed games,” NetGames 2002

Related Work: Load Balancing  Game space can be divided into a grid and servers manage the regions.  B. Ng et al., “Multiserver support for large-scale distributed virtual environments,” IEEE Transactions on Multimedia 2005  M. Hori et al., “Scalability issues of dynamic space management for multiple-server networked virtual environments,” PACRIM 2001  T. Duong and S. Zhou, “A dynamic load sharing algorithm for massively multiplayer online games,” ICON 2003  Glen Deen, “Making Quake II Massively Multiplayer with OptimalGrid,” ICPP 2005 Keynote  Game space can be divided into cells and a server can manage cells that are not adjacent in game space  B. Vleeschauwer et al., “Dynamic microcell assignment for massively multiplayer online gaming,” NetGames 2005

Our Approach  Server Cluster with stateless servers.  Server coverage is dynamic (move, resize)  Servers perform simple greedy actions to achieve global balanced state.

How it Looks

System Architecture  Load balancing Core runs the virtual environment and performs data exchange and validation  Gateways  Aggregate client data  Provide fast connection to the Core via MPLS  Filter updates from the Core for individual clients

System Architecture (cont’d)  Servers are stateless to avoid migration costs.  If update state is close to total state this will work. (action games)  Application specific routing is used to deliver messages to the correct server.  Gateways and Servers run in sync. 1.Gateways aggregate client data and push it to the Core 2.Core pushes new data to the gateways

Load Balancer Properties Properties of a good load balancer i.Achieve even load distribution ii.Efficiently handle sparse environments iii.Allow dynamic points of interest Observation:  In DVEs, clients tend to cluster around points of interest

Game Space Partitioning  It is possible to use rectangles to partition game space into non-overlapping regions.  Rectangles are easier and faster to use than more complex polygons. Server 2Server 1 Server 3

Load Mitigation  If the server load exceeds a Threshold, then it has to migrate some client to another server Server 2’s Threshold of 3 is exceededBalance is reached

Problems with Rectangle Bounds  When using rectangles for boundaries inefficiencies will occur!

Intersection Resolution  Servers can migrate their clients to other servers if their coverage areas overlap and MaxClients will not be exceeded in the target server. Server 2 Server 1 Server 2 Server 2 and Server 1 overlap!Overlap is resolved

Additional Rules  Client transfer is rejected if coverage of the server will exceed MaxArea (configurable parameter)  If coverage of the server exceeds MaxArea then set Threshold to 0 to start migrations

Tradeoff  If MaxClients >> Threshold then load variance will increase; however, conflicts will be easier to resolve.

Single Server Tasks 1.NP to process the physics and cheat detection for N clients. 2.N 2 to search for local client interactions. 3.Total run time = NP + N 2 i.e. O(N 2 ) N 2 can be reduced to N 1.5 by using quad trees. For large enough N, the interaction search will take more time than computing physics!

Tasks of a Distributed Server i 1.N i P to process the physics and cheat detection for its own clients. 2.N i 2 to search for local client interactions. 3.M-1 to determine the K intersecting and adjacent rectangles that are managed by other M-1 servers. 4.N i Σ K k=0 N k to determine the clients that potentially can interact with its clients. 5.N i +K to determine the cilent that is closest to the closest server if migration is required. 6.KN i to determine which current clients are within other servers’ coverage so that they can be migrated there, provided the max capacity is not reached.

Runtime Complexity Single Server Complexity O(N 1.5 )  N total number of clients Distributed Server Complexity O(N i (N i ∑ K k=0 N k ) 1.5 )  K is the number of overlapping servers  N i number of clients at one server It is essential to keep N i as close to N/ServerNum and K as small as possible!

Experimental Setup  10x10 Km map with 64 points of interest (evenly spaced)  64 Servers  Clients (increased by 100)  Servers can do only 3 migrations when trying to reach Threshold.  30 minute run time  Parameters  Threshold = ClientNum / 64  MaxClients = Threshold * 1.5  MaxArea = 10 Km x 10 Km / 64

Client Behavior  A client chooses a point of interest in MapLen / 3 radius and goes there.  Client chooses a speed at random with 5 m/s being the max.  While en-route, it can choose another point with a probability (tested every cycle)  When it arrives at the point of interest, it traverses a 50 m x 50 m region for 300 m total. Then it chooses another destination.

Client Behavior (cont’d)  Random Way Point model is not considered accurate for some games:  S. Tan et al., “Networked game mobility model for first-person-shooter games,” NetGames 2005  In our case, the map has no geographical obstacles, clients can change direction while en-route, and clients prefer way points that are closer.  Hence, the client mobility model we chose is adequate.

Load Balancing of Clients  Figure (a) shows that the average is close to the ideal distribution.  Property (i) is satisfied (good balancing).  Figure (b) shows the increase in Std. deviation as the number of clients increases. This is because of the balancing tradeoff.

Total Covered Area  When the number of clients is small, the server cover a small percentage of the total area.  Property (ii) is satisfied (sparse areas)

Coverage Overlap  Number of overlaps is K from analysis.  K increases very slowly as number of clients increases.  Since K is small and the load is close to even, our distributed system will outperform a single server.

Dynamic Points of Interest  We have conducted 10 runs with 2000 clients.  The configuration is the same for all the runs except the seeds.  The points of interest are randomly chosen on the map at the start.

Dynamic Points of Interest (cont’d)  Small variance between the runs.  Average Std. deviation was 8.6  Average server overlap count was 1.83 This shows that the system can handle dynamic points of interest that are unknown ahead of time. Property (iii) is satisfied

Large Number of Clients  clients.  100 servers.  100 points of interest.  Result  Std. deviation of 35.2  Average K 2.67 This shows potential that the system can scale to very large number of clients.

Conclusions  Our load balancer takes simple greedy actions.  Good load balancing properties are satisfied.  Each Server can in turn be a Core, thus leading to good hierarchical scaling properties!  Future work: Adjust system parameters as clients join and leave.