1. 1 Enhancing Neighborship Consistency for Peer-to-Peer Distributed Virtual Environments Jehn-Ruey Jiang, Jiun-Shiang Chiou and Shun-Yun Hu Department of Computer Science and Information Engineering National Central University

2. 2 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

3. 3 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

4. 4 DVE (1) Distributed Virtual Environments (DVEs) are computer-generated virtual world where multiple geographically distributed users can assume virtual representatives (or avatars) to concurrently interact with each other A.K.A. Networked Virtual Environments (NVEs)

5. 5 DVE (2) Examples of DVEs include early DARPA SIMNET and DIS systems, as well as currently booming Massively Multiplayer Online Games (MMOGs).

6. 6 Adaptive Computing and Networking Lab, CSIE, NCU Massively Multiplayer Online Games MMOGs are growing quickly –8 million registered users for World of Warcraft –Over 100,000 concurrent players –Billion-dollar business

7. 7 Adaptive Computing and Networking Lab, CSIE, NCU

8. 8

9. 9

10. 10

11. 11 Adaptive Computing and Networking Lab, CSIE, NCU

12. 12 DVE (3) 3D virtual world with –People (avatar) –Objects –Terrain –Agents –… Each avatar can do a lot of operations –Move –Chat –Using items –…

13. 13 Issues for DVE Scalability –To accommodate as many as participants Consistency –All participants have the same view of object states Persistency –All contents (object states) in DVE need to exist persistently Reliability –Need to tolerate H.W and S.W. failures Security –To prevent cheating and to keep user information and game state confidentially.

14. 14 Adaptive Computing and Networking Lab, CSIE, NCU The Scalability Problem (1) Client-server: has inherent resource limit Resource limit [Funkhouser95]

15. 15 Adaptive Computing and Networking Lab, CSIE, NCU The Scalability Problem (2) Peer-to-Peer: Use the clients’ resources Resource limit [Keller & Simon 2003]

16. 16 Adaptive Computing and Networking Lab, CSIE, NCU You only need to know some participants ★ : self ▲: neighbors Area of Interest (AOI)

17. 17 Neighborship Consistency (1) Definition # current AOI neighbors observed # current AOI neighbors

18. 18 :is observed neighbor Neighborship Consistency (2) An example Neighborship Consistency = 4 / 5 = 80% :is actual neighbor

19. 19 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

20. 20 Related Work (1): DHT-based: SimMUD B. Knutsson, H. Lu, W. Xu and B. Hopkins, “Peer-to-peer Support for Massively Multiplayer Games,” in Proceedings of INFOCOM 2004. Authors are from Department of Computer and Information Science, University of Pennsylvania

21. 21 Related Work (1): DHT-based: SimMUD [Knutsson et al. 2004] (UPenn) Pastry (DHT mapping) + Scribe (Multicast) Fixed-Sized Regions Coordinators

22. 22 SimMUD -- Introduction Proposes use of P2P overlays to support Massively multiplayer games (MMG) Primary contribution of paper: –Architectural (P2P for MMG) –Evaluative

23. 23 SimMUD -- Introduction PASTRY (P2P overlay) SCRIBE (Multicast support) MMG GAME

24. 24 SimMUD -- Introduction Players contribute memory, CPU cycles and bandwidth for shared game state Persistent user state is centralized –Example: payment information, character Allows central server to delegate to peers the dissemination and the process of intensive game states

25. 25 Distributed Game Design GAME STATES –Game world divided into connected regions –Regions are controlled by different coordinates

26. 26 Distributed Game Design

27. 27 Distributed Game Design Game design based on fact that: –Players have limited movement speed –Limited sensing capability –Hence data shows temporal and spatial localities –Use Interest Management Limit amount of state player has access to

28. 28 Distributed Game Design Players in same region form interest group State updates relevant to group disseminated only within group Player changes group when going from region to region

29. 29 Distributed Game Design GAME STATE CONSISTENCY –Must be consistent among players in a region –Basic approach: employ coordinators to resolve update conflicts –Split game state management into three classes to handle update conflicts: Player state Object state The Map

30. 30 Distributed Game Design Player state –Single writer multiple reader –Position change is most common event Use best effort multicast to players in same region Use dead reckoning to handle loss or delay

31. 31 Distributed Game Design Object state –Use coordinator-based mechanism for shared objects –Each object assigned a coordinator –Coordinator resolves conflicting updates and keeps current value

32. 32 Distributed Game Design Map –Maps are considered read-only because they remain unchanged during the game play. –They can be created offline and inserted into the system dynamically. –Dynamic map elements are handled as objects.

33. 33 Game on P2P overlay Map game states to players –Group players & objects by region –Map regions to peers using pastry Key –Each region is assigned ID –Live Node with closest ID becomes coordinator –Random Mapping reduces chance of coordinator becoming member of region (reduces cheating) –Currently all objects in region coordinated by one Node –Could assign coordinator for each object

34. 34 Game on P2P overlay Shared state replication –Lightweight primary- backup to handle failures –Failure detected using regular game events –Dynamically replicate coordinator when failure detected –Keep at least one replica at all times –Uses property of P2P (route message with key K to node ID, say N, closest to K)

35. 35 Game on P2P overlay Shared state replication (contd..) –The replica kept at M which is the next closest to key K – If new node T added which is closer to K than coordinator N Forwards messages to coordinator N until all states of K are transferred from N to T Takes over as coordinator and N becomes a replica

36. 36 Game on P2P overlay Catastrophic failure –Both coordinator and replica dead –Problem solved by cached information from nodes interested in the area

37. 37 Experimental Results Prototype Implementation of “SimMud” Used FreePastry (open source) Maximum simulation size constrained by memory to 4000 virtual nodes Players eat and fight every 20 seconds Remain in a region for 40 seconds Position updates every 150 millisec by multicast

38. 38 Experimental Results Base Results –No players join or leave –300 seconds of game play –Average 10 players per region –Link between nodes have random delay of 3- 100 ms to simulate network delay

39. 39 Experimental Results (Base results)

40. 40 Experimental Results (Base results)

41. 41 Experimental Results (Base results)

42. 42 Experimental Results (Base results) 1000 to 4000 players with 100 to 400 regions Each node receives 50 –120 messages 70 update messages per second –10 players * 7 position updates Unicast and multicast message take around 6 hops (but 50 hops in the worst case)

43. 43 Experimental Results Breakdown of type of messages –99% messages are position updates –Region changes take most bandwidth –Message rate of object updates higher than player-player updates Object updates multicast to region Object update sent to replica Player player interaction effects only players

44. 44 Experimental Results Effect of Population Growth –As long as average density remains same, population growth does not make difference Effect of Population Density –Ran with 1000 players, 25 regions –Position updates increases linearly per node –Non – uniform player distribution hurts performance

45. 45 Experimental Results Three ways to deal with population density problem –Allow max number of players in region –Different regions have different size –System dynamically repartitions regions with increasing players

46. 46 Experimental Results Effect of message aggregation –Since updates are multicast, aggregate them at root –Position update aggregated from all players before transmit –Cuts bandwidth requirement by half –Nodes receive less messages

47. 47 Experimental Results

48. 48 Experimental Results

49. 49 Experimental Results Effect of network dynamics –Nodes join and depart at regular intervals –Simulate one random node join and depart per second –Per-node failure rate of 0.06 per minute –Average session length of 16.7 minutes (close to 18 minutes for a FPS game -- Half Life) –Average message rate increased from 24.12 to 24.52

50. 50 Related Work (2): Neighbor-list Exchange [Kawahara et al. 2004] (Univ. of Tokyo) Fully-distributed Nearest-neighbors List exchange High transmission Overlay partition

51. 51 Related Work (3): Mutual Notification: Solipsis [Keller & Simon 2003] (France Telecomm R&D) Links with AOI neighbor Mutual cooperation Inside convex hull Potentially slow discovery Inconsistent topology

52. 52 Related Work (4): P2P Hybrid (DHT, super-nodes, neighbor list exchange: MOPAR Yu, A. and Vuong, S. T., “MOPAR: a mobile peer-to-peer overlay architecture for interest management of massively multiplayer online games,” In Proceedings of the international Workshop on Network and Operating Systems Support For Digital Audio and Video (Stevenson, Washington, USA, June 13 - 14, 2005). NOSSDAV '05.

53. 53 Related Work (5): Voronoi-based Overlay Network : VON SY Hu, JF Chen, TH Chen, “VON: a scalable peer-to-peer network for virtual environments,” IEEE Network, 2006

54. 54 Design Goals Observation: –for virtual environment applications, the contents we want are messages from AOI neighbors –Content discovery is a neighbor discovery problem Solve the Neighbor Discovery Problem in a fully- distributed, message-efficient manner. Specific goals: –Scalable  Limit & minimize message traffics –Responsive  Direct connection with AOI neighbors

55. 55 Voronoi Diagram 2D Plane partitioned into regions by sites, each region contains all the points closest to its site Neighbors Site Region

56. 56 Design Concepts –Each node constructs a Voronoi of its neighbors –Identify enclosing and boundary neighbors –Mutual collaboration in neighbor discovery Use Voronoi to solve the neighbor discovery problem ● node i and the big circle is its AOI ■ enclosing neighbors ▲ boundary neighbors ★ both enclosing and boundary neighbors ▼ normal AOI neighbors ◆ irrelevant nodes

57. 57 Procedure (JOIN) 1)Joining node sends coordinates to any existing node Join request is forwarded to acceptor 2)Acceptor sends back its own neighbor list joining node connects with other nodes on the list Acceptor’s region Joining node

58. 58 Procedure (MOVE) 1)Positions sent to all neighbors, mark messages to B.N. B.N. checks for overlaps between mover’s AOI and its E.N. 2)Connect to new nodes upon notification by B.N. Boundary neighbors New neighbors

59. 59 Procedure (LEAVE) 1)Simply disconnect 2)Others then update their Voronoi new B.N. is discovered via existing B.N. Leaving node (also a B.N.) New boundary neighbor

60. 60 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

61. 61 Factors affecting Neighborship Consistency Mobility – Speed AOI-radius Node failure –Possibly causes overlay partition

62. 62 Mobility

63. 63 Overlay Partition (1/2) Nodes divide into two or more groups Each group can not be aware each other May destroy neighborship consistency In pure P2P DVE –May be impossible to detect and recover –Have to avoid it

64. 64 Overlay Partition (2/2) × × × × × ×

65. 65 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

66. 66 Proposed Solutions Adaptive AOI –AOI radius increases with speed increasing Detecting critical nodes –Critical nodes are the nodes in some regions where the density is low –It is used to avoid overlay partition

67. 67 Adaptive AOI

68. 68 Critical Nodes (1/4)

69. 69 Critical Nodes (2/4) Periodically detect by node itself # AOI neighbors Neighbor_level = Average # AOI neighbors If a node detects that its neighbor level is lower than a pre-specified threshold, it regards itself as a critical node. A critical node will perform the backup operation to send to all connected neighbors the backup message.

70. 70 Critical Nodes (3/4) A Backup message –Sent by critical nodes –Contains a neighbor list of the critical node Stock neighbor list –Stores nodes from backup message –Uses to periodically check Check operation –Check all nodes in its stock neighbor list periodically –To discovery missed neighbors

71. 71 Critical Nodes (4/4) A B C … … … … … … … … … Neighbor list of node A Stock neighbor list of node A … … … … Neighbor list of node B B C C

72. 72 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

73. 73 Simulation Parameters 1000 nodes in a 1200-unit by 1200-unit plane constant velocity of 5 units per time-step AOI radius is 100 units (the number of directly connected neighbors is limited to be 20) Clustered distribution: nodes cluster into three groups in the first 300 steps and can move randomly after 300 steps. Random distribution: nodes can move randomly in all simulation steps.

74. 74 Mobility (1/2)

75. 75 Mobility (2/2)

76. 76 Node Failure (1/8)

77. 77 Node Failure (2/8)

78. 78 Node Failure (3/8)

79. 79 Node Failure (4/8)

80. 80 Node Failure (5/8)

81. 81 Node Failure (6/8)

82. 82 Node Failure (7/8)

83. 83 Node Failure (8/8)

84. 84 Outline Introduction –Background –P2P DVEs Factors affecting Neighborship Consistency Proposed Solutions Simulation Results Conclusion

85. 85 Conclusion Address two factors –Node mobility –Node failure Improve neighborship consistency by –Adaptive AOI buffer –Critical node detection

86. 86 Q&A