1. Scalable, Controllable, Efficient and Convincing Crowd Simulation Mankyu Sung University of Wisconsin-Madison

2. What is Crowd? Crowd A group of individuals Share a same environment Have a same goal but may act in different way Unique Natural Phenomenon

3. What is Crowd Simulation? Crowd Simulation Artificial creation of virtual crowds Collection of behaviors of each individual Behaviors Selection of motions over time Real CrowdVirtual Crowd

4. Challenges Highly complex crowd behaviors Ex) Street environment : crossing, sitting, window- shopping, walking with friends… Not easy to maintain real-time performance Real time animation of more than 1,000 characters is hard Not easy to achieve convincingness Realistic motions, Realistic behaviors Need to minimize the control parameters

5. Research Goal : Four demands for crowd simulation Crowd simulation method that meets demands ScalabilityControllability Efficiency Convincingness Our goal

6. (1) Scalability Definition: Ability of crowd simulation to continue to function well when the environment become more complicated. Scalable Character Memory Agent’s memory is not proportional to the complexity of environment. Scalable Authoring The whole simulation should be broken into a set of small simulations. Author new simulation through simple copy&paste

7. (2) Controllability Definition: Ability to specify particular behaviors to characters by setting constraints Constraints High-level scenario for controlling crowd flow No specific individual Low-level individual constraints Specific individual

8. (3)Efficiency Definition: Fast processing of algorithm Fast motion synthesis Fast collision detection Linear simulation performance

9. (4)Convincingness Definition: Ability to cause viewers to believe specific aspects of crowds. Visually convincing behavior No discontinuity on motion No collisions between characters Semantically convincing behavior No unreasonable behaviors (depending on animator’s intention)

10. Outline 1. Features of crowds 2. Two-level crowd simulations method 3. Probability scheme 4. Collision detection through MOBB(Motion Oriented Bounding Box) – (Briefly) 5. Constrained motion synthesis 6. Summary and future work

11. Solution : Observe real crowds What features crowds have? Anonymity Hard to distinguish one from others It does not matter who is doing, but what is happening there matters Locality Behaviors depend on location and social relationship Variety People behave differently

12. Solution : Two-Level crowd simulation framework Situations (What should each individual do ?) Motion Synthesis (How to show motions?) Motion Synthesis (How to show motions?) High-level Low-level Situations (What should each individual do ?)

13. An Environment Example: Model of Street Simulation: Real time, Reactive Rendering: Unreal Game Engine for playback

14. Observation: Behavior Depends on Situation Store Window In front of Store Window Possibly: stop to window shop Friends Together: Possibly: Stop to talk Probably: Have same goal Doorway In front of Doorway Possibly: open door, enter Unlikely: stand blocking door In a hurry: Check for traffic Run across street In Crosswalk: Walk across street once you’ve started Use crosswalk: Wait for green light Start crossing Sidewalk: Walk here Street: Generally, don’t walk here

15. Managing Environmental Complexity: Situation-Based Approach Many different situations An agent only needs a few at a time Each situation has a different set of local behaviors Plugs information into agents for local behaviors Scalable authoring and scalable behaviors Store Window In front of Store Window Possibly: stop to window shop Friends Together: Possibly: Stop to talk Probably: Have same goal Doorway In front of Doorway Possibly: open door, enter Unlikely: stand blocking door Use crosswalk: Wait for green light Start crossing Sidewalk: Walk here

16. Solution : Two-Level crowd simulation framework Situations (Sung et el, EG 04) Motion Synthesis High-level Low-level Collision Avoidance (Sung et al CS-TR-05-1592) Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05)

17. Low-level : Two motion synthesis techniques Probability Scheme (Sung et al EG 04) Stochastic crowd control Short term view of agents Constrained Motion Synthesis (Sung et al SCA 05) Planning-based behaviors Have constraints for individuals Use Snap-Together-Motion (Gleicher et al I3D 03)

18. Related Work: Inspirations from Research Crowd Modelling (e.g. Reynolds 1987, Musse 2001, Blue 1998, Henderson, 1974, Bouvier et al. 1997, Helbing et al 2000, …) Simple agents, simple rules Fast, scalable, emergent patterns but…difficult to generalize to complex behaviors Smart Environments (e.g. Kallman et al 1998, Farenc et al. 1999, Thomas et al 2000, Michael et al 2003, …) Objects tell agents what to do Scalable authoring, complex behaviors but… difficult to compose or get variability

19. Related Work Inspirations from Systems The Sims (Maxis/EA Games) Smart environment/ Smart object Scalable authoring Does not consider the social relations Commercial softwares Massive, SoftImage/Behavior, Character Studio, AI-Implant… Author scenes, not environments Similar means, Different ends Agent-centric approach

20. Solution : Probability Scheme Situations (Sung et el, EG 04) Collision Avoidance (Sung et al CS-TR-05-1529) High-level Low-level Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05) Probability Scheme (Sung et al EG 04)

21. Probability scheme : Agent Architecture Agents: Discrete set of actions (from mograph) Randomly choose from distributions Behavior functions provide distributions All aspects of agents can be updated dynamically

22. Probability Scheme: Simple Default Agents Default agents very simple Wander, don’t bump into things, … Extend agents as necessary to achieve complex behaviors

23. Situations: Extensible Agent Situations extend agents Add Actions Add Behavior Functions Add Sensors and Rules that inform Behavior Functions

24. Example Default agent can’t cross the street. How an agent crosses the street… Enters a Crosswalk Situation Crosswalk situation extends agent Sensor to see traffic lights Behavior Functions to cross a street Behavior Functions to stop Rules to wait for the light to change Remove extensions when done

25. Composing Behaviors: Action Selection Agent Left Right Straight ?

26. Composing Behaviors: Probability Scheme Behavior Function A Agent Left Right Straight.5.3.2

27. Composing Behaviors: Probability Scheme Behavior Function A Agent Left Right Straight.5.3.2 Behavior Function B.1.2.4.25.33

28. Situations Compose Agent can be in multiple situations Agent has union of all the things that different situations put in

29. Authoring: Painting Interface Author environments (not characters) Set of situation types Paint into environments Mix situations to make complex/compound ones

30. Experiments Gallery environment Street environment

31. Solution : Collision Avoidance Situations (Sung et al, EG 04) Collision Avoidance (Sung et al CS-TR-05-1529) High-level Low-level Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05)

32. Collision Detection : MOBB(Motion Oriented Bounding Box) Problem : Collision test between motions A motion is a sequence of poses (frames) Solution : 3D Bounding box of motion trajectory (X, Y, time) x y t x y t

33. MOBB Trees : Hierarchical subdivision of motions Level 1 Level 2 Level 3 root level1 level2

34. MOBB Trees : Construction of MOBB trees Local Axis-Aligned Method Motion trajectory Bounding Sylinder frame_ start frame_ end d v1v1 v2v2 v3v3 v4v4

35. MOBB Trees : Intersection tests Given two trees, intersections are tested from the top node to the bottom node until they reach leaf nodes A00 A10A11 B00 B10B11 MOBB(A)MOBB(B) A10 A11 B11 B10

36. Experiments Frame-by- frame MOBB Short motions (2.1 sec) 4.6 x 10 -6 sec2.3 x 10 -6 sec Long motions (42 sec) 68.3 x 10 -6 sec5.9 x 10 -6 sec

37. Solution : Constrained Motion Synthesis Situations (Sung et al, EG 04) Collision Avoidance (Sung et al CS-TR-05-1529) High-level Low-level Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05) Constrained Motion Synthesis (Sung et al SCA 05)

38. The Goal: Motion Synthesis for Crowds Problem : Constrained motion synthesis Positions, Orientation, Poses, Time duration Requirement Fast performance Accurate meeting constraints High quality motions Complicated environment Pose Orientation Position Time duration Target Initial

39. Our approach : Synthesize crowds one individual at a time Structured Motion graphs for low-level synthesis ( Gleicher et al. ’03) Must adapt to crowds Individual motions must be found very quickly Pure discrete synthesis cannot meet continuous constraints

40. Adapting Graph based Synthesis : Two-level synthesis Coarse search for global path planning Finer search for detailed motion synthesis Quickly find long motions in complex environments Incorporate continuous motion adjustment Discrete search to roughly satisfy constraints Additional displacements for precision Improves speed and accuracy

41. Related Work (1) Graph based motion synthesis (e.g. Arikan 2002, Arikan 2003, Gleicher 2003, Kovar 2002, Hue 2004, Lee 2002, Lee 2004, Reitsma 2004 ) Connecting discrete finite clips with simple interpolation or displacement mapping -Create new motions strictly by attaching clips → Hard to satisfy constraints exactly - Do not consider crowds

42. Related Work (2) Procedural Motion Synthesis (Bouvier 1997, Boulic 1990, Sun 2001, Boulic 2004) Controllable but not as realistic as motion capture data Motion Blending (Guo 1996, Park 2004, Petteré 2003) Continuous control over trajectory Limited and computationally costly

43. Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Initial Obstacle Target waypoints

44. Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Obstacle Initial Target waypoints 1 23

45. Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Obstacle Initial Target Backward Motion(M b ) Forward Motion(M f ) 3 2 1 Initial’

46. Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Forward motions Backward motions > ε Compare all pair of motions and return minimum cost Cost function : How close are they? C(M f, M b )

47. Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Old Motionsc Old Motions New motions Random select and Replace a clip < ε

48. Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed paths 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Obstacle Initial Target waypoints Joining

49. Demos Time constrained demo Spelling out SCA Box deliveryBig crowds

50. Performance results Example# of agent Duration (sec) AVG Time (sec) Total Time (sec) Time constrained 2014.80.214.2 Spelling out “SCA” 7021.30.1812.6 Box delivery4030.50.156 Big crowds50025.60.03517.5

51. Validation : A Virtual City A city A block # of characters : 2,000

52. Validation : Demand Satisfaction : Scalability Scalable memory Scalable Authoring Divide the whole city simulation into many small situations and reuse them through copy and paste

53. Validation : Demand Satisfaction : Controllability Situations show particular behaviors Can put benches any places with random orientations Able to control crowd flow (crossing the street) by connecting two opposite situations

54. Validation : Demand Satisfaction : Efficiency Fixed environment Fixed density Crowd simulation performance (excluding rendering)

55. Validation : Demand Satisfaction : Efficiency Average speed Collision detection through MOBB 1.7065e-005 sec Constrained motion synthesis 0.106 sec

56. Validation : Demand Satisfaction : Convincingness Visual Convincingness No collisions No noticeable discontinuity on motions Semantical Convincingness Depending on situations, crowd show different specific behaviors

57. Contributions Scalable, controllable and convincing two- level crowd simulation framework (Sung et al, EG 04) Situation-based approach Probability scheme Fast collision detection through MOBB trees (Sung et al, CS-TR-05-1529) Fast constrained motion synthesis (Sung et al, SCA 05)

58. Limitation and Future Work Limits to motion data Behaviors depend on available motion data Offline constrained motion synthesis Searching time is needed Crowd rendering Fast skinning of character is needed User interfaces More intuitive user interface is required

59. Thanks House of Moves Collaborators Motion Data Funding Michael Gleicher Stephen Chenney Demian Gordon Ohio State University Lucas Kovar HyunJoon Shin Aaron San Filippo NSF CCR-9984506 and CCR-0204372 MIC(Ministry of Information and Communication of Korea)