Scalable, Controllable, Efficient and Convincing Crowd Simulation Mankyu Sung University of Wisconsin-Madison

What is Crowd? Crowd A group of individuals Share a same environment Have a same goal but may act in different way Unique Natural Phenomenon

What is Crowd Simulation? Crowd Simulation Artificial creation of virtual crowds Collection of behaviors of each individual Behaviors Selection of motions over time Real CrowdVirtual Crowd

Challenges Highly complex crowd behaviors Ex) Street environment : crossing, sitting, window- shopping, walking with friends… Not easy to maintain real-time performance Real time animation of more than 1,000 characters is hard Not easy to achieve convincingness Realistic motions, Realistic behaviors Need to minimize the control parameters

Research Goal : Four demands for crowd simulation Crowd simulation method that meets demands ScalabilityControllability Efficiency Convincingness Our goal

(1) Scalability Definition: Ability of crowd simulation to continue to function well when the environment become more complicated. Scalable Character Memory Agent’s memory is not proportional to the complexity of environment. Scalable Authoring The whole simulation should be broken into a set of small simulations. Author new simulation through simple copy&paste

(2) Controllability Definition: Ability to specify particular behaviors to characters by setting constraints Constraints High-level scenario for controlling crowd flow No specific individual Low-level individual constraints Specific individual

(3)Efficiency Definition: Fast processing of algorithm Fast motion synthesis Fast collision detection Linear simulation performance

(4)Convincingness Definition: Ability to cause viewers to believe specific aspects of crowds. Visually convincing behavior No discontinuity on motion No collisions between characters Semantically convincing behavior No unreasonable behaviors (depending on animator’s intention)

Outline 1. Features of crowds 2. Two-level crowd simulations method 3. Probability scheme 4. Collision detection through MOBB(Motion Oriented Bounding Box) – (Briefly) 5. Constrained motion synthesis 6. Summary and future work

Solution : Observe real crowds What features crowds have? Anonymity Hard to distinguish one from others It does not matter who is doing, but what is happening there matters Locality Behaviors depend on location and social relationship Variety People behave differently

Solution : Two-Level crowd simulation framework Situations (What should each individual do ?) Motion Synthesis (How to show motions?) Motion Synthesis (How to show motions?) High-level Low-level Situations (What should each individual do ?)

An Environment Example: Model of Street Simulation: Real time, Reactive Rendering: Unreal Game Engine for playback

Observation: Behavior Depends on Situation Store Window In front of Store Window Possibly: stop to window shop Friends Together: Possibly: Stop to talk Probably: Have same goal Doorway In front of Doorway Possibly: open door, enter Unlikely: stand blocking door In a hurry: Check for traffic Run across street In Crosswalk: Walk across street once you’ve started Use crosswalk: Wait for green light Start crossing Sidewalk: Walk here Street: Generally, don’t walk here

Managing Environmental Complexity: Situation-Based Approach Many different situations An agent only needs a few at a time Each situation has a different set of local behaviors Plugs information into agents for local behaviors Scalable authoring and scalable behaviors Store Window In front of Store Window Possibly: stop to window shop Friends Together: Possibly: Stop to talk Probably: Have same goal Doorway In front of Doorway Possibly: open door, enter Unlikely: stand blocking door Use crosswalk: Wait for green light Start crossing Sidewalk: Walk here

Solution : Two-Level crowd simulation framework Situations (Sung et el, EG 04) Motion Synthesis High-level Low-level Collision Avoidance (Sung et al CS-TR ) Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05)

Low-level : Two motion synthesis techniques Probability Scheme (Sung et al EG 04) Stochastic crowd control Short term view of agents Constrained Motion Synthesis (Sung et al SCA 05) Planning-based behaviors Have constraints for individuals Use Snap-Together-Motion (Gleicher et al I3D 03)

Related Work: Inspirations from Research Crowd Modelling (e.g. Reynolds 1987, Musse 2001, Blue 1998, Henderson, 1974, Bouvier et al. 1997, Helbing et al 2000, …) Simple agents, simple rules Fast, scalable, emergent patterns but…difficult to generalize to complex behaviors Smart Environments (e.g. Kallman et al 1998, Farenc et al. 1999, Thomas et al 2000, Michael et al 2003, …) Objects tell agents what to do Scalable authoring, complex behaviors but… difficult to compose or get variability

Related Work Inspirations from Systems The Sims (Maxis/EA Games) Smart environment/ Smart object Scalable authoring Does not consider the social relations Commercial softwares Massive, SoftImage/Behavior, Character Studio, AI-Implant… Author scenes, not environments Similar means, Different ends Agent-centric approach

Solution : Probability Scheme Situations (Sung et el, EG 04) Collision Avoidance (Sung et al CS-TR ) High-level Low-level Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05) Probability Scheme (Sung et al EG 04)

Probability scheme : Agent Architecture Agents: Discrete set of actions (from mograph) Randomly choose from distributions Behavior functions provide distributions All aspects of agents can be updated dynamically

Probability Scheme: Simple Default Agents Default agents very simple Wander, don’t bump into things, … Extend agents as necessary to achieve complex behaviors

Situations: Extensible Agent Situations extend agents Add Actions Add Behavior Functions Add Sensors and Rules that inform Behavior Functions

Example Default agent can’t cross the street. How an agent crosses the street… Enters a Crosswalk Situation Crosswalk situation extends agent Sensor to see traffic lights Behavior Functions to cross a street Behavior Functions to stop Rules to wait for the light to change Remove extensions when done

Composing Behaviors: Action Selection Agent Left Right Straight ?

Composing Behaviors: Probability Scheme Behavior Function A Agent Left Right Straight.5.3.2

Composing Behaviors: Probability Scheme Behavior Function A Agent Left Right Straight Behavior Function B

Situations Compose Agent can be in multiple situations Agent has union of all the things that different situations put in

Authoring: Painting Interface Author environments (not characters) Set of situation types Paint into environments Mix situations to make complex/compound ones

Experiments Gallery environment Street environment

Solution : Collision Avoidance Situations (Sung et al, EG 04) Collision Avoidance (Sung et al CS-TR ) High-level Low-level Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05)

Collision Detection : MOBB(Motion Oriented Bounding Box) Problem : Collision test between motions A motion is a sequence of poses (frames) Solution : 3D Bounding box of motion trajectory (X, Y, time) x y t x y t

MOBB Trees : Hierarchical subdivision of motions Level 1 Level 2 Level 3 root level1 level2

MOBB Trees : Construction of MOBB trees Local Axis-Aligned Method Motion trajectory Bounding Sylinder frame_ start frame_ end d v1v1 v2v2 v3v3 v4v4

MOBB Trees : Intersection tests Given two trees, intersections are tested from the top node to the bottom node until they reach leaf nodes A00 A10A11 B00 B10B11 MOBB(A)MOBB(B) A10 A11 B11 B10

Experiments Frame-by- frame MOBB Short motions (2.1 sec) 4.6 x sec2.3 x sec Long motions (42 sec) 68.3 x sec5.9 x sec

Solution : Constrained Motion Synthesis Situations (Sung et al, EG 04) Collision Avoidance (Sung et al CS-TR ) High-level Low-level Probability Scheme (Sung et al EG 04) Constrained Motion Synthesis (Sung et al SCA 05) Constrained Motion Synthesis (Sung et al SCA 05)

The Goal: Motion Synthesis for Crowds Problem : Constrained motion synthesis Positions, Orientation, Poses, Time duration Requirement Fast performance Accurate meeting constraints High quality motions Complicated environment Pose Orientation Position Time duration Target Initial

Our approach : Synthesize crowds one individual at a time Structured Motion graphs for low-level synthesis ( Gleicher et al. ’03) Must adapt to crowds Individual motions must be found very quickly Pure discrete synthesis cannot meet continuous constraints

Adapting Graph based Synthesis : Two-level synthesis Coarse search for global path planning Finer search for detailed motion synthesis Quickly find long motions in complex environments Incorporate continuous motion adjustment Discrete search to roughly satisfy constraints Additional displacements for precision Improves speed and accuracy

Related Work (1) Graph based motion synthesis (e.g. Arikan 2002, Arikan 2003, Gleicher 2003, Kovar 2002, Hue 2004, Lee 2002, Lee 2004, Reitsma 2004 ) Connecting discrete finite clips with simple interpolation or displacement mapping -Create new motions strictly by attaching clips → Hard to satisfy constraints exactly - Do not consider crowds

Related Work (2) Procedural Motion Synthesis (Bouvier 1997, Boulic 1990, Sun 2001, Boulic 2004) Controllable but not as realistic as motion capture data Motion Blending (Guo 1996, Park 2004, Petteré 2003) Continuous control over trajectory Limited and computationally costly

Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Initial Obstacle Target waypoints

Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Obstacle Initial Target waypoints 1 23

Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Obstacle Initial Target Backward Motion(M b ) Forward Motion(M f ) Initial’

Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Forward motions Backward motions > ε Compare all pair of motions and return minimum cost Cost function : How close are they? C(M f, M b )

Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed motions 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Old Motionsc Old Motions New motions Random select and Replace a clip < ε

Algorithm 1. Rough planning 1. PRM query 2. Fine planning 1. Greedy search 2. Create seed paths 3. If distance > ε Randomly select and replace a clip 4. Joining with adjustment Example Obstacle Initial Target waypoints Joining

Demos Time constrained demo Spelling out SCA Box deliveryBig crowds

Performance results Example# of agent Duration (sec) AVG Time (sec) Total Time (sec) Time constrained Spelling out “SCA” Box delivery Big crowds

Validation : A Virtual City A city A block # of characters : 2,000

Validation : Demand Satisfaction : Scalability Scalable memory Scalable Authoring Divide the whole city simulation into many small situations and reuse them through copy and paste

Validation : Demand Satisfaction : Controllability Situations show particular behaviors Can put benches any places with random orientations Able to control crowd flow (crossing the street) by connecting two opposite situations

Validation : Demand Satisfaction : Efficiency Fixed environment Fixed density Crowd simulation performance (excluding rendering)

Validation : Demand Satisfaction : Efficiency Average speed Collision detection through MOBB e-005 sec Constrained motion synthesis sec

Validation : Demand Satisfaction : Convincingness Visual Convincingness No collisions No noticeable discontinuity on motions Semantical Convincingness Depending on situations, crowd show different specific behaviors

Contributions Scalable, controllable and convincing two- level crowd simulation framework (Sung et al, EG 04) Situation-based approach Probability scheme Fast collision detection through MOBB trees (Sung et al, CS-TR ) Fast constrained motion synthesis (Sung et al, SCA 05)

Limitation and Future Work Limits to motion data Behaviors depend on available motion data Offline constrained motion synthesis Searching time is needed Crowd rendering Fast skinning of character is needed User interfaces More intuitive user interface is required

Thanks House of Moves Collaborators Motion Data Funding Michael Gleicher Stephen Chenney Demian Gordon Ohio State University Lucas Kovar HyunJoon Shin Aaron San Filippo NSF CCR and CCR MIC(Ministry of Information and Communication of Korea)