Physics-based Sound Synthesis with a Novel Friction Model Zhimin Ren, Hengchin Yeh, Ming C. Lin Department of Computer Science University of North Carolina

How can it be done? Foley artists manually make and record the sound from the real-world interaction Lucasfilm Foley Artist

How about Computer Simulation? Physical simulation drives visual simulation Sound synthesis can also be automatically generated through physical simulation Videos w/ and w/o audio

Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results

Overview of Sound Simulation The complete pipeline for sound simulation Sound Synthesis Sound Propagation Sound Rendering

Overview of Sound Simulation Sound Synthesis

Overview of Sound Simulation Sound Propagation

Overview of Sound Simulation Sound Rendering

Our Focus Sound Synthesis

Our Focus Sound Synthesis Simplified sound simulation pipeline

Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results

Previous Work Vibration analysis: Modal Analysis Interaction modeling Simple shape: Van den Doel and Pai 98 FEM: O’Brien et al. 02 Spring-mass system: Raghuvanshi and Lin 07 Interaction modeling Friction approximation (fast): Van den Doel et al. 01 (repetitive, and no connected with visual components) Friction simulation (accurate): Avanzini et al. 02 (too slow to be handled at audio rate) Impact and rolling: Raghuvanshi and Lin 06 (complicated interactions like friction not handled)

Constraints Lacks the flexibility for users to interactively do content design Material properties Sound synthesis from physical simulation relies on good interaction handling models Frictional contacts are hard to handle User interaction with the virtual objects is limited and non-intuitive

Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results

Our System Set-up Tablet Support User Interface

Main Results (1) An interactive and flexible work flow for sound synthesis User control over material parameters Intuitive interaction and real-time auditory feedback for easy testing and prototyping No event synchronization issue: users with little or no foley experience can start sound design and creation quickly Easy integration with game engines

Main Results (1) A new frictional contact model for sound synthesis Fast, allows real-time interaction Simulates frictional interactions at different levels: Macro shape Meso bumpiness Micro roughness Better matches their virtually-simulated visual counterparts

Main Results (3) Natural and intuitive interface for manipulating and interacting with objects in the virtual environment Applying forces in a familiar and natural manner Interaction with no or little training

System Overview

System Overview Sound synthesis module Modal Analysis: Raghuvanshi & Lin (I3D 2006) Impulse response

System Overview Interaction handling module State detection: lasting and transient contacts Converting interactions into impulses

Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results

Modal Analysis Deformation modeling Vibration of surface generates sound Sound sampling rate: 44100 Hz Impossible to calculate the displacement of the surface at sampling rate Represent the vibration pattern by a bank of damped oscillators (modes) Standard technique for real-time sound synthesis

Modal Analysis Discretization An input triangle mesh  a spring-mass system A spring-mass system  a set of decoupled modes Refer to previous work

Modal Analysis The spring-mass system set-up Each vertex is considered as a mass particle Each edge is considered as a damped spring

Modal Analysis Coupled spring-mass system to a set of decoupled modes

Modal Analysis A discretized physics system We use spring-mass system Small displacement, so consider it linear Stiffness Damping Mass Stiffness Damping Mass

Modal Analysis Rayleigh damping And diagonalizing Solve the Ordinary Differential Equation (ODE) Rayleigh damping And diagonalizing Now, solve this ODE instead

Modal Analysis Substitute (z are the modes) Solve the ODE Substitute (z are the modes) Now, solve this ODE instead

Modal Analysis General solution External excitation defines the initial conditions

Modal Analysis Assumptions In most graphics applications, only surface representations of geometries are given A surface representation is used in modal Analysis Synthesized sound appears to be “hallow”

Modal Analysis Summary An input triangle mesh  A spring-mass system  A set of decoupled modes

Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results

State Detection

State Detection Distinguishing between lasting and transient contacts In contacts? In lasting contacts?

Interaction Handling Lasting contacts  a sequence of impulses Transient contacts  a single impulse New slides to explain the object interaction to contact to impulse ()

Impulse Response Dirac Delta function as impulse excitation General solution with initial condition given by the impulse, we have: 37-39 link them smoothly

Impulse Response

Handling Lasting Contacts i.e. Frictional contacts How to add the sequence of impulses? The model has to be fast and simple, because…

Handling Lasting Contacts Update Rate: 100Hz Update Rate: 44100Hz

Handling Lasting Contacts The interaction simulation has to be stepped at the audio sampling rate: 44100 Hz The update rate of a typical real-time physics simulator: on the order of 100’s Hz Not enough simulation is provided by the physics engine An customized interaction model for sound synthesis

Our Solution Decompose the interaction into difference levels Different update rates at different levels Combined results offer a good approximation

Our Solution Three levels of simulation Macro level: simulating the interactions on the overall surface shape Meso level: simulating the interactions on the surface material bumpiness Micro level: simulating the interactions on the surface material roughness

Three-level Simulation Macro level: Geometry information Update rate: 100’s Hz Update rate does not need to be high The geometry information is from the input triangle mesh, and contacts are reported by collision detection in the physics engine. Live demo of only macro-level simulation enabled

Three-level Simulation Meso level: Bumpiness Bump mapping is ubiquitous in real-time graphics rendering Bump maps are visible to users but transparent to physics simulation

What Is Bump Mapping? Perturb vertex normals for shading No geometry details + = Image Courtesy to Wikipedia

Three-level Simulation Meso level simulation Makes sure visual and auditory cues are consistent Attends to surface bumpiness details Update rate: Event queue: 100’s Hz Event processor: 44100 Hz

Three-level Simulation Meso level simulation details (1) Event queue is update at 100’s Hz. Linear velocity and position information from the physics simulator. An event handler traverse back one time step to collect all “bumping” events in last time step

Three-level Simulation Meso level simulation details (2) Events from last time step are made up in this time at audio rate resolution. Latency: 10ms. 200ms latency tolerance (Bonneel et al. 08) Event Queue 1 step in physics simulation Event Processor Live demo of only meso-level simulation enabled and both macro and meso-level simulation enabled

Three-level Simulation Micro level simulation: Van den Doel et al. 01 Fractal noise is used to simulate the micro-level interaction Live demo of only micro-level simulation enabled And both micro, meso, and macro-level simulation enabled

Three-level Simulation Advantages: Fast and simple. Makes real-time sound synthesis driven by complex interaction possible. Captures the richness of sound varying at three levels of resolution Visual and auditory feedbacks are consistent

Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results

System Implementation Tablet support Material manipulation Users are allowed to change material parameters Testing “new materials” right away Material blending: linear interpolation Integration with a general physics engine and graphics engine Physics engine: Open Dynamics Engine (ODE) Graphics rendering engine: Open Source 3D Rendering Engine (OGRE)

Results and Possible Applications User interface demo Pre-loaded materials User adjustment of the parameters

Results and Possible Applications Virtual instrument demo Flexible - customized instrument with different material properties and sound qualities Complicated interaction is supported – not limited to percussion instrument

Summary

Summary An interactive and flexible work flow for sound synthesis A new frictional contact model for sound synthesis Natural and intuitive interface for manipulating and interacting with objects in the virtual environment

Future Work More natural and realistic material blending schemes More intuitive material selection tool (material pallet) Add room acoustic filters and 3D auralization Extend the surface mesh modal synthesis to tetrahedral meshes User study and validation Enabling tools for vision-impaired

Acknowledgements Army Research Office Carolina Development Foundation Intel Corporation National Science Foundation RDECOM