Physics-based Sound Synthesis with a Novel Friction Model Zhimin Ren, Hengchin Yeh, Ming C. Lin Department of Computer Science University of North Carolina
How can it be done? Foley artists manually make and record the sound from the real-world interaction Lucasfilm Foley Artist
How about Computer Simulation? Physical simulation drives visual simulation Sound synthesis can also be automatically generated through physical simulation Videos w/ and w/o audio
Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results
Overview of Sound Simulation The complete pipeline for sound simulation Sound Synthesis Sound Propagation Sound Rendering
Overview of Sound Simulation Sound Synthesis
Overview of Sound Simulation Sound Propagation
Overview of Sound Simulation Sound Rendering
Our Focus Sound Synthesis
Our Focus Sound Synthesis Simplified sound simulation pipeline
Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results
Previous Work Vibration analysis: Modal Analysis Interaction modeling Simple shape: Van den Doel and Pai 98 FEM: O’Brien et al. 02 Spring-mass system: Raghuvanshi and Lin 07 Interaction modeling Friction approximation (fast): Van den Doel et al. 01 (repetitive, and no connected with visual components) Friction simulation (accurate): Avanzini et al. 02 (too slow to be handled at audio rate) Impact and rolling: Raghuvanshi and Lin 06 (complicated interactions like friction not handled)
Constraints Lacks the flexibility for users to interactively do content design Material properties Sound synthesis from physical simulation relies on good interaction handling models Frictional contacts are hard to handle User interaction with the virtual objects is limited and non-intuitive
Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results
Our System Set-up Tablet Support User Interface
Main Results (1) An interactive and flexible work flow for sound synthesis User control over material parameters Intuitive interaction and real-time auditory feedback for easy testing and prototyping No event synchronization issue: users with little or no foley experience can start sound design and creation quickly Easy integration with game engines
Main Results (1) A new frictional contact model for sound synthesis Fast, allows real-time interaction Simulates frictional interactions at different levels: Macro shape Meso bumpiness Micro roughness Better matches their virtually-simulated visual counterparts
Main Results (3) Natural and intuitive interface for manipulating and interacting with objects in the virtual environment Applying forces in a familiar and natural manner Interaction with no or little training
System Overview
System Overview Sound synthesis module Modal Analysis: Raghuvanshi & Lin (I3D 2006) Impulse response
System Overview Interaction handling module State detection: lasting and transient contacts Converting interactions into impulses
Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results
Modal Analysis Deformation modeling Vibration of surface generates sound Sound sampling rate: 44100 Hz Impossible to calculate the displacement of the surface at sampling rate Represent the vibration pattern by a bank of damped oscillators (modes) Standard technique for real-time sound synthesis
Modal Analysis Discretization An input triangle mesh a spring-mass system A spring-mass system a set of decoupled modes Refer to previous work
Modal Analysis The spring-mass system set-up Each vertex is considered as a mass particle Each edge is considered as a damped spring
Modal Analysis Coupled spring-mass system to a set of decoupled modes
Modal Analysis A discretized physics system We use spring-mass system Small displacement, so consider it linear Stiffness Damping Mass Stiffness Damping Mass
Modal Analysis Rayleigh damping And diagonalizing Solve the Ordinary Differential Equation (ODE) Rayleigh damping And diagonalizing Now, solve this ODE instead
Modal Analysis Substitute (z are the modes) Solve the ODE Substitute (z are the modes) Now, solve this ODE instead
Modal Analysis General solution External excitation defines the initial conditions
Modal Analysis Assumptions In most graphics applications, only surface representations of geometries are given A surface representation is used in modal Analysis Synthesized sound appears to be “hallow”
Modal Analysis Summary An input triangle mesh A spring-mass system A set of decoupled modes
Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results
State Detection
State Detection Distinguishing between lasting and transient contacts In contacts? In lasting contacts?
Interaction Handling Lasting contacts a sequence of impulses Transient contacts a single impulse New slides to explain the object interaction to contact to impulse ()
Impulse Response Dirac Delta function as impulse excitation General solution with initial condition given by the impulse, we have: 37-39 link them smoothly
Impulse Response
Handling Lasting Contacts i.e. Frictional contacts How to add the sequence of impulses? The model has to be fast and simple, because…
Handling Lasting Contacts Update Rate: 100Hz Update Rate: 44100Hz
Handling Lasting Contacts The interaction simulation has to be stepped at the audio sampling rate: 44100 Hz The update rate of a typical real-time physics simulator: on the order of 100’s Hz Not enough simulation is provided by the physics engine An customized interaction model for sound synthesis
Our Solution Decompose the interaction into difference levels Different update rates at different levels Combined results offer a good approximation
Our Solution Three levels of simulation Macro level: simulating the interactions on the overall surface shape Meso level: simulating the interactions on the surface material bumpiness Micro level: simulating the interactions on the surface material roughness
Three-level Simulation Macro level: Geometry information Update rate: 100’s Hz Update rate does not need to be high The geometry information is from the input triangle mesh, and contacts are reported by collision detection in the physics engine. Live demo of only macro-level simulation enabled
Three-level Simulation Meso level: Bumpiness Bump mapping is ubiquitous in real-time graphics rendering Bump maps are visible to users but transparent to physics simulation
What Is Bump Mapping? Perturb vertex normals for shading No geometry details + = Image Courtesy to Wikipedia
Three-level Simulation Meso level simulation Makes sure visual and auditory cues are consistent Attends to surface bumpiness details Update rate: Event queue: 100’s Hz Event processor: 44100 Hz
Three-level Simulation Meso level simulation details (1) Event queue is update at 100’s Hz. Linear velocity and position information from the physics simulator. An event handler traverse back one time step to collect all “bumping” events in last time step
Three-level Simulation Meso level simulation details (2) Events from last time step are made up in this time at audio rate resolution. Latency: 10ms. 200ms latency tolerance (Bonneel et al. 08) Event Queue 1 step in physics simulation Event Processor Live demo of only meso-level simulation enabled and both macro and meso-level simulation enabled
Three-level Simulation Micro level simulation: Van den Doel et al. 01 Fractal noise is used to simulate the micro-level interaction Live demo of only micro-level simulation enabled And both micro, meso, and macro-level simulation enabled
Three-level Simulation Advantages: Fast and simple. Makes real-time sound synthesis driven by complex interaction possible. Captures the richness of sound varying at three levels of resolution Visual and auditory feedbacks are consistent
Outline Sound Simulation Pipeline Previous Work Contribution & System Overview Modal Analysis Interaction Handling System Implementation & Results
System Implementation Tablet support Material manipulation Users are allowed to change material parameters Testing “new materials” right away Material blending: linear interpolation Integration with a general physics engine and graphics engine Physics engine: Open Dynamics Engine (ODE) Graphics rendering engine: Open Source 3D Rendering Engine (OGRE)
Results and Possible Applications User interface demo Pre-loaded materials User adjustment of the parameters
Results and Possible Applications Virtual instrument demo Flexible - customized instrument with different material properties and sound qualities Complicated interaction is supported – not limited to percussion instrument
Summary
Summary An interactive and flexible work flow for sound synthesis A new frictional contact model for sound synthesis Natural and intuitive interface for manipulating and interacting with objects in the virtual environment
Future Work More natural and realistic material blending schemes More intuitive material selection tool (material pallet) Add room acoustic filters and 3D auralization Extend the surface mesh modal synthesis to tetrahedral meshes User study and validation Enabling tools for vision-impaired
Acknowledgements Army Research Office Carolina Development Foundation Intel Corporation National Science Foundation RDECOM