2. 1 The NESS Project Work supported by the European Research Council MUSICA Seminar Series University of Edinburgh 13 February 2013 Stefan Bilbao, Paul Graham, Alan Gray, Brian Hamilton Kostas Kavoussanakis, James Perry, Alberto Torin, and Craig Webb Acoustics Group EPCC

3. 2 Background 2

4. 3 3

5. 44 The HPC Centre of the University of Edinburgh –70 staff –£4M turnover (almost) all from external sources HPC Research Training Visitor Programmes European Projects Facilities Technology Transfer

6. 5 Distributed digital asset management system –Accurate supervision of the progress of a film production –Secure asset transfer between entities working on the film –Complements existing film editing software (e.g. Nucoda) –Software support for current day-to-day manual workflow processes Case Study: FilmGrid 5 Photo by Brad & Sabrina

7. 6 Case Study: FilmGrid 6 Photo by Jeremy Keith

8. 77 Academic: –Permanent staff with degrees in various backgrounds: Physics, Computer Science, Mathematics, Chemistry, Biology… –Education and Training provider: MSc in HPC Bespoke training on HPC, accelerators, multi-core computing Strengths: –Project management –Interdisciplinary projects: small pilots, distributed programmes –User support Technical: –Software development for academia and industry –Code optimisation (serial, parallel, accelerators) –Simulation and modelling –Wide language, operating system and computer architecture expertise http://www.epcc.ed.ac.uk/

9. 8 NESS An ERC funded project (PE6: Informatics), running at Edinburgh, 2012—2017. The goal: digital sound synthesis… …of musical sound Today: –Digital sound synthesis and physical modeling –Ness project: structure and activities –Group member presentations

10. 9 Digital Sound Synthesis A longstanding attempt to move away from the use of recorded sound (sampling) Analogous to computer graphics rendering? At the philosophical level…yes At the technical level…not really! “Sampling” “Synthesis”

11. 10 Abstract Digital Sound Synthesis Early synthesis methods (1950s—1960s): based on simple heuristic building blocks--- efficient and easy to program… Frequency modulation synthesis: a very successful variant! Still extremely popular! Can be difficult to control (lots of user input), and sound quality is generally very artificial! Wavetables: Sinusoidal Oscillators: FM trumpet FM bell

12. 11 Physical Modeling Sound Synthesis Physical models: –based on physical descriptions of musical “objects”… –can be computationally demanding… –potentially very realistic * sound –control parameters: few in number, and perceptually meaningful * realism needs a good definition, if there is not a real-world reference! A fair degree of hybridization abounds: physical modeling + sampling is analogous to, say, motion capture!

13. 12 Methods: Lumped Mass Spring Networks networks of masses/springs/dampers + simple ODE solver basis for “Cordis” and “Cordis Anima” systems (Cadoz and collaborators, Grenoble, from 1979 to present!) generally abstract (modular), but if put in a regular arrangement, it is possible to simulate distributed objects (strings, membranes, etc.) Earliest large scale attempt at physical modeling synthesis: Cymbal Timpani

14. 13 Methods: Modal Synthesis b asis for “Modalys” synthesis system (IRCAM, 1985—present) geared towards linear objects (with interesting extensions to the nonlinear case…Volterra series, e.g.) a lot of offline precomputation (modal shapes, frequencies) A different approach: decompose dynamics of vibrating object into modes…

15. 14 Methods: Digital Waveguides Yet another approach: decompose dynamics of vibrating object into traveling waves… developed at CCRMA, Stanford University, 1985— present roots in early scattering-based speech synthesis methods (Kelly Lochbaum, 1962) basis for many synthesis systems, including Yamaha VL1 (1994) meant for simulating distributed linear objects in 1D (strings, acoustic tubes)…extremely efficient! Guitar

16. 15 Methods: Time-stepping Methods (FDTD, FVTD, etc.) The obvious approach: represent dynamics of vibrating object on a grid and integrate using direct solvers… distinct roots in musical acoustics investigations, and a few early synthesis attempts (Ruiz, vibrating string, 1969!) tools exist to handle virtually any system…much more general than other methods but…a lot of specialization work for audio applications…

17. 16 NESS: Target Systems Brass Instruments Electromechanical Instruments Nonlinear Plate and Shell Vibration Trying to span the full range of musical acoustic systems… difficult to approach using other physical modeling techniques… Trumpet Gong Cymbal

18. 17 NESS: Target Systems Modular Synthesis Environments Embeddings and Spatialization Room Acoustics Modelling Excerpt: Orbit, G. Delap, 2009 Snare Drum

19. 18 Sample rates and bandwidth A basic constraint for audio: choice of sample rate F s, and time step k = 1/F s Constraint 1 (necessary): Need to be able to fully render audio up to limits of human audio perception, so: F s ≥ 40 000 Hz, k ≤1/40000 s Constraint 2 (desirable): Don’t want to render audio above this range, for efficiency reasons, so: F s ≈ 40 000 Hz, k ≈ 1/40000 s Time step is small…lots of computational work to do… Some aspects of time stepping methods need to be reconsidered in this light!

20. 19 Grids and Bandwidth Limitation Suppose operation at a given sample rate…need to choose the grid carefully, for perceptual reasons: decrease in operation count Increase in grid spacing decrease in output bandwidth an additional constraint in numerical design…certain techniques (grid refinement) are dangerous in an audio context x 2 x 4 Simple string model…

21. 20 Audibility of Numerical Dispersion Example: thin bar, simple explicit FD method: Mistuning! Exact: 44100 Hz: Phase velocity Exact and Numerical Careful design necessary…methods with free parameters allow a means of tuning the scheme…at the price of linear system solutions (hard on GPU!) Numerical Dispersion: speed of wave propagation is incorrect, numerically!

22. 21 Numerical Instability A critical concern in synthesis design for non expert users… Linear membrane instability Nonlinear shock wave instability Not too hard to deal with… Harder to deal with without compromising audio output…

23. 22 Modular Instability Consider two rudimentary systems Problems can appear here if the connection is not handled properly: Stable Connection Unstable Connection Mass/ spring Ideal String Difficulties are compounded for more complex systems…

24. 23 Energy-based Stability Numerical energy conserved to machine accuracy: Extremely useful in debugging, and in designing complex modular systems: …giving a stability guarantee

25. 24 Computational Costs and HPC Audio sample rates are high: 44 100 Hz, 48 000 Hz, 92 000 Hz… Flop rates/memory requirement scale as power of sample rate (2,3,4)… arithmetic operations/second output, at 48 000 Hz: optimal realtime performance on commercially available single core Nonlinear plates/shells Brass instruments Electromechanical Instruments 10 6 10 7 10 8 10 9 10 10 11 10 12 10 13 10 14 10 15 10 16 10 17 Small embeddingsSmall rooms Concert Halls Musical use/experimentation: reasonable compute time (no overnight jobs!) Solutions: Parallel implementations (GPU, e.g.) New algorithmic issues: parallelizability, memory management, stability in finite precision

26. 3-D Room Acoustics (Brian Hamilton, Acoustics Group) 25

27. Simulating 3-D Room Acoustics A wave is a spatial field that changes over time Sound propagates as a pressure wave Simulating sound wave propagation: –Pick some 3-D lattice (grid) of points –Calculate sound pressure at each point –Iterate in time... Problem to solve: –How to do this as efficiently as possible? –Any audible artifacts? How to minimise them? 26

28. Spatial Lattices 27 Which to choose?

29. Spatial Lattices 28 Many choices...

30. Spatial Lattices 29 Waves should propagate uniformly in every direction Symmetry is key! Stacking fruit

31. Numerical Dispersion Numerical dispersion → wave speed error Simulated waves propagate along axes of grid Wave speed depends on grid orientation!

32. Example: Without Dispersion 31

33. Example: With Dispersion 32

34. Wave Speed Error 33 We want the error to be isotropic (direction independent) Delicate cancellation of error in space and time

35. Wave Speed Error 34

36. Wave Speed Error 35

37. Audible Artifacts: Examples 36 Castanet (clean) Castanet (dispersive) Guitar (clean) Guitar (dispersive)

38. 37 Percussion Instruments MembranesPlates and Shells L Low excitation! NL High excitation! L = Linear, NL = Non-linear (Alberto Torin, Acoustics)

39. 38 Linear Plates w = transverse displacement  = density, H = thickness, D = stiffness parameter - There is no interaction between different modes!

40. 39 Non-linear Plates Add non-linear terms to previous equation F = Airy’s function, E = Young’s modulus von Kármán equations for non-linear plates

41. 40 Non-linear Plates - Energy exchange between different modes is allowed! - Crashes, Pitch glide effects

42. 41 Air coupling Add the pressure on the plate Introduce the acoustic field , that obeys the wave equation Add coupling conditions between the air and the plate

43. 42 Numerical schemes Stability and Energy conservation Need for a Fast algorithm – Bottleneck of the code is the solution of a sparse linear system (matrices involved have a few non-zero entries) – We can use iterative solvers… works well for the simple plate needs extra work when air coupling is present

44. 43 Example: MultiPlate3D Roll gesture Several strikes

45. 44 What is a GPU? Graphics processing unit –Originally designed for rendering 3D graphics fast –Now also used for general purpose computations (GPGPU) Very well suited for problems like ours –Especially the 3D ones –Same simple computation required for huge number of points 44

46. 45 Porting Process Port from Matlab to C –Faster than Matlab, will run anywhere –Easier to debug and modify than CUDA –Good basis for CUDA port 45 Matlab C CUDA Optimized CUDA Port from C to CUDA –Some code (e.g. setup code) remains in C –Time critical main loop is ported to CUDA Optimize CUDA code –Gain high performance (as far as possible)

47. 46 Example 3dabc code –Simulates a 3D box of air with various boundary conditions Run times: 46 VersionRun time (s)Speed up Matlab311x C0.44170x CUDA0.0786395x Matlab version not optimised Small simulation size - would expect larger speed-up from C to CUDA for large size

48. 47 Large-scale 3D virtual acoustics (Craig Webb, Acoustics) Computing 3D wave propagation in a virtual space. Dynamic simulations, with full wave behavior. Can inject dry audio to produce reverberation. Or embed virtual instruments.

49. 48 Computation Size At a sample rate of 44.1kHz : 1 cubic metre requires 422 thousand grid points. 1 second of output requires 185G floating-point operations. 2,000 cubic metres – 370T operations. A 10 second simulation – 3.7P operations, that’s 3,700,000,000,000,000. Use multiple GPU cards to accelerate the model. With 4 cards, speedup over serial C code is in the range x100 ~ x140. Under an hour per second of simulation, instead of 5 days in serial C. Requires 10Gb of memory at single precision.

50. 49 Hall Simulation: Dry Audio Input Audio examples of 2,000 cubic metre hall 1.Dry guitar input : Output : 2.More guitars : 3.Opera singer (anechoic) : Output : 1.Can move sound around during runtime :

51. 50 Embedded Instruments Timpani Drum The timpani drum is a good test case for 3D physical modeling. We use a non-linear circular membrane, attached to a parabolic shell with fixed boundaries. This is then placed inside the room simulations, and we can model any number of timpani inside the space. These can then be played together, by specifying the timing and type of strikes on each drum. Audio examples One Timpani : Two Timpani : Three Timpani : Four Timpani :

52. 51 Creative Uses: Composition A new world of sound for musicians and composers---fully multichannel, synthetic music environments But---a learning curve! As for any mature instrument design…

53. 52 Control and Interfaces NESS: audio only! Not really any attempt at building live, performable instruments, or developing complex interfaces… Two subsequent levels of work: –Figuring out useful, parsimonious ways of representing input –UI design (simple!)

54. 53 NESS Thank you for your attention Questions? 53