2. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Advanced Granular Synthesis for Next-Generation Games Game Developer's Conference 2007 San Francisco Leonard J. Paul Vancouver Film School Game Audio Instructor info {at} VideoGameAudio.com 604-685-5808 x4035

3. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Overview 1) How to utilise granular for speech, sound effects, music and engines 2) Prototyping granular synthesis using Pure Data and Open Sound Control 3) How to add granular synthesis to your sound driver

4. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Using granular for speech sound effects music and engines 1)

5. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com 1) Selection order (fwd/rev or freeze) 2) Pitch shift 3) Amplitude range 4) Spatialization / panning 5) Grain duration 6) Grain density (grains/sec, # of grain voices) 7) Envelope (ASR shape or windowing function) 8) DSP effect (reverb, filtering...) 9) Feedback amount (for granular delay lines) Granular Parameters

6. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Game Parameters ● Granular parameters driven by game state ● (ex. RPM -> pitch) ● Physics allow for continuous control over playback of sample ● (ex. ragdoll -> foley) ● Input parameters need to be carefully tuned to drive synthesis ● (ex. instrument played by novice)

7. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Game Audio Data Diagram Game (Physics, GFX, AI etc..) Game Audio Code Granular Driver Code DAC  Commercia l Sound Tool In-house Sound Tool Audio Data

8. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Speech ● Time warping speech for variation ● Change formants of voice ● Realtime granular effects to change character of voice

9. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Sound Effects ● Crossfade morph via grains ● Spacialization in 5.1/7.1 ● Contact physics for sliding/scraping ● Add variation to existing sounds

10. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Engines ● Add as a layer for variation ● Pre-segmentation and playback via RPM/load band: Load RPM Samples Low RPM & low load grains

11. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Music and Ambiences ● Stretch a sound for a granular drone ● Tempo-sync grains for interesting gating effects ● Merge ambiences by selecting grains from multiple files ● Making an ambience sound more musical and abstract

12. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Prototyping Granular Synthesis using Pure Data and OSC 2)

13. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Coding PD Objects ● FlexT is a cross-platform method of generating custom C++ PD objects: http://grrrr.org/ext/flext/ ● Easily possible to port part of the Synthesis Toolkit (STK) to PD using Code Blocks or any other DSP code

14. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Flext (gain~ object code) class gain: public flext_dsp { FLEXT_HEADER(gain,flext_dsp) float factor; gain(): factor(0) { AddInSignal("In"); // add signal inlet AddInFloat("Gain"); AddOutSignal("Out"); // add signal outlet FLEXT_ADDMETHOD(1,m_gain); } virtual void m_signal(int n,t_sample *const *insigs,t_sample *const *outsigs) { const t_sample *in = insigs[0]; t_sample *out = outsigs[0]; while(n--) *(out++) = *(in++)*factor; } void m_gain(float db) { factor = pow(10.f,db*0.05f); } FLEXT_CALLBACK_F(m_gain); };

15. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com OSC Layer for Prototyping ● Add an Open Sound Control (OSC) layer to your game to communicate with a rapid prototyping audio environment such as PD ● Example of using OSC with a Half-life 2 Source Mod ● Possible to compile PD to a language such as Java (pd2j2me)

16. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com OSC Code for Sending Msg #include "osc/OscOutboundPacketStream.h" #include "ip/UdpSocket.h" #define ADDRESS "127.0.0.1" // localhost but can be any address #define PORT 7000 #define OUTPUT_BUFFER_SIZE 1024 int main(int argc, char* argv[]) { UdpTransmitSocket transmitSocket( IpEndpointName( ADDRESS, PORT ) ); char buffer[OUTPUT_BUFFER_SIZE]; osc::OutboundPacketStream p( buffer, OUTPUT_BUFFER_SIZE ); // create buffer p << osc::BeginMessage( "/test" ) << "hi world" << osc::EndMessage; // message transmitSocket.Send( p.Data(), p.Size() ); // send message }

17. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Adding OSC to Half-life 2 ● Download the OSCpack library from: http://www.audiomulch.com/~rossb/code/oscpack/ ● Download Visual C++ 2005 Express, Half- Life 2 and create a new mod ● Add the OSC files to the mod source (some tweaking needed) and modify SoundEmitter.cpp etc. ● This is a similar process to adding OSC to your own sound driver

18. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Adding Granular Synthesis to your Sound Driver 3)

19. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Granular Sound Driver ● PD C++ code can be ported to the next- gen consoles as the consoles are primarily software based ● Synthesis should be creatively used in conjunction with samples for sound design ● Advantage is that real-time control over synthesis parameters is from game parameters

20. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Granular Code Examples Ross Bencina – Granular Delay Line ● Uses Borland C++ Builder http://www.audiomulch.com/~rossb/ Synthesis Toolkit (STK) – Granulate ● Can use flext to port to PD http://ccrma.stanford.edu/software/stk/ SyncGrain~ for PD (from Csound) ● Ported using flext http://footils.org/cms/pms/

21. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Related Research ● Concatenative Sound Synthesis Sound "Mosaic" Analysis & Resynthesis Uses grains from one sample to approximate grains of a second sample www.mat.ucsb.edu/~b.sturm/sand/VLDCMCaR/VLDCMCaR.html

22. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Related Research ● Natural Grain Resynthesis Make grains that preserve transients http://www.cs.ubc.ca/~reynald/naturalgrains.html

23. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Related Research ● Pure Data Compiler (pd2j2me) Compile PD code to Java http://www.uow.edu.au/~mh675/publications/nime_2005.pdf public void met0Bang() { double a; a = counter.bang(); a = a * 3; a = a / 4; }

24. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Example ● Example of gameplay utilizing granular synthesis

25. Web:: VideoGameAudio.com Email:: info{at}videoGameAudio.com Questions? Email:info-at-VideoGameAudio.com Web:www.VideoGameAudio.com