1. A Roadmap: Beyond Big API’s Why Software Component Frameworks May Be Interactive Audio’s Future Chris Grigg chrisg@control-g.com

2. Thesis Big audio API’s compromise delivered audio quality To use them, you have to call their functions; To use them, you have to call their functions; Every API uses a different interface; Every API uses a different interface; So Mixing or Changing APIs is too hard. So Mixing or Changing APIs is too hard. Many custom API’s & media types have no tools. Many custom API’s & media types have no tools. This sucks because: No single API does everything well, or runs on all platforms. Time invested in custom API features doesn’t port. No single API does everything well, or runs on all platforms. Time invested in custom API features doesn’t port.

3. Not about specific faults of particular API’s Not about specific faults of particular API’s A Structural Problem of all big audio API’s A Structural Problem of all big audio API’s Programming itself the bottleneck. Programming itself the bottleneck. Thesis

4. We want to be able to construct runtime audio systems out of arbitrary functional blocks from multiple vendors Our Studios Have Many Different Synthesizer Brands Because They’re All Good At Different Things – Our Runtime Systems Need Similar Diversity

5. In this ‘What If’ world: Units of software organization become Smaller Units of software organization become Smaller They become Modular They become Modular Each Unit Takes On a Specific Function Each Unit Takes On a Specific Function Units Have to be able to Talk to one another Units Have to be able to Talk to one another An Architecture Fundamentally Different from our current big, do-everything Audio API development & use practices.

6. Things We Need To Get There 1 Software Component Framework with Object Model 2 Audio Signal Interconnect Mechanism 3 Control Signal Interconnect Mechanism 4 Scriptable Control

7. 1 A Standardized Audio Software Component Framework with Object Model

8. If our audio blocks are components, we need a framework to hold them.

9. Typical Component Framework Framework Software Component A Manager Maintains Component Instances & Directs Host Calls Instances of Various Component Classes Software Component B Software Component C Software Component D Host Application Framework Manager

10. GUI Authoring Tool Uses Same Framework & Components As Runtime Select, Configure, & Connect Components Save Configuration for Use at Runtime

11. Having a Uniform, Implementation-Independent Object Model allows: Single, Universal Authoring Tool Single, Universal Authoring Tool Cross-Platform Content Cross-Platform Content Cross-Platform Runtime Communication Cross-Platform Runtime Communication

12. 2 & 3: Interconnection Audio Signals & Control Signals

13. Connecting Anything to Anything Requires Control & Audio Interconnect Standards

14. Most Component Frameworks Don’t Have Interconnections to Deal with No off-the-shelf solution – We need a development effort

15. 2 Audio Interconnection Mechanism Between Components

16. Audio Signal Flow Between Components …Doesn’t Require Multiple Buffers

17. Audio Signal Flow Using Single Buffer Multiplexed Operations on a Single Buffer

18. Representing Interconnections in the Framework Object Model

19. Block Diagram View

20. Object Model View: Components and Connections Everything derives from class Component and class Connection

21. Benefits I Better Programmer/Soundmaker Communication Better Programmer/Soundmaker Communication Increases Clarity of Purpose Increases Clarity of Purpose Simpler, Faster Code Development Simpler, Faster Code Development Promotes Audio API Proliferation Promotes Audio API Proliferation Encourages View of Audio APIs as Interchangeable Encourages View of Audio APIs as Interchangeable Simplifies Porting of Game Engines and Titles Simplifies Porting of Game Engines and Titles Encourages Porting Components to New Platforms Encourages Porting Components to New Platforms

22. A Message-Based Mechanism for Control Signal Interconnection Between Software Components 3

23. Uh… why “control signals”?

24. To avoid messes like this. Rat’s Nest of Function Calls

25. Like MIDI Cables for Software Components Unified Control Signal – Single-Point Connection

26. Assemble Controlled Sound Systems from Software Components Like a Recording Studio: Audio Connections & Control Connections

27. Host Game Uses Same Control Mechanism Still Like a Recording Studio: Audio Connections & Control Connections

28. Components Can Process Control Signals (not just audio) Examples: Y-Cord, Merger, Adaptor

29. Easy Communication with External Devices Using Same Model Consistent Binary Control Signal Format Makes This Possible

30. Multi-Component ‘Templates’ Add Configuration Power Example: Mixer Channel Strip Template Many Components Instantiated in Single Step with Internal Routing, Configuration, Initial Settings

31. Control Signals Give Us Knobs Both the Programmer and the Soundmaker

32. So what kinds of messages are in the Control Signals?

33. Standardize the Control Signal Connection But Not the Command Set, Parameter Set, etc.

34. Arbitrary Messages On The Connection How Do You Know What Messages a Component Accepts? You Ask It. Don’t Predefine the Message Vocabulary. Let Every Component Define Its Own.

35. Every Component Has Its Own Command Set …And Publishes It This Is How The GUI Authoring Tool Knows What Each Component Can Do.

36. Simple Piano Component Two Control Inputs Simple Example

37. Simple Piano Component Control Interface Description ControlInput1: Name: MIDI In Protocol: MIDI Messages Interpretation: (MIDI Implementation Chart appears here ) ControlInput2: Name: Sustain Switch In Protocol: Boolean Messages Interpretation: low bit 1 = pedal down, 0 = up Ask the Component for its Interface Description and you get a list of the Messages that its Control Inputs Understand

38. 3D Panner Component One Control Input More Complex Example

39. 3D Panner Component Control Interface Description Commands: SetVolume( Fixed16 ) -- Master out volume Mute( Boolean ) -- Mute (TRUE) or Pass audio SetSpatialPosition( Std3DVector ) -- 3-space The Messages that the Control Input Understands

40. But Weren’t We Trying to Get Away from Function Calls? Commands: SetVolume( Fixed16 ) -- Master out volume Mute( Boolean ) -- Mute (TRUE) or Pass audio SetSpatialPosition( Std3DVector ) -- 3-space This looks just like a Big Audio API!

41. True, this semantic is similar to the HLL function calls... But it’s expressed differently at runtime. SetVolume( Fixed16 ) Mute( Boolean ) SetSpatialPosition( Std3DVector ) Not as compile-based function calls, but rather in Data-Based Messages that can be routed dynamically at runtime.

42. How do Components Publish their Interface Descriptions at Runtime and Authoring Time? Some Invention Is Required Here. Learn from existing solutions to similar problems: COM COM CORBA’s IDL CORBA’s IDL Apple Event Terminology Resources Apple Event Terminology Resources XML’s DTD XML’s DTD MAX Objects MAX Objects

43. 4 Adding the Scripting Layer

44. Scripted Control of all Audio Components & Media With Enough Component Types and Media Types, the ‘Ultimate Interactive Audio Architecture’ Sound Dept. Delivers Platform-Independent Media & Scripts Components, Framework & Script Player Get Ported to All Platforms Game Uses Same Sound Event Cues On All Platforms

45. Benefits II Simplified Control of Sound From Game Simplified Control of Sound From Game Increased API Experimentation Flexibility Increased API Experimentation Flexibility Flexible Patching of Control Sources & Destinations Flexible Patching of Control Sources & Destinations Encourages Plug-In Style Development Encourages Plug-In Style Development Ability to Transform Game Control Variables Ability to Transform Game Control Variables Sound Designer Control over Interactivity Sound Designer Control over Interactivity Auto-Porting of Interactive Audio Content Auto-Porting of Interactive Audio Content

46. Conclusion An essentially non-commercial idea: The limits of big audio APIs are structural. The limits of big audio APIs are structural. They favor programmer contributions while marginalizing soundmaker contributions – through poor or missing tools, but also by virtue of their fundamental compile-based architectures. They favor programmer contributions while marginalizing soundmaker contributions – through poor or missing tools, but also by virtue of their fundamental compile-based architectures. This dynamic hurts the delivered audio quality. This dynamic hurts the delivered audio quality. Open standards for component software appear to offer a way out, and might serve as the basis of a generalized interactive audio methodology as universal and stable as MIDI has been. Open standards for component software appear to offer a way out, and might serve as the basis of a generalized interactive audio methodology as universal and stable as MIDI has been.

47. Because Media is a Harsh Master.