1. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers1 Program auralisation Mapping of program data and events to sound –LogoMedia –Sonnet –ADSL (auditory domain specification language) –LISTEN –Jackson & Francioni (1992) parallel programs –Brown & Herschberger Largely untested (empirically)

2. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers2 Example (Vickers & Alty, 2002a, 2002b)

3. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers3 Motif design Points of interest –Construct Entry –Evaluation of condition –Execution of body –Exit from construct Boolean True=major, False=minor Drones

4. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers4 Selection tunes Metaphorically asking & answering a question. Tune rises on entry and descends on exit

5. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers5 Iteration tunes Major/Minor chord to show when condition evaluated Heard at start of WHILE loops and at end of REPEAT loops FOR loops increase/decrease the pitch each time round the loop

6. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers6 Variations All selections use same basic tune, modified for individual constructs All iterations use another basic tune, again modified

7. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers7 Auditory parentheses & other statements As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs All other statements are currently represented by a drum beat, one per statement.

8. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers8 IF…ELSE (false) a := 1 ; IF a = 10 THEN Writeln (‘A is 10’) ELSE Writeln (‘A is not 10’) ;

9. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers9 WHILE (2 iters) cntr := 1; WHILE cntr <= 2 DO cntr := cntr + 1 ;

10. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers10 FOR (6 iters) FOR cntr := 1 TO 6 DO Writeln (cntr) ;

11. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers11 Sequence & nesting Clue to nesting is in the drone If you hear a continuous note then you are inside a construct. Each construct has a drone. For nested constructs you hear multiple drones simultaneously Drone pitch increases as nesting deepens

12. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers12 Example program Program Exemplar VAR counter : Integer ; BEGIN counter := 1 ; WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ; END.

13. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers13 Assistive technology Visually-impaired cannot make same use of VDUs as those with (corrected-to)normal eyesight Web browsers for the blind –E.g. Brookestalk http://www.brookes.ac.uk/schools/cms/research/speech/btalk.htm http://www.brookes.ac.uk/schools/cms/research/speech/btalk.htm Diagram readers –Audiograph (Rigas & Alty, 1997) Musical representation of chemical compounds In what ways do you think sound could help people with vision impairments? How would these techniques benefit the rest?

14. CG087 Time-based Multimedia Assets School of Informatics Auditory displayDr Paul Vickers14 References Brewster, S. (1998). “The Design of Sonically-Enhanced Widgets.” Interacting with Computers 11: 211-235. Fitch, W. T. & Kramer, G. (1994). Sonifying the Body Electric: Superiority of an Auditory over a Visual Display in a Complex, Multivariate System, in Auditory Display. G. Kramer. Reading, MA: Addison-Wesley. XVIII: 307-326 King, R. D. & Angus, C. (1996). “PM—Protein Music.” CABIOS 12(3): 251-252. Rigas, D. I. & Alty, J. L. (1997). The Use of Music in a Graphical Interface for the Visually Impaired, in Proc Interact '97, Sydney, Chapman & Hall, London. Sturm, B. L. (2002). Music from the Ocean, Composerscientistrecordings www.composerscientist.com. www.composerscientist.com Vickers, P. & Alty, J. L. (2002a). “Musical Program Auralisation: A Structured Approach to Motif Design.” Interacting with Computers 14(5): 457-485. Vickers, P. & Alty, J. L. (2002b). “When Bugs Sing.” Interacting with Computers 14(6): 793-819.