1. School of Informatics Auditory External Representations October, 2002Paul Vickers1 When Bugs Sing Adventures with Auditory External Representations of Programs Dr Paul Vickers School of Informatics, Northumbria University www.paulvickers.com/auralisation Professor James L. Alty Dean of Faculty of Science, Loughborough University

2. Auditory External Representations School of Informatics October, 2002Paul Vickers2 Whoah! Say what? Musical approach helps programmers catch bugs –New Scientist (2359, 7/9/02)New Scientist (2359, 7/9/02) –http://www.newscientist.com/news/news.jsp?id=ns99992757 Programming tool makes bugs sing –Technology Research News (7/8/02)Technology Research News (7/8/02) –http://www.trnmag.com/Page_One/2002/Page_One_080702.html

3. Auditory External Representations School of Informatics October, 2002Paul Vickers3 Introduction Background to the project Sound in HCI A case for music & musical grammars The cAitlIn system Demonstration Experiment Future work

4. Auditory External Representations School of Informatics October, 2002Paul Vickers4 Background Interests in music, computing, & programming Hunch that music could be used to assist in the programming process Develop and test a musical framework for communication of program information

5. Auditory External Representations School of Informatics October, 2002Paul Vickers5 Sound in HCI Increasing research focus since mid ‘80s (Sara Bly) Belief that non-speech audio useful as interaction medium Nascent field of auditory display spawns techniques like: –Sonification –Audification –Auralisation

6. Auditory External Representations School of Informatics October, 2002Paul Vickers6 Examples Auditory icons Earcons Data visualisation (sonification & audification) Assistive technologies –e.g. web browsers for the blind

7. Auditory External Representations School of Informatics October, 2002Paul Vickers7 Program auralisation Mapping of program data and events to sound –LogoMedia –Sonnet –ADSL (auditory domain specification language) –LISTEN –Jackson & Francioni (1992) parallel programs –Brown & Herschberger Largely untested (empirically)

8. Auditory External Representations School of Informatics October, 2002Paul Vickers8 Why music? Gives multiple audio streams in parallel Structured and organised according to specific rules (“organised sound”: Varese) Time-ordered. Visual techniques are spatial. Can music’s grammars be used to carry real information? Vickers, P. and Alty, J.L., Using Music to Communicate Computing Information. Interacting with Computers, 2002. 14(5): pp. 434-455.

9. Auditory External Representations School of Informatics October, 2002Paul Vickers9 Musical grammars? Difficulty with MGs (linguistically) is that vocabulary needs to be large –Nouns, adjectives, verbs But small vocabs are successful (Hankinson & Edwards, 2000) Some programming languages have small vocabs  good candidate for successful mapping

10. Auditory External Representations School of Informatics October, 2002Paul Vickers10 Theoretical framework Musical grammars as communication medium Mapping of syntactic features of Pascal to music Pascal construct hierarchy  motif hierarchy Motifs organised around tonal musical framework Use of surface & deep characteristics Meter & rhythm more useful than harmony Percussion adds emphasis Drones maintain context

11. Auditory External Representations School of Informatics October, 2002Paul Vickers11 Musical framework Parallel & structured Grammar allows separate musical components to work with rather than against each other Time-ordered –Visual techniques primarily spatial –Programs operate in temporal domain –Audio well suited to monitoring state changes over time

12. Auditory External Representations School of Informatics October, 2002Paul Vickers12 cAitlIn system Computer Audio Interface to Locate Incorrect Nonsense Uses musical motifs to render Pascal language constructs at run time Motif design is hierarchic as is organisation of constructs

13. Auditory External Representations School of Informatics October, 2002Paul Vickers13 Hierarchic motif design Vickers, P. and Alty, J.L., Musical Program Auralisation: A Structured Approach to Motif Design. Interacting with Computers, 2002. 14(5): pp. 456-484.

14. Auditory External Representations School of Informatics October, 2002Paul Vickers14 Design of motifs Points of interest –Construct Entry –Evaluation of condition –Execution of body –Exit from construct Boolean True=major, False=minor Drones

15. Auditory External Representations School of Informatics October, 2002Paul Vickers15 Selection tunes Metaphorically asking & answering a question. Tune rises on entry and descends on exit

16. Auditory External Representations School of Informatics October, 2002Paul Vickers16 Iteration tunes Major/Minor chord to show when condition evaluated Heard at start of WHILE loops and at end of REPEAT loops FOR loops increase/decrease the pitch each time round the loop

17. Auditory External Representations School of Informatics October, 2002Paul Vickers17 Variations All selections use same basic tune, modified for individual constructs All iterations use another basic tune, again modified

18. Auditory External Representations School of Informatics October, 2002Paul Vickers18 Auditory parentheses As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs

19. Auditory External Representations School of Informatics October, 2002Paul Vickers19 Other statements All other statements are currently represented by a drum beat, one per statement.

20. Auditory External Representations School of Informatics October, 2002Paul Vickers20 cAitlIn musical grammar Uses motif classes and variations to render Pascal constructs Allows –abstraction –nesting (tonality & harmony preserved across motifs) Uses grammatical rules (e.g. a tonal music system)

21. Auditory External Representations School of Informatics October, 2002Paul Vickers21 cAitlIn musical grammar cont Renders information not directly communicated by source code or other visualisation techniques –e.g. construct persistence (nesting)  a new grammar, conveys –extra meaning –same meaning but from different perspective

22. Auditory External Representations School of Informatics October, 2002Paul Vickers22 Advantages Over Auditory Icons –symbolic mappings = poor system aural ecology Syntactic design  expect retention Displays both flow and state An execution-specific program structure diagram

23. Auditory External Representations School of Informatics October, 2002Paul Vickers23 Demonstration

24. Auditory External Representations School of Informatics October, 2002Paul Vickers24 Examples Generic selection motif Generic iteration motif

25. Auditory External Representations School of Informatics October, 2002Paul Vickers25 IF (true) a := 10 ; IF a = 10 THEN Writeln (‘A is 10’) ;

26. Auditory External Representations School of Informatics October, 2002Paul Vickers26 IF…ELSE (false) a := 1 ; IF a = 10 THEN Writeln (‘A is 10’) ELSE Writeln (‘A is not 10’) ;

27. Auditory External Representations School of Informatics October, 2002Paul Vickers27 CASE…ELSE match a := 3 ; CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ; END ;

28. Auditory External Representations School of Informatics October, 2002Paul Vickers28 WHILE 2 iters cntr := 1; WHILE cntr <= 2 DO cntr := cntr + 1 ;

29. Auditory External Representations School of Informatics October, 2002Paul Vickers29 REPEAT 3 iterations cntr := 1 ; REPEAT cntr := cntr + 1 ; UNTIL cntr > 3 ;

30. Auditory External Representations School of Informatics October, 2002Paul Vickers30 FOR…TO 6 iters FOR cntr := 1 TO 6 DO Writeln (cntr) ;

31. Auditory External Representations School of Informatics October, 2002Paul Vickers31 Sequence & nesting Clue to nesting is in the drone If you hear a continuous note then you are inside a construct. Each construct has a drone. For nested constructs you hear multiple drones simultaneously Drone pitch increases as nesting deepens

32. Auditory External Representations School of Informatics October, 2002Paul Vickers32 Example program Program Exemplar VAR counter : Integer ; BEGIN counter := 1 ; WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ; END.

33. Auditory External Representations School of Informatics October, 2002Paul Vickers33 Score

34. Auditory External Representations School of Informatics October, 2002Paul Vickers34 Score

35. Auditory External Representations School of Informatics October, 2002Paul Vickers35 Structure Diagram

36. Auditory External Representations School of Informatics October, 2002Paul Vickers36 Experiment 1 Listening test to assess comprehensibility of motifs 22 subjects, 60 auralisations Recognition rate better than by chance and with very little training Hierarchic organisation allowed class level identification & abstraction Vickers, P. and Alty, J.L., Musical Program Auralisation: A Structured Approach to Motif Design. Interacting with Computers, 2002. 14(5): pp. 456-484.

37. Auditory External Representations School of Informatics October, 2002Paul Vickers37 Experiment 2 – Bug location Questions –Do subjects locate more bugs with the additional auralisation information than without? –Does the musical experience of subjects affect their ability to make use of program auralisations? 22 subjects, 8 programs, 4 auralised, 4 normal Given: spec, outline design, input, output, & expected output Then, ten minutes to locate bug with either –Above documentation, or –Above documentation + auralisation Vickers, P. and Alty, J.L., When Bugs Sing. Interacting with Computers in press, 2002.

38. Auditory External Representations School of Informatics October, 2002Paul Vickers38 Experiment 2 – Bug location Experimental design (within subjects, repeated measures) –Group A had 1, 3, 5, & 7 without auralisation and 2, 4, 6, & 8 with auralisation –Group B had 1, 3, 5, & 7 with auralisation and 2, 4, 6, & 8 without

39. Auditory External Representations School of Informatics October, 2002Paul Vickers39 Experiment 2 – Bug location Problem with complexity (1, 3, 5, & 7 as a group had higher complexity than other programs)

40. Auditory External Representations School of Informatics October, 2002Paul Vickers40 Experiment 2 – Bug location When results adjusted for complexity, observe a significant difference in debugging scores: t(21)=2.10, p<0.05 Bug location time not affected:t(21)=1.98, p=0.06 Auralisations led to increase in subjects perceived workload (measured with NASA TLX): t(21)=2.12, p=0.046 Musical knowledge/experience had no effect: regression model 0.16 <= p <= 1.71

41. Auditory External Representations School of Informatics October, 2002Paul Vickers41 Experiment 2 – Bug location Best candidates were programs –where output contained no clues to bug location –with a high complexity, complex boolean expressions

42. Auditory External Representations School of Informatics October, 2002Paul Vickers42 Limitations No comparison with visual task yet Cognitive latency of motifs? No longitudinal results

43. Auditory External Representations School of Informatics October, 2002Paul Vickers43 Future work Follow-up studies Extend the grammar to include assignment statements and procedure calls Assistive technologies-blind programmers? Longitudinal studies

44. Auditory External Representations School of Informatics October, 2002Paul Vickers44 More auralisation? Is further auralisation needed? –Perturbations in flow pinpoint general region of bug. –Non-flow bugs very hard to detect anyway & require variable inspection Inspection difficult to render in audio

45. Auditory External Representations School of Informatics October, 2002Paul Vickers45 Real grammar? Translation or transliteration? –Have we simply rendered written Pascal as musical Pascal (e.g. written vs spoken English) –Like writing “I want to go to bed” as “Ich will gehen zum Bett” (transliteration) Is there a new grammar? –“Ich will zum Bett gehen” (translation)

46. Auditory External Representations School of Informatics October, 2002Paul Vickers46 In progress Preparing EPSRC bid to investigate Auditory External Representations (c.f. EPSRC CRUSADE project at COGS in Sussex - du Boulay et al)

47. Auditory External Representations School of Informatics October, 2002Paul Vickers47 Major features With cAitlIn we have: –A musical equivalent for various syntactic components of a programming language = musical grammar –Extended the earcon technique (q.v.) –An audio program structure diagram (execution path)

48. Auditory External Representations School of Informatics October, 2002Paul Vickers48 Major features –A MPAS usable by non-musicians –Shown that music can be used to communicate information about program state and structure –In turn, this info can assist bug location Empirical evaluation Organising principles Opened new line of enquiry & research Auditory External Representations

49. School of Informatics October, 2002Paul Vickers49 www.paulvickers.com/auralisation paul.vickers@northumbria.ac.uk