Grasping Graphs by Ear: Sonification of Interaction with Hidden Graphs Leena Vesterinen Department of Computer Sciences University of Tampere Finland March, 2005 AAFG 2005

The goal of the game “Hidden Graphs” is a blind inspection of the hidden graphs. Player is to capture as many features as possible in the graph following the guiding sound signals. 3 different sounds are used to guide the player actions in detection of the hidden graphs. 3 major concepts are employed: Capture radius, Directional- predictive sound signals (DPS) and Basic behavioral patterns (BBP). Player task: Player is to choose the right capturing strategy and recognize the hidden graph. Researcher task: Researcher is to optimize the “dialogue” with player through basic behavioral patterns coordinated to directional- predictive sound signals, and to facilitate shaping the personal behavioral strategy. L. Vesterinen p 02_ Grasping Graphs by Ear

L. Vesterinen p 03_ Grasping Graphs by Ear For instance, by grasping virtual graphs, children would develop skills in cross-modal coordination: The use of special basic behavioral patterns for efficient inspection primarily within the game field, and learning how and when is suitable to apply one or another gesturing in dependence on discovered features. Basic cognitive processes: Sound feedbacks learning and experience should progress through the game from the concrete level to more abstract level. Gradual improvement on hearing of feedbacks (and/or haptics) should finally form the personal Behavioral strategy.

L. Vesterinen p 04_ Sonification: Use of non-speech audio to convey information. The sonification field is composed of the following three components: (1) psychological research in perception and cognition; (2) sonification tools for research and application; (3) sonification design and application. In particular, sonification is a potential solution for communication and interpretation of data. Plenty of research is done on sonification and applied to numerous application domains: to provide navigation cues, information visualization (charts and graphs) and non-visual drawing. Regarding to our study - remarkable work is done by B.N. Walker and J. Lindsay, 2004 based on the work of Tran et al., [B.N. Walker and J. Lindsay, 2004] [Tran et al. 2000] [Walker et al., 2003] [Franklin et al., 2004] [Holland et al., 2002] Grasping Graphs by Ear

L. Vesterinen p 05_ The capture radius of an auditory *beacon as the range at which the system considers a user to have research the *waypoint where beacon in positioned. In practise, as a participant moves close enough to waypoint, sound signal is given as an indication for leading the player towards the goal, capturing the graph. Directional-predictive sounds (DPS) are used in relation to capture radius, for graph inspection. 3 unique DPS sounds are combinations of pure sine wave signals with variable tone, pitch and volume. Grasping Graphs by Ear * Waypoint: The coordinates of a specific location. * Beacon: The object at the specific location in the coordinate system.

Pilot Game Testing – 4 subjects: intermediate/experienced, with a normal vision, normal hearing Software: ‘Hidden Graphs’ game, PC version Hardware: AceCad AceCat Flair USB graphics tablet on a standard laptop with two speakers Conditions: silent, closed room Testing procedure: during the test, subjects were blindfolded, hearing the sounds from two speakers. 3 separate test sessions, due to high concentration level and duration of time during the test (>60min< ). 30 games in each session. Each game (1/30) involved playing at the preliminary inspection phase and confirmation phase. In the preliminary inspection phase the player captured the graph trying to memorise the graph. Later player entered the confirmation mode and captured the same graph again as accurately as possible. L. Vesterinen p 06_ Grasping Graphs by Ear

Behavioral strategies: BBP1: ‘Spiral’ and straight line gestures were applied as the basic behavioral patterns for the graph capturing. Player is to scale, change direction or speed of the gesture during the inspection, in relation to the DPS- signals. CS – crossing sound: Player is capturing the graph inside the capture radius. Apply straight line gesture. BS- backward sound: Player is getting out of the capture radius. Apply spiral gesture (scale to big). TS – towards sound: Player is returning towards the capture radius. Apply spiral gesture (scale to small). L. Vesterinen p 07_ Grasping Graphs by Ear

BBP2: ‘S’ –shape and straight line gestures were applied as the basic behavioral patterns for the graph capturing. Player is to scale, change direction or speed of the gesture during the inspection, in relation to the DPS- signals. CS – crossing sound: Player is capturing the graph inside the capture radius. Apply straight line gesture. BS- backward sound: Player is getting out of the capture radius. Apply ‘S’- shape gesture (scale to big). TS – towards sound: Player is returning towards the capture radius. Apply ‘S’- shape gesture (scale to small). BBP3: Combination of the BBP1 and BBP2 behavioral patterns following the same rule format. L. Vesterinen p 08_ Grasping Graphs by Ear Movement vectors

L. Vesterinen p 09_ Directional-predictive signals Grasping Graphs by Ear TS BS CS

L. Vesterinen p 10_ Graphs used in testing Grasping Graphs by Ear 5 different shape graphs were used in the game testing.

L. Vesterinen p 11_ level: Rc = 20 pxls – 1 st level; Rc = 15 pxls – 2 nd level; Rc = 10 pxls – 3 rd level Grasping Graphs by Ear The average distance to graph inspected and the time spent in the training phase in dependence on the game level The relative frequency of DPS- sounds used (the average number is in a log. scale) for graph inspection in the training phase in dependence on the game level

L. Vesterinen p 12_ level: Rc = 20 pxls – 1 st level; Rc = 15 pxls – 2 nd level; Rc = 10 pxls – 3 rd level Grasping Graphs by Ear The relative frequency (the average number) of DPS-sounds used for graph inspection at the confirmation phase in dependence on the game level. Percentage of inspected points. The average distance to graph inspected and the time spent at the confirmation phase in dependence on the game level. Std.dev. of the ave.distance in each level.

L. Vesterinen p 13_ Grasping Graphs by Ear Comparison ratios for the BBP1, 2 & 3 correlation (prel.insp.) > < Typical correlation coefficients

Level 1, capture radius 20 pixels – starting level. The frequency of the levels within the 30 games set in testing: over 90% of the games played in level 2 or 3, capture radius 10 or 15 pixels. 4% of the games played in level 4. Starting position of the capture was free and did not make difference on performance. Separating mouse from the tablet would cause confusion for the player about the position. Smaller the capture radius, longer the confirmation phase and more pixels captured in the target graph. The average distance to the graph within the different levels, did not vary much. Although, std. deviation shows that the ave. distances in different levels are variable. Std. dev. on ave. distance within levels in BBP2, is the largest. Different sounds (CS-BS, BS-TS, CS-TS) in each behavioral pattern associated strongly and positively, resulting in high correlation. L. Vesterinen p 14_ Grasping Graphs by Ear

L. Vesterinen p 15_ Grasping Graphs by Ear Overall, there was a little correlation between the 3 different behavioral strategies, for CS/BS sounds. The most, positive correlation was found in capture radius 15 and 20, between BBP1 and BBP3. Statistics indicated that the graph 2 got the least points inspected in each level, in all 3 behavioral strategies. We can conclude from the calculated statistics that the BBP3 appears to be the most efficient behavioral strategy to be used for the smaller capture radius (Rc = 10). Overall, seems that the BBP1, 2 and 3 are very close to each other in performance measures, therefore no absolute and clear best BB strategy could be stated. For future development, more testing could be done using limited test time, with additional behavioral strategies, increased number of test players and larger data set to make further conclusions.

References: Brewster S. A, Using Non-Speech Sounds to Provide Navigation Cues to, ACM transactions on computer human interaction, 5, 3, 1998 Brown L.M and Brewster S.A, Drawing by ear: interpreting sonified line graphs, Proceedings of the 2003 international Conference on auditory displays, 6-9 July, Brown L. M. et al., Design guidelines for audio presentation of Graphs and Tables, Proceedings of the 2003 International Conference on Auditory Displays, 6-9 July, Evreinov G, ‘Hidden Graphs’ game, Tampere university Franklin K. F and Roberts J.C, Pie Chart Sonification, Proceedings Information Visualization (IV03), pages 4-9. IEEE Computer Society, July 2003 Franklin K.M and Roberts J.C, A Path Based Model for Sonification, Information Visualization, pages IEEE Computer Society, July Holland S, Morse D.R and Gedenryd H, AudioGPS: Spatial Audio Navigation with a Minimal Attention Interface, ACM Personal and Ubiguitous computing, 6, 4, pages 253 – 259, Jacobson R, Representing Spatial Information Through Multimodal Interfaces, IEEE 6 th International conference on Information Visualization (IV’02), July , 2002 L. Vesterinen p 16_

Kramer G, ed. Auditory Display: Sonification, Audification, and Auditory Interfaces, Proc. Volume XVIII, Reading MA: Addison Wesley, Walker B.N and Lindsay J, “Auditory Navigation Performance is Affected by Waypoint Capture Radius,” in Proc. of ICAD 04, Sydney, Australia, July 6-9, 2004 Walker B. N. and Lindsay J, Effect of Beacon Sounds on Navigation Performance in a Virtual Reality Environment, Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, 6-9 July Tran T. V.; Letowski T.; Abouchacra K. S. Tran, Evaluation of acoustic beacon characteristics for navigation tasks. Ergonomics, 1 June 2000, 43, 6, pp (21). Web site (2005) Web site (2004) Web site (2003) Click here to go to first slide L. Vesterinen p 17_