1. Enhanced Rendering of Fluid Field Data Using Sonification and Visualization Maryia Kazakevich May 10, 2007

2. 2 Multi-Modal Project Overview Input: –Fluid field with velocity vector, pressure, plus potentially density, temperature and other data –Changes with time Output: –Visualization of the given fluid field –Sound characterizing given fluid field Ambient: global to the whole field Local: at the point or area of interaction Local region: particles of the specific subset area around the pointer contribute to the sound

3. 3 Structure –Each rendering program is independent of any other Solution Data Server Max/MSP Program Main Program as Max/MSP object Visualization Program Haptic Program Haptic Device Image Sound

4. 4 Sonification Types Contrast vs. Inverted Amplitude Modulation –velocity value is mapped to either increase or decrease in amplitude of the sound Amplitude vs. Frequency Modulation –highest velocity value is mapped to either loudest noise or highest pitch noise Before vs. after interpolation –many separate sounds for each vertex in the local area vs. 1 sound of the interpolated value at the position of virtual pointer

5. 5 Usability Study Setup Focus on usability study –Compare Visual, Audio and Multi-Modal interface –Influence of audio setup –Simple rendering complex not always better, allows real-time calculations learning process Overall and single trials have to be fairly short –fatigue –sound sensitivity Many short trials –Large nodes in small field

6. 6 Usability Study Setup Visual and/or Audio cues, haptic - navigation

7. 7 Usability Study Setup Trials for each person: 3 groups of people: –frequency modulation –positive amplitude modulation (contrasted sound) –negative amplitude modulation (inverted sound)

8. 8 Usability Study Results worst results for the audio-only system: –participants are slower in locating the goal –participants are less efficient in exploring the volume –Participants are less precise in locating the goal

9. 9 Usability Study Results Specific system setup might help to improve performance:

10. 10 Usability Study Results Multi-modal vs. Visual-only interface: Equal or better results for the multi-modal system –participants explore less space –participants are much faster in locating the goal position

11. 11 Usability Study Results Specific system setup helps to improve performance

12. 12 Usability Study Results In a specific audio setup of multi-modal system, participants are more precise in locating the goal:

13. 13 Usability Study Conclusion Multi-modal system is more powerful than either pure visual or pure audio systems Specific mapping parameters influence system performance Different audio parameters are better for audio- only than for multi-modal interface Different audio parameters might be better for different conditions

14. 14 Further work More sophisticated rendering algorithms –for visualization –for sonification –for haptic representation of the data Sonifying along pathlines, streaklines, streamlines, streamribbons and streamtubes Sonifying other data parameters Fully surrounding sound Self-adjusting system

15. 15 Exploration example

16. The End

17. 17 Types of sonification Iconic Sonification –mapping data to sounds that are associated with certain phenomena Direct Conversion Sonification –mapping data to sound to listen for patterns that are represented in the data. Direct conversion sonification. Can be as simple as taking the frequencies of the waves and making sound waves with the same frequencies, which is most useful as long as the frequencies are at pitches that our ears can hear. Musical Sonification –mapping data to sound in a musical way

18. 18 Sonification Classification Audification: sound samples are directly obtained from data values. Ordered list of numbers is directly taken as PCM (Pulse-code modulation) data for sound Earcons: are combinations of acoustic patterns, e.g. musical motives whose rhythm, timbre, intervals are determined by data of the message. Can be used for navigation tasks or orientation in hierarchies, e.g. classification trees or directory trees, and to communicate more complex messages. Auditory Icons are acoustic marker sounds to signal an event. Meaning of the sound is connected to the information by metaphorical association, e.g. when dragging a file on the computer desktop to the trash can, a crushing sound represents the deletion action Parameterized auditory icon: communicate information about the event in the sound, e.g. sound level or complexity depends on file size being deleted Parameter Mapping: acoustic attributes are mapped to data attributes. High- dimensional data displays can be obtained. No generic way to connect data features with sound event attributes, mapping can be complex in high- dimensional case Model-based: simulating physical properties of the model

19. 19 Sound Synthesis Additive Synthesis: Subtractive Synthesis: filtering out undesired parts from a spectrally broadband input signal FM-Synthesis: Granular Synthesis: composing a larger acoustic event from the superposition of thousands of very short sonic “grains” Sound Sampling: recorded and digitized sound pieces are stored in a sound vector and reproduced by using table-lookup Physical Models for Sound Synthesis: simulating physical processes, e.g. simulate the essential properties of physical instruments in order to get similar means of sound control in models as exist in their real counterparts

20. 20 Applications of Auditory Display Navigation Displays for the Blind. provide location-based information, replace visual input Alarm Sounds to direct attention. Human Computer Interaction. use sound to portray information about actions, e.g. deletion sound when dragging a symbol to the trash can Virtual Reality. increase immersion effect, and communicate information about the interacting objects Process Monitoring. Listeners can habituate to a sound pattern but remain attentive to even subtle changes. (stock market, medical monitors, …) Online-Feedback. Auditory feedback is rendered to give a direct feedback on the action, either to analyze or track processes, to enable an interactive refinement of own actions (e.g. using sonification in rehabilitation) Exploratory Data Analysis is the application of listening to learn about the system. Wherever data is available, sonification may provide a new view on the data, and may be the factor for detecting the unexpected, for discovering new regularities or features in the data.

21. 21 CFD Sonification Advantages –Multi-dimensional data –Complimentary information, concurrent display –Occlusion –Precise location Applications –Model geometry/ grid sonification –Solution process monitoring –Design of aircrafts, cars, etc. –Earth sciences involving fluid mechanics (atmospheric or ocean sciences)

22. 22 Sound waves in a flow Periodic vortex shedding phenomenon sometimes gives rise to audible frequencies –Sudden change of flow direction cause turbulence, separation and wakes (regions of reduced velocity) –Large eddies or vortices are produced at regular frequency and produce pressure disturbance in the flow –Sound is the disturbance of pressure of the medium with an audible frequency range (20Hz ~ 20kHz) –Shedding frequency is determined by Strouhal number St =

23. 23 Fluid Field Calculation Governing differential equations –Time-Averaged Equations for Turbulent flow –General differentiation equations Grid created for the flow domain Discretization –Replacing continuous information with discrete values –Discretizing equations connection grid point values Matrix of simultaneous equations Solving a set of equations –Guess the pressure field –Solve a set of equations –Recalculate the value of p –Repeat

24. 24 Fluid Field Visualization Timeline: line made up of all particles that were marked at the same time Pathline: line that a single particle traces out over time, line you get from a long exposure photograph highlighting a single particle Streakline: locus of all particles that passed through a prescribed fixed point during a specific interval of time. A line traced by the continuous injection at a certain point of dye, smoke, or bubbles Streamline: curve everywhere tangential to instantaneous velocity vectors, short exposure photograph Streamtube: streamline with some thickness or a tubular region surrounded by streamlines Streamribbon: streamline that has been given some width and a rotation based on the local vorticity of the field Hyperstreamline: a tube with an elliptical cross section, where the parameters for the ellipse are provided by tensor values –While glyphs provide an instantaneous representation of a tensor quantity, hyperstreamlines describe the change in the tensor quantity.

25. 25 Probability Distribution assigns a probability to every subset of the real numbersprobabilitysubsetreal numbers The probability distribution of a real-valued random variable X is characterized by its cumulative distribution function:cumulative distribution function Standard normal distribution is a continuous symmetric distribution that follows bell-shaped curve mean and variance uniquely and independently determines the distribution. many measurement variables have distributions that are approximately normal distribution becomes arbitrarily close to a normal distribution, as the number of observations grows large.

26. 26 Skewness & Kurtosis Skewness is a measure of asymmetry of probability distribution –>0: observations are clustered more to the left –<0: clustering to the right Kurtosis is a measure of relative ‘peakedness’ of the curve defined by distribution of the observations –> 3: distribution is more peaked than standard normal distribution –< 3: distribution is flatter than the standard normal distribution

27. 27 Error, deviation mean deviation –is the mean of the absolute deviations of a set of data about the data's mean: standard deviation –is a measure of the spread out the data are. It is the root mean square (RMS) deviation of values from their arithmetic mean:root mean squarearithmetic mean Standard error –is a statistic indicating the accuracy of an estimate

28. 28 P-value A p-value is a measure of how much evidence you have against the null hypothesis Null hypothesis is that all the group means are the same –H0 :      … =  n – Alternative hypothesis is that not all the group means are equal Small p-value: evidence against null hypothesis and for the alternative that group means differ –Observed data unlikely to occur if null hypothesis is correct. Data is inconsistent with null hypothesis Large p-value: no evidence to conclude that the means differ –Data is consistent with null hypothesis

29. 29 F-test F-ratio = MS A /MS error The F-test is used for comparisons of the components of the total deviationF-test Can be used to test the hypothesis that the means of multiple normally distributed populations, all having the same standard deviation, are equal.normally distributedstandard deviation resulting test statistic value would then be compared to the corresponding entry on a table of F-test critical values The p-value describes significance of F-ratio

30. 30 ANOVA (analysis of variance) a way of comparing means, tells if means or average scores of different conditions are different. Structural model: X ij =  j + e ij Between-groups Factors –Different subjects in each condition Repeated-measure –Same subjects in all conditions