© 1998 Peter B.L. Meijer On cross-modal sensory streams... The vOICe by Peter B.L. Meijer The vOICe by Peter B.L. Meijer Philips Research Laboratories.

Slides:

Advertisements

Similar presentations

Revised estimates of human cochlear tuning from otoacoustic and behavioral measurements Christopher A. Shera, John J. Guinan, Jr., and Andrew J. Oxenham.

Advertisements

The Perception of Speech. Speech is for rapid communication Speech is composed of units of sound called phonemes –examples of phonemes: /ba/ in bat, /pa/

Periodicity and Pitch Importance of fine structure representation in hearing.

The Perception of Speech. Speech is for rapid communication Speech is composed of units of sound called phonemes –examples of phonemes: /ba/ in bat, /pa/

CEN352, Dr. Ghulam Muhammad King Saud University

A.Diederich– International University Bremen – Sensation and Perception – Fall Frequency Analysis in the Cochlea and Auditory Nerve cont'd The Perception.

Development of sound localization

EE2F1 Speech & Audio Technology Sept. 26, 2002 SLIDE 1 THE UNIVERSITY OF BIRMINGHAM ELECTRONIC, ELECTRICAL & COMPUTER ENGINEERING Digital Systems & Vision.

MPEG Audio Compression by V. Loumos. Introduction Motion Picture Experts Group (MPEG) International Standards Organization (ISO) First High Fidelity Audio.

Changing qualia, acquiring new qualia Class 4 Changing qaulia between sight and sound Re-wired ferret Soundscape New qualia – magnetic north.

The Auditory System. Audition (Hearing)  Transduction of physical sound waves into brain activity via the ear. Sound is perceptual and subjective. 

Sound source segregation (determination)

EE2F1 Speech & Audio Technology Sept. 26, 2002 SLIDE 1 THE UNIVERSITY OF BIRMINGHAM ELECTRONIC, ELECTRICAL & COMPUTER ENGINEERING Digital Systems & Vision.

The Perception of Speech

THE HUMAN EAR AND SIMPLE TESTS OF HEARING Ear Anatomy  Outer Ear  Auricle, external auditory canal and the tympanic membrane  Middle Ear  An air filled.

Sensation and Perception: Hearing

Chapter 4 Powerpoint: Hearing

Psychoacoustics: Sound Perception Physics of Music, Spring 2014.

Audio Compression Usha Sree CMSC 691M 10/12/04. Motivation Efficient Storage Streaming Interactive Multimedia Applications.

Media Representations - Audio

Effect of the Phase on Speech and Musical signals using DWT By: Hafiz, Malik ECE418,DSPIISpring,02.

UNIT 4: SENSATION & PERCEPTION Module 12. Sensation & Perception Sensation: the process by which you detect physical energy from your environment and.

School of Computer Science & Information Technology G6DPMM - Lecture 15 Media Design III – Video & Animation.

Sensory Experience Alters Response Strength, Selectivity and Temporal Processing of Auditory Cortex Neurons Mike Kilgard University of Texas at Dallas.

Hearing Subtitle. The Physics of Sound  Frequency: The number of cycles a sound wave completes in a given period of time  Amplitude: the Strength of.

Ch 111 Sensation & Perception Ch. 11: Sound, The Auditory System, and Pitch Perception © Takashi Yamauchi (Dept. of Psychology, Texas A&M University) Main.

Psychology of Music Learning Miksza Music and Brain Research …from Peretz & Zatorre (2005)

Seeing with Sounds Amir Amedi’s Lab

Gammachirp Auditory Filter

Chapter 5, Objective 11 & 12 Mikayla Tucker Kiara Giles AP Psychology, 7 th Period.

Fundamentals of Sensation and Perception SOUND AND THE EARS ERIK CHEVRIER OCTOBER 5 TH, 2015.

Chapter 4 Sensation What Do Sensory Illusions Demonstrate? Streams of information coming from different senses can interact. Experience can change the.

Sensation and Perception: Hearing. Sound Sound comes in waves. The waves are produced by vibration. Sound is vibration Ex. Clap your hands causes the.

+ Assistive Technology By Lyndsay RHodes. + Screen Reader A screen reader is a software application for people with severe visual impairments. A screen.

Introduction to psycho-acoustics: Some basic auditory attributes For audio demonstrations, click on any loudspeaker icons you see....

2 How do we construct our representations of the external world? To represent the world, we must detect physical energy (a stimulus) from the environment.

Sensation Sensation: your window to the world Perception: interpreting what comes in your window.

WebCT You will find a link to WebCT under the “Current Students” heading on It is your responsibility to know how to work WebCT!

© 2007 Peter B.L. Meijer The vOICe in Neuroscience 2007: Cross-modal plasticity & visual sounds Amedi et al., Nature Neuroscience, June 2007: Shape conveyed.

© 2003 Peter B.L. Meijer Philosophy and The vOICe What does that mean? 17th Century “Molyneux's Question” revisited? Q: Would someone born blind, in case.

OTHER RESEARCH IN SIGNAL PROCESSING AND COMMUNICATIONS IN ECE Richard Stern Carnegie Mellon University (with Dave Casasent, Tsuhan Chen, Vijaya Kumar,

1 What is Multimedia? Multimedia can have a many definitions Multimedia means that computer information can be represented through media types: – Text.

Development of Sound Localization 2 How do the neural mechanisms subserving sound localization develop?

CSD 2230 HUMAN COMMUNICATION DISORDERS Topic 3 Introductory Anatomy and Physiology of the Speech, Voice and Auditory System The Auditory System.

Perceptual organization How do we form meaningful perceptions from sensory information?

1 PSYCHOLOGY (8th Edition, in Modules) David Myers PowerPoint Slides Aneeq Ahmad Henderson State University Worth Publishers, © 2007.

Auditory Computation Overview: I. Physiological Foundations II. Elements of Auditory Computation.

AP Psychology Unit 4 Module 20

HIGH-RESOLUTION SINUSOIDAL MODELING OF UNVOICED SPEECH GEORGE P. KAFENTZIS, YANNIS STYLIANOU MULTIMEDIA INFORMATICS LABORATORY DEPARTMENT OF COMPUTER SCIENCE.

PSYCHOACOUSTICS: SOUND PERCEPTION PHYSICS OF MUSIC, SPRING 2016.

SENSATION The basics, vision, and hearing, and the other senses.

Sensation and Perception Unit 7

A map of periodicity orthogonal to frequency representation in the cat auditory cortex. Gerald Langner, Ben Godde, and Hubert R. Dinse Examples of auditory.

A proposed Proof-of-Concept experiment

Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind Ella Striem-Amit, Laurent Cohen, Stanislas Dehaene,

Information-Theoretic Listening

Unit 5: Senation & Perception Day 1: Sensory Thresholds & The Eye

CHAPTER 10 Auditory Sensitivity.

Psychoacoustics: Sound Perception

Grudge Modules 12 – 15.

Lab 6: Sound Analysis Fourier Synthesis Fourier Analysis

Multimodal Human-Computer Interaction New Interaction Techniques 22. 1

III Digital Audio III.6 (Mo Oct 22) The MP3 algorithm with PAC.

Noise Aperiodic complex wave

Special Senses Chapter 8.

Hearing Loss Adapted from NYDBC / Susie Morgan.

CEN352, Dr. Ghulam Muhammad King Saud University

AP Psychology Sept. 28th Objective Opener

Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind Ella Striem-Amit, Laurent Cohen, Stanislas Dehaene,

Presentation transcript:

© 1998 Peter B.L. Meijer On cross-modal sensory streams... The vOICe by Peter B.L. Meijer The vOICe by Peter B.L. Meijer Philips Research Laboratories Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida

© 1998 Peter B.L. Meijer The vOICe: “Seeing with your Ears” Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations

© 1998 Peter B.L. Meijer The vOICe: “Seeing with your Ears” Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations Basic facts: Blind people cannot see... but usually they can hearBlind people cannot see... but usually they can hear Multimedia computing allows audio-visual transformationsMultimedia computing allows audio-visual transformations VideoAudio Mental Video VideoAudio The vOICe Human brain So translate video to audio!?

© 1998 Peter B.L. Meijer The vOICe implementations HardwareprototypeHardwareprototype Java applet Windows95/98/NTWindows95/98/NT

© 1998 Peter B.L. Meijer The vOICe image-to-sound mapping Sum of sinusoids sinusoids Frequency Time Amplitude

© 1998 Peter B.L. Meijer The vOICe image-to-sound mapping Frequenc y Time Spectrographic synthesis

© 1998 Peter B.L. Meijer Could this really work? Check: map sounds back to images Could this really work? Check: map sounds back to images The vOICe image-to-sound mapping Yes! (at least technically) TheoryTheory

© 1998 Peter B.L. Meijer The vOICe image reconstruction “Soundscape”“Soundscape” The vOICe SpectrogramSpectrogram

© 1998 Peter B.L. Meijer The vOICe image reconstruction “Soundscape”

© 1998 Peter B.L. Meijer The vOICe and human speech Pronouncing one image per second Pronouncing one image per second

© 1998 Peter B.L. Meijer The vOICe mental imagery? Representation may dramatically affect recognition! Representation may dramatically affect recognition!

© 1998 Peter B.L. Meijer The vOICe limitations Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea)

© 1998 Peter B.L. Meijer The vOICe limitations Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity

© 1998 Peter B.L. Meijer The vOICe limitations Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity Issues: 3Frequency-Time uncertainty Physiology / Perception:Physiology / Perception: 3just-noticeable difference (JND) 3critical bands (human cochlea) ?auditory streaming/segregation ?neural pathways & bandwidth ?neural processing & plasticity Psychology / Education:Psychology / Education: ?sounds (too) unpleasant ?minimum required results ?acceptable training effort ?motivation for learning ?training programs Psychology / Education:Psychology / Education: ?sounds (too) unpleasant ?minimum required results ?acceptable training effort ?motivation for learning ?training programs

© 1998 Peter B.L. Meijer The vOICe approach 3needs more research 3is cheap technology 3is non-invasive The vOICe approach 3needs more research 3is cheap technology 3is non-invasive Conclusions

© 1998 Peter B.L. Meijer The vOICe approach 3needs more research 3is cheap technology 3is non-invasive 3is available Now! The vOICe approach 3needs more research 3is cheap technology 3is non-invasive 3is available Now! Conclusions