1. Audio Location Accurate Low-Cost Location Sensing James Scott Intel Research Cambridge Boris Dragovic Intern in 2004 at Intel Research Cambridge Studying for PhD at University of Cambridge

2. Overview of talk Audio for fine-grained location Prototype: detecting human sounds Evaluation Application area: 3D user interfaces

3. Background Fine-grained location systems have been built using ultrasound  e.g. Bats (AT&T), Cricket (MIT)  Achieving ~3cm 3D accuracy 95% of the time Many end-user devices have audible-range I/O built in, but few have ultrasound Can we use integrated/off-the-shelf audio hardware for location?

4. Audio-based location with off-the-shelf hardware Many tx  Speakers in environment as tx  Mobile phones or PDAs as rx  Privacy-preserving Many rx  Computer microphones as rx  Can use mobile devices for tx  BUT can also use human sounds for tx

5. Locating human sounds Need coverage from at least 4 mics  Unknowns: X,Y,Z,t (t=time of sound)  More is better since occlusion happens Users do not need special “tag”  No per-user setup required  Lowers costs and increases simplicity User identity is not provided  Many apps do not need identity  Anonymity is good for privacy  Could fuse with identity e.g. from RFID

6. Aims of prototype Fine-grained location sensing with hardware accessible to end users What accuracy can we obtain for locating human sounds, e.g. finger clicking, hand clapping? Application area: 3D user interfaces using human sounds

7. Prototype Use standard PC Add 6 PCI sound cards and 6 mics  Total cost of sound hardware ~£100  From dabs.com Fedora Core 2 Linux distribution Java software

8. Signal Detection System Architecture PositioningTiming Signal Detection Signal Detection

9. Signal Detection Problem: identify the same part of the same sound in audio streams from multiple mics Amplitude-threshold algorithm  Keep track of current noise floor  Mark sample as “significant” when amplitude is is at least F times noise floor. (F ≈ 2.5) Properties  Very good at detecting sharp sounds  Equally important: ignores other sounds  Robust to noisy environments and cheap mics

10. Timing Need time sync for sound streams  1ms error ≈ 30cm in space Problem: Linux/Java introduce delays  Buffering and scheduling result in variable delays of >1ms Solution: hacked sound driver  Timestamp taken at interrupt made available to Java app (via /proc)  Does not account for interrupt delay  Around 200 lines of C code

11. Positioning Survey of microphone positions is currently done manually  See orthogonal work on self-surveying Use well-studied Levenberg- Marquardt technique to find 3D position (and sound generation time)

12. 1D evaluation First evaluated 1D performance for relative distance Use two mics and a 6x7 grid of test points 20 hand claps and 20 finger clicks at each point Microphones Y X 60cm

13. 1D results: hand clapping

14. 1D results: finger clicking

15. Implications of 1D results Our mics are usable ~60º either side of axis Our mics have a maximum range of 4m  Drops to ~2m in very noisy conditions Implications for deployment  Density of microphones required to sense location in a space Finger clicking has median 1D error <5cm  At least some of this due to human error

16. 3D experiment setup 20 finger clicks at 4x4x3 test points on 60cm grid  Total clicks: ~1000. Very sore fingers. Microphones at 2 heights, and much more spread in X,Y than in Z  This might be typical for real deployments Y X 60cm Key: Microphone at 60cm high Microphone at 120cm high

17. Lollipops!

18. 3D distance error

19. What do I think it’s good for? 3D user interfaces  When I click here in future, do this  Extend computer input beyond desk/lap Situated interfaces  Add a light switch by the bed  Remote control without a losable device Inspiration: SPIRIT (AT&T) which allows Bats to be used as 3D pointers

20. What do you think it’s good for? Accessible user interfaces  Elderly, disabled Activity inferencing  Fusion of location with sound recognition Performance art  Spotlights follow sounds Tracking planes in an air show!  Well, maybe not…

21. Visualisation To help deploy and demo it  UbiComp, Mobisys Allows  placement of mics  creation of “buttons” in 3D By mouse or finger Used to create an mp3 player demo

22. Demo video – Accuracy

23. Demo video – User Interface

24. Conclusions 3D location sensing for under £100 of consumer sound peripherals Accuracy: better than 28cm (3D) for 90% of finger clicks  Improves to 10cm for 2D and repeated clicks Sound-based user interfaces Happy to provide source and specs  james.w.scott@intel.com