1 Department of Electrical and Computer Engineering SDP15: Project Sauron Project Sauron Preliminary Design Review Senior Design Project Fall 2015

2 Department of Electrical and Computer Engineering SDP15: Project Sauron Meet The Team Walter Brown CSE & CS Jose LaSalle EE Zach Goodman EE Omid Meh CSE & EE Advisor: Tilman Wolf

3 Department of Electrical and Computer Engineering SDP15: Project Sauron Context Surveillance of crowded areas Crime Lot of noise

4 Department of Electrical and Computer Engineering SDP15: Project Sauron Goal Develop a real-time recording system to localize and track targets in a noisy environment and output clear audio and video.

5 Department of Electrical and Computer Engineering SDP15: Project Sauron Current Solutions? Video Surveillance Non-Directional Noisy Audio Data intensive Phone Tapping Need to breach carrier network No/Minimal visual Directional Microphone Track one at a time No visual

6 Department of Electrical and Computer Engineering SDP15: Project Sauron Our Solution: Sauron Input video feed Arbitrarily distributed mics Requires calibration Target tracking and AV recording Parallel recording of multiple targets Speech to text

7 Department of Electrical and Computer Engineering SDP15: Project Sauron Who cares Government Homeland security Law Enforcement Forensics Corporate Facility security Residential areas

8 Department of Electrical and Computer Engineering SDP15: Project Sauron System Requirements Operate within human voice frequency spectrum 80Hz - 3kHz (Necessitates 6kHz sampling frequency) Adapt to arbitrary Microphone Distribution Microphone Distribution must operate in the Gunness Student Center Real-time Video (30 frames per second) Position updates (12 updates per second) Video, audio, and transcript must be accurate within 5 seconds. Accurate Visually flag target within a meter Must be able to isolate audio within a cubic foot

9 Department of Electrical and Computer Engineering SDP15: Project Sauron System Overview

10 Department of Electrical and Computer Engineering SDP15: Project Sauron Block Diagram - High Level

11 Department of Electrical and Computer Engineering SDP15: Project Sauron Block Diagram

12 Department of Electrical and Computer Engineering SDP15: Project Sauron Processor Algorithm 1.User selects target from video input 2.Obtain visual description of target 3.Repeat: 1.Find region in video where target could be 2.Map visual coordinates to delay differences 3.Find loudest human voice near delay differences 4.Isolate sound source and highlight the detected target 5.Audio to Text

13 Department of Electrical and Computer Engineering SDP15: Project Sauron Assumptions Target visible to the camera Constant noise source `

14 Department of Electrical and Computer Engineering SDP15: Project Sauron Risks vs. Payoffs Risk: Unethical applications Limited audio and video coverage Hard to conceal Payoff: Easy identification of individuals for user Arbitrary setup

15 Department of Electrical and Computer Engineering SDP15: Project Sauron Cost ItemQuantityUnit CostTotal Cost Microphone8$30$240 Audio Amplifier8$5$40 8-channel Receiver (ADC)1$100 Digital Fish-Eye Camera1$100 Computer1N/A Software (Matlab, Labview)N/A Total$480

16 Department of Electrical and Computer Engineering SDP15: Project Sauron MDR Deliverables Reproduce Acoustic Beamformer Functionality Establish Hardware Setup Microphone to ADC Establish Interface Microphone Input into MATLAB Be able to analyze 8 channels in MATLAB Development of real-time software framework Video input into MATLAB

17 Department of Electrical and Computer Engineering SDP15: Project Sauron Presentation Overview Problem Statement Social Impact Our Project: The Acoustic Beamformer System Requirements Block Diagram Description of Components Alternatives Current Development Status MDR Deliverables Schedule

18 Department of Electrical and Computer Engineering SDP15: Project Sauron Problem Statement Difficult to localize and track clear audio of a targeted individual in a noisy environment in real time.

19 Department of Electrical and Computer Engineering SDP15: Project Sauron Problem Areas Surveillance Who said what? Identify risky individuals Parse through conversations for keywords

20 Department of Electrical and Computer Engineering SDP15: Project Sauron Social Relevance Security Homeland Corporate Surveillance Audio & Visual Forensics?

21 Department of Electrical and Computer Engineering SDP15: Project Sauron Solution: The Acoustic Beamformer Input: 8 microphone outputs Video camera Output: video feed and selected conversations in form of text and audio Utilize Beamforming Signal Processing Techniques

22 Department of Electrical and Computer Engineering SDP15: Project Sauron System Requirements Operate within the human voice frequency spectrum Ideally 300 Hz- 3kHz Localize a sound source using arbitrary mic distribution Correlation b/t video and sound Transcript within reasonable time 15 ft. 40 o (Not to Scale)

23 Department of Electrical and Computer Engineering SDP15: Project Sauron Related SDPs Kelly 2003 Goeckel 2004 Goeckel 2006 Zink 2014

24 Department of Electrical and Computer Engineering SDP15: Project Sauron Final System Functionality Goal: Automatically determine target location using Audio Video Automatically transcribe targeted conversation real-time and save

25 Department of Electrical and Computer Engineering SDP15: Project Sauron Block Diagram Add Hardware – video Software – video Output – transcript, isolated audio track Remove Outputs - Speakers

26 Department of Electrical and Computer Engineering SDP15: Project Sauron Microphones Bla Microphone Omnidirectional Analog output Frequency range: 100 Hz – 15 kHz Sensitivity -42 dB +/ dB SPL S/N Ratio 62 dB

27 Department of Electrical and Computer Engineering SDP15: Project Sauron Video Camera Ideally infinite focal length High resolution Height of human head >= 80 pixels Transmission is fast hard to break or block authenticated Sturdy elevated mount om/shared/static/4d21d5 3f35f7d png

28 Department of Electrical and Computer Engineering SDP15: Project Sauron A-D Converter Need 8 channel inputs Need USB out because modern computers do not support serial interfaces Options we are investigating: MC USB-DIO24/37

29 Department of Electrical and Computer Engineering SDP15: Project Sauron Computer Software MATLAB for simulation and initial analysis Custom software for real-time processing DSP processing library Visualization Audio output

30 Department of Electrical and Computer Engineering SDP15: Project Sauron Alternatives: Nontechnical Direction of interest manually fixated Pass around a microphone Physically turn a microphone or camera to target point Cheap and accessible Inefficient, more time consuming than electronic methods

31 Department of Electrical and Computer Engineering SDP15: Project Sauron Current Development Status Researched similar previous SDP projects 3 main issues keeping others from succeeding: Problems integrating A/D Converters Choosing substandard microphones Assuming plane waves (sources likely too close) Updates on this groups progress Compiled several MATLAB simulations

32 Department of Electrical and Computer Engineering SDP15: Project Sauron Proposed MDR Deliverables Single microphone to A-D Microphone Input into MATLAB Be able to analyze one channel in MATLAB Parallel development of real-time software framework Video input into MATLAB

33 Department of Electrical and Computer Engineering SDP15: Project Sauron Schedule TaskAreaResponsibleDue Date Test mics - determine if they meet specsHardwareNick18-Oct Choose A-DHardwareNick1-Nov Initial MATLAB simulationsMATLABJimmy6-Oct MATLAB Parameter SweepMATLABJimmy6-Oct MATLAB Parameter OptimizationMATLABJimmy10-Oct MATLAB Source ScanMATLABJimmy18-Oct MATLAB Source TrackMATLABJimmy18-Oct Find DSP librarySoftwareRebecca18-Oct Design software structureSoftwareRebecca18-Oct Test A-D into computerHardwareNick8-Nov Process signal in softwareSoftwareRebecca1-Nov Process multiple inputs in softwareSoftwareRebecca15-Nov Get signal->computer & see in MATLABMATLABGeneralMDR MDR Review WeekMisc18-Nov MDR Draft ReportDocumentationJimmy27-Nov MDR Final ReportDocumentationJohn13-Dec Algorithm DesignMiscJohnOngoing