Spatiotemporal Saliency Map of a Video Sequence in FPGA hardware David Boland Acknowledgements: Professor Peter Cheung Mr Yang Liu.

Slides:



Advertisements
Similar presentations
Motion.
Advertisements

The fundamental matrix F
Change Detection C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real time tracking,” IEEE Trans. On PAMI, 22(8): , Aug 2000.
A Survey on Tracking Methods for a Wireless Sensor Network Taylor Flagg, Beau Hollis & Francisco J. Garcia-Ascanio.
A KLT-Based Approach for Occlusion Handling in Human Tracking Chenyuan Zhang, Jiu Xu, Axel Beaugendre and Satoshi Goto 2012 Picture Coding Symposium.
Computer Vision Optical Flow
Motion Tracking. Image Processing and Computer Vision: 82 Introduction Finding how objects have moved in an image sequence Movement in space Movement.
Characterization Presentation Neural Network Implementation On FPGA Supervisor: Chen Koren Maria Nemets Maxim Zavodchik
Motion Detection And Analysis Michael Knowles Tuesday 13 th January 2004.
Processor Technology and Architecture
Chapter 16 Control Unit Operation No HW problems on this chapter. It is important to understand this material on the architecture of computer control units,
Announcements Project1 artifact reminder counts towards your grade Demos this Thursday, 12-2:30 sign up! Extra office hours this week David (T 12-1, W/F.
Computing motion between images
Object Detection and Tracking Mike Knowles 11 th January 2005
Implementation of a Software- based GPS Receiver Anthony J. Corbin Dr. In Soo Ahn Thursday, June 25, 2015.
Chapter 16 Control Unit Implemntation. A Basic Computer Model.
Chapter 4 Processor Technology and Architecture. Chapter goals Describe CPU instruction and execution cycles Explain how primitive CPU instructions are.
Pipelined Processor II CPSC 321 Andreas Klappenecker.
Chapter 15 IA 64 Architecture Review Predication Predication Registers Speculation Control Data Software Pipelining Prolog, Kernel, & Epilog phases Automatic.
Informationsteknologi Friday, October 19, 2007Computer Architecture I - Class 61 Today’s class Floating point numbers Computer systems organization.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Numerical Recipes (Newton-Raphson), 9.4 (first.
EE392J Final Project, March 20, Multiple Camera Object Tracking Helmy Eltoukhy and Khaled Salama.
Sub-Nyquist Sampling DSP & SCD Modules Presented by: Omer Kiselov, Daniel Primor Supervised by: Ina Rivkin, Moshe Mishali Winter 2010High Speed Digital.
Intel Architecture. Changes in architecture Software architecture: –Front end (Feature changes such as adding more graphics, changing the background colors,
5-Stage Pipelining Fetch Instruction (FI) Fetch Operand (FO) Decode Instruction (DI) Write Operand (WO) Execution Instruction (EI) S3S3 S4S4 S1S1 S2S2.
Impulse Embedded Processing Video Lab Generate FPGA hardware Generate hardware interfaces HDL files HDL files FPGA bitmap FPGA bitmap C language software.
Distortion Correction ECE 6276 Project Review Team 5: Basit Memon Foti Kacani Jason Haedt Jin Joo Lee Peter Karasev.
Lecture 16: Reconfigurable Computing Applications November 3, 2004 ECE 697F Reconfigurable Computing Lecture 16 Reconfigurable Computing Applications.
1 What is OO Design? OO Design is a process of invention, where developers create the abstractions necessary to meet the system’s requirements OO Design.
An FPGA Implementation of the Ewald Direct Space and Lennard-Jones Compute Engines By: David Chui Supervisor: Professor P. Chow.
Motion Analysis using Optical flow CIS750 Presentation Student: Wan Wang Prof: Longin Jan Latecki Spring 2003 CIS Dept of Temple.
Pyramidal Implementation of Lucas Kanade Feature Tracker Jia Huang Xiaoyan Liu Han Xin Yizhen Tan.
1 Implementation in Hardware of Video Processing Algorithm Performed by: Yony Dekell & Tsion Bublil Supervisor : Mike Sumszyk SPRING 2008 High Speed Digital.
Jason Li Jeremy Fowers 1. Speedups and Energy Reductions From Mapping DSP Applications on an Embedded Reconfigurable System Michalis D. Galanis, Gregory.
Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image.
A Programmable Single Chip Digital Signal Processing Engine MAPLD 2005 Paul Chiang, MathStar Inc. Pius Ng, Apache Design Solutions.
Copyright © 2004, Dillon Engineering Inc. All Rights Reserved. An Efficient Architecture for Ultra Long FFTs in FPGAs and ASICs  Architecture optimized.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
November 29, 2011 Final Presentation. Team Members Troy Huguet Computer Engineer Post-Route Testing Parker Jacobs Computer Engineer Post-Route Testing.
Course14 Dynamic Vision. Biological vision can cope with changing world Moving and changing objects Change illumination Change View-point.
CS 351/ IT 351 Modeling and Simulation Technologies HPC Architectures Dr. Jim Holten.
Visual Odometry David Nister, CVPR 2004
A Design Flow for Optimal Circuit Design Using Resource and Timing Estimation Farnaz Gharibian and Kenneth B. Kent {f.gharibian, unb.ca Faculty.
A New Class of High Performance FFTs Dr. J. Greg Nash Centar ( High Performance Embedded Computing (HPEC) Workshop.
1 Implementation of Polymorphic Matrix Inversion using Viva Arvind Sudarsanam, Dasu Aravind Utah State University.
Recursive Architectures for 2DLNS Multiplication RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR 11 Recursive Architectures for 2DLNS.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Distortion Correction ECE 6276 Project Review Team 5: Basit Memon Foti Kacani Jason Haedt Jin Joo Lee Peter Karasev.
Motion Segmentation at Any Speed Shrinivas J. Pundlik Department of Electrical and Computer Engineering, Clemson University, Clemson, SC.
An FFT for Wireless Protocols Dr. J. Greg Nash Centar ( HAWAI'I INTERNATIONAL CONFERENCE ON SYSTEM SCIENCES Mobile.
Motion tracking TEAM D, Project 11: Laura Gui - Timisoara Calin Garboni - Timisoara Peter Horvath - Szeged Peter Kovacs - Debrecen.
1 ”MCUDA: An efficient implementation of CUDA kernels for multi-core CPUs” John A. Stratton, Sam S. Stone and Wen-mei W. Hwu Presentation for class TDT24,
Hiba Tariq School of Engineering
Motion and Optical Flow
Motion Detection And Analysis
SoC and FPGA Oriented High-quality Stereo Vision System
A Framework for Automatic Resource and Accuracy Management in A Cloud Environment Smita Vijayakumar.
Video-based human motion recognition using 3D mocap data
Fast and Robust Object Tracking with Adaptive Detection
Pipelining and Vector Processing
Centar ( Global Signal Processing Expo
Arithmetic Logical Unit
Motion Estimation Today’s Readings
Presented by: Yang Yu Spatiotemporal GMM for Background Subtraction with Superpixel Hierarchy Mingliang Chen, Xing Wei, Qingxiong.
Wavelet “Block-Processing” for Reduced Memory Transfers
Announcements more panorama slots available now
Announcements Questions on the project? New turn-in info online
Announcements more panorama slots available now
Memory System Performance Chapter 3
Real time signal processing
Presentation transcript:

Spatiotemporal Saliency Map of a Video Sequence in FPGA hardware David Boland Acknowledgements: Professor Peter Cheung Mr Yang Liu

What is Spatiotemporal Saliency? Saliency – parts of a scene that appear pronounced Spatiotemporal Saliency – parts of a scene that appear pronounced in video

Why Important? General environments are complex and dynamic Human eye handles this by focusing upon salient objects Real-time algorithm to emulate this has many uses:  Image processing  Surveillance  Machine vision  Navigation…

The Problem Spatiotemporal Saliency algorithms have high computational complexity.  Store stack of video frames  Unsuitable for real-time Need algorithm with reduced memory requirements

Overview Introduce Algorithm and section completed Brief background Implementation  Software Model  Hardware Model Results Optimisations (if time) Summary

Algorithm For Spatiotemporal Saliency

Feature Tracking Module Object tracking generally achieved through monitoring optical flow Optical flow: “the distribution of apparent velocities of movement of brightness patterns in an image” Several Algorithms – None perfect Good Trade complexity vs. accuracy – Lukas Kanade Algorithm

Lukas Kanade Algorithm Definition of problem:  Let I and J are two consecutive images  Let u = [ux, uy ] be an image point in I  Find v = u + d = [ux+dx, uy+dy] where v is a similar point on J Points not tracked equally due to aperture problem. Solution is to minimise error:

Lukas Kanade Solution where Find (Iteratively Refine)

Pyramidal Lukas Kanade Algorithm Lukas Kanade Algorithm assumes small motion Handle Larger motion with window size  But Lose Accuracy Solution  Create Hierarchy of images Each image ½ as large  Perform Lukas Kanade on each level to get guess  Map guess to lower levels

Pyramidal Lukas Kanade Algorithm Track feature between two images at the highest level to obtain guess for new feature location Map guess to lower levels, obtain better guess Find final pixel location Apply LK Apply LK, start at guess

Implementation – Software Model Why?  Results to test the hardware against  Useful during debugging stage Choice of Software Language: Matlab  Matrix calculations  Maps well to hardware  Simple for fast development Method:  Apply feature detection algorithm to find co-ordinates  Apply Pyramidal Lukas Kanade to track co-ordinates

Software Model - Demo

Implementation – Hardware Aims:  Fit onto the FPGA  Clock Frequency 65MHz for VGA Not Straightforward:  Initial design emulate software correctly: Well over 200% size of FPGA Initial Design 4MHz

Hardware Considerations Choice Software Language: Handel-C Minimise expensive operations  Memory Accesses  Multiplication  Division Maintain Precision  Floating point precision unavailable General Optimisations  Minimise Delay Path or Logic Depth  Minimise Fan-out

Memory Considerations – Building Hierarchy To build image of higher level:  Iterate over even pixels  Collect mask of values surrounding the pixel  Weight as shown on right  Sum Repeat recursively on output for higher levels

Memory Considerations – Building Hierarchy Pixels re-used:  Store locally  Reduce Memory reads

Memory Considerations – Building Hierarchy

Memory Considerations – Optical Flow Only read once values once from main memory Also reduce fan-out

Multiplications Avoid via left-shifting Pre-compute results whenever possible Use Dedicated Multipliers  Combined for large multiplications

Division Considerations Division Costly process Handel-C designs hardware to implement in one cycle. Large number of bits implies large delay Solution: Spread over multiple cycles  Long Division Slow – unbounded stage  Binary Search If limit range of optical flow per iteration [-1 1]

Division Considerations 0.5 B 0.75 B 0.25 B 0.125B B 0.625B 0.825B 0.25 B 0.5 B 0.75 B 1 B 0 B ≥ < < < < < < < ≥ ≥ ≥ ≥ ≥ ≥ A/B=x≡ A=B*x

Division Considerations 0.5 B 0.75 B 0.25 B 0.125B 0.375B 0.625B 0.825B 0.25 B 0.5 B 0.75 B 1 B 0 B / / /

Hardware Testing Test against software model  Store Feature co-ordinates & tracked locations from software model  Load feature co-ordinates in hardware  Track in hardware  Compare difference Vary number of fractional bits  Examine importance/cost of different fractional precision

Accuracy Results (I)

Accuracy Results (II)

Area Results

Speed Results

Results Summary Final design only uses 1/6 FPGA Use 4/5/6 fractional bits for good accuracy Speed short of desired (approx 50 MHz)  ISE estimates cautious  Pipelining can increase this Reduced Loop control

Optimisations Final Design only uses 1/6 FPGA. Use space to increase Speed:  Pipelined Hardware  Parallel Hardware

Pipelined Architecture I

Pipelined Architecture II

Parallel Architecture

Summary Spatiotemporal Saliency framework Role of optical flow within framework Steps to create & test hardware implementation Effective method to find optical flow  High Speed/Accuracy, small area Optimisations to achieve this Further Improvements possible  Some performance advantages over other hardware optical flow implementations Optical flow useful beyond Spatiotemporal Saliency Framework