Download presentation
Presentation is loading. Please wait.
Published byDarrell Lambert Modified over 6 years ago
1
Video Google – A google approach to Video Retrieval
2
Effectively precompute matches Textual analogy
Introduction Problem: Retrieve key frames and shots that of a video containing a particular object or scene with the ease and accuracy of Google. Approach: Effectively precompute matches Textual analogy 2
3
Architecture Visual Word User-End Storage Indexing 3
4
Dhruvan Dileep Nishant Pradeep Pramod Sunil
Video Google –Visual words Dhruvan Dileep Nishant Pradeep Pramod Sunil
6
MSER Maximally Stable Extremal Regions
A Maximally Stable Extremal Region (MSER) is a connected component of an appropriately thresholded image
7
SA The Shape Adapted regions are invariant to affine transformations.
The SA regions tend to be centered on corner like features.
8
SIFT Scale Invariant Feature Transform
Invariant to image scaling and rotation Partially invariant to changes in illumination and viewpoint 128 dimensional descriptor
9
Clustering Clustering Techniques Agglomerative O(n2) space. Kmeans
Aim : To vector quantize descriptors into clusters to be used as Visual words Clustering Techniques Agglomerative O(n2) space. Kmeans O(n+k) space, O(n*k*e) time complexity Fast Kmeans Triangulation inequality used. O(n*k) space. Distance calculations reduced to ~ n than n*k*e
10
Statistics 19 Half an hour videos: Classification points – 9 hours Points Clusters Time SA 102823 4000 3hr 45 mins MSER 508191 2000 2hr 50 mins
11
Clustering Evaluation
12
DB and API Indexing/Retrieval Vid_id Frame_id Pos_x Pos_y V_id Visual Words UI
13
Results
14
Future Work Vocabulary Tree for interest point classification Increase the visual vocabulary through efficient clustering.
15
Indexing and Retrieval in Vgoogle
D Pavan Kumar B Rakesh Babu B Naveen Kumar Ankur Jaiswal V Sreekanth P Kowshik J Shashank
16
Overview Visual Words Indexing Results Query
17
Set of visual words in the query rectangle
Input format Pre-processing Video Id Frame Id pos_x pos_y Visual word Id Query Set of visual words in the query rectangle
18
Output format Retrieved Results Rank Video Id Frame Id
19
Objectives Efficient Indexing Fast Retrieval Time Good Recall
20
Approach … Removing the common words Reverse Indexing
Ranking of results
21
Indexing and Retrieval in Document Retrieval
Stop list Used to remove the common words. Inverse File Structure An entry for each word in the corpus followed by a list of all the documents in which it appears. Spatial Consistency Ranking Use the ordering and separation of words to calculate the relevance of a document.
22
Stop list In textual context
Words are extracted from text. Words are filtered based on the level of usefulness. For instance words which are independent of subject or event being described are filtered out. Removing such words will have no effect on the results. E.g.: The way the school is long and hard when walking in the rain. Removing `the` will have no effect on the result.
23
Stop list (contd.) In the current context
Stop list - list of visual words. Occur very often or very less. Determine stop list boundaries empirically. Advantages Reduce number of mismatches Reduce size of inverted file Meaningful visual vocabulary
24
Stop list (contd…)
25
Inverse File Structure
Inverted File structure for Indexing Popular DS in Document Retrieval Mapping from words to Document Less query time compared to Forward indexing Forward Indexing – Sequential Inverted Indexing – Random
26
D1051 D3 D23 D25 D1 D1 D3 D8 ……. D2029 D2 D8 D100 ……. ……. ……. ……. …….
Words D1051 Movie D3 D23 D25 D1 Spain D1 D3 D8 ……. D2029 Table D2 D8 D100 ……. ……. ……. ……. ……. D12 D1078 D102 D25 Song
27
Visual Analogy Words ~ Visual Words Documents ~ Frames
Query vector ~ visual words in Sub-Part of frame
28
D1051 V1 D3 D23 D25 D1 V2 D1 D3 D8 ……. D2029 V3 D2 D8 D100 ……. ……. …….
Visual words D1051 V1 D3 D23 D25 D1 V2 D1 D3 D8 ……. D2029 V3 D2 D8 D100 ……. ……. ……. ……. ……. D12 D1078 D102 D25 Vn
29
Ranking the results - tf-idf
Document – vector of word frequencies Each component of the vector is given some weight Standard Weighting Method TF-IDF
30
Ranking the results - tf-idf
Each document is represented as a vector < t1, t2, t3, … ti,…, tk-1, tk > nid - number of occurrences of ith word in document d. nd total number of words in document d. ni number of occurrences of ith visual word in whole database. N number of documents in the whole database IDF – down weights most frequent words Ranked by cosine of angle between query vector and all document vectors.
31
Ranking the results – Spatial Consistency
There it is. That’s what I ….. been .... have …. I have been there once , while …….. “Google increases the probability of documents having all the search words close to one another"
32
Spatial Consistency Ranking
Spatial arrangement of objects in images. Spatial consistency measure - Re-rank the results Neighboring matches in the query region lie in a surrounding area in the retrieved image.
33
Spatial Consistency Ranking
Search area is defined by 15 nearest neighbors. A neighbor in the surrounding area in the retrieved image counts as a vote. Match with no support / hits is rejected. Repeat this for every match. Total number of votes decides the rank.
34
V V Number of votes = 3
35
V1 Vn V2 V3 ……. 10 4 9 7 2 3 8 14 Frame 1 Frame 2 Frame 3 Frame N 57
4 9 7 2 3 8 14 Frame 1 Frame 2 Frame 3 Frame N 57 23 36 Visual words
36
After Spatial Consistency
Initial Match After Stoplist After Spatial Consistency
37
Future Work More efficient implementation of spatial consistency.
Improve the retrieval time.
38
Chetan Chhaya Nishant Revanth Sandeep Sheetal
USER INTERFACE Chetan Chhaya Nishant Revanth Sandeep Sheetal
39
Objective Build a web interface for retrieving shots from news video database which matches the given image query Display the ranked list of shots eg Date, Channel, Maximum match, Month
40
Input & Output
41
About The Interface… The interface constitutes of the following three parts. Database Schema Data Directories Source Code Files
42
Database Schema All the videos and metadata corresponding to the videos is stored in SQL database which can be queried using MySQL. Following two tables used: Table Table 2 videos videoId channelId Date channels channelId channelName
43
Data Directories Contains following five directories where data is stored Thumbnails Keyframes Stories Shots videos
44
Source Files The interface part consists of 8 files. index.cgi
server.cgi shots.cgi keyframes.cgi SelectRect.js display.cgi play.cgi conf.py Each file is a module.
45
index.cgi Home page of the Interface.
This page lists todays videos as thumbnail of first keyframe corresponding to the first shot of the video. It also gives the user option to select specific videos based upon the criterias of date and channel through comboboxes.
47
Server.cgi User can be directed to this page from any of the pages since all give the user choice to select from the combo boxes. This page lists the results of the user selection from the comboboxes(based upon the criterias of date and channels) The displayed result shows the thumbnail of first keyframe of each video.
49
shots.cgi Page used to display the shots of the video selected from the previous page. The constituting stories of the videos are displayed on the screen one after another. Corresponding to each story ,we display the thumbnail of the keyframe of all the shots in that particular story.
51
keyframes.cgi This page displays the keyframe of the selected shot in its original size( 352 X 288 here ). The user can select a rectangle region from the frame to query the database. This query ( Rectangle Co-ordinates of the selected region (xmin, ymin, xmax, ymax , videoId , shotId) are passed to display.cgi.
53
SelectRect.js This module of the library takes care of user-interaction. Its a selection tool, basically to select a part of the image. It is a JavaScript code which works with just two clicks on the required part of the image, first click denotes the start-coordinates and the second click denotes the end-coordinates. Input to the module: Loaded Image Output: Selected co-ordinates.
54
display.cgi This page is used to display the results of the query.
The matching keyframe and its adjacent keyframes are displayed for all the results. Their corresponding thumbnails are displayed on the screen. The input for this file comes from the indexing module.
56
Play.cgi This page is used to play the video starting from the matching frame. We use an embedded Quicktime player to play the video. The functionalities of the player include: seek, play and, pause. funcionalities controlled by some buttons using Javascript.
58
conf.py This file contains information regarding the Directory paths of the Data Directories and Database details. It is imported in all the cgi files used.
59
Thank You
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.