数字视频技术 --- 视觉数据挖掘与内容搜索 数字视频技术

数据挖掘  The process of extracting patterns from data.  The process of analyzing data from different perspectives and summarizing it into useful information.

图像与视频中的数据挖掘

图像数据挖掘系统模型 Images Database Index and Storage Feature Extraction Query Result Query Image Lookup

对象识别与语义解读有本质不同 Here word recognition is solved, we can access the meaning of the words. But yet, we are far from having solved language understanding. Detecting objects is just one small piece of understanding scenes (it might not even be the hardest). Images and sequence tell stories, and the structure of those stories are as complex as sentences, paragraphs and books. “ 老板坐车去了机场，但是因为交通堵塞，他 没能赶上飞机。 ”

Vision as a source of semantic information slide credit: Fei-Fei, Fergus & Torralba

Object categorization sky building flag wall banner bus cars bus face street lamp slide credit: Fei-Fei, Fergus & Torralba

Scene and context categorization outdoor city traffic … slide credit: Fei-Fei, Fergus & Torralba

Qualitative spatial information slanted rigid moving object horizontal vertical slide credit: Fei-Fei, Fergus & Torralba rigid moving object non-rigid moving object

内容搜索 – 场景认识的核心技术 Techniques for understanding a visual scene  Video Shot Detection  Video Object Segmentation  Blob Detection  Object class recognition  Context  Semantic

内容搜索 – 场景认识的系统构架

内容搜索 – 场景的认识 1) Object representation based on intrinsic features: Local features no car Classifier p( car | V L ) 2) Detection strategy: Sky Mountain Buildings cars 3) The scene representation

内容搜索 – 图像特征 Low Level: Color Texture Edge/Shape Object Level: Regions

内容搜索 – 场景的认识

Athlete Horse Grass Tree Saddle Wind Athlete Rock Grass Tree Sky Rope Athlete Snow Tree Sky Snowboard Large # of uninitialized images Small # of automatically initialized images Visual Text C Nr O R NF X Ar Nt Z S T Sky Athlete Tree Mountain Rock Class: Rock climbing Athlete Mountain Tree Rock Sky Ascent Sky Athlete Water Tree sailboat Class: Sailing Athlete Sailboat Tree Water Sky Wind LearningModel Recognition Tree Athlete Snowboard Snow Class: Snowboarding Athlete Snowboard Tree Snow Sky Powder

场景的认识 – 应用示例

内容搜索 – 对象的认识

内容搜索 – 对象的认识方法

注意问题  Image similarity is subjective and context- dependent.  In addition, we are using low-level image features. (semantic gap)  Thus, it is VERY difficult to express the user’s concept by these features.

对象的认识 – 例子

对象的认识 – 应用示例

内容搜索 – 行为的认识 Traffic analysis Object classification

内容搜索 – 行为认识的方法 Behaviour and Event Analysis

行为认识 – 应用示例

挑战与方向 “In general, current systems have not yet had significant impact on society due to an inability to bridge the semantic gap between computers and humans.”

挑战与方向： semantic gap （语义）

挑战与方向： Context （语境） 2 1

挑战与方向