Personalized Abstraction of Broadcasted American Football Video by Highlight Selection Noboru Babaguchi (Professor at Osaka Univ.) Yoshihiko Kawai and Takehiro Ogura (NHK) Tadahiro Kitahashi (Professor at Kwansei Gakuin Univ.) IEEE Transactions on Multimedia, 2004

Outline Introduction Related Work Method of detecting significant events in video stream Method of generating video abstracts Experimental results Conclusion

Introduction Video abstract  Creating shorter video clips or video posters from an original video stream Two schemes of video abstraction  Temporally compressing the amount of the video data Smith et al., Lienhart et al., He et al., Oh et al., Babaguchi  Provide image keyframe layouts representing the whole video contents Yeung and Yeo, Uchihashi et al., Chang et al., Toklu et al.

Introduction This method of abstracting sports video  Specifically broadcasted TV programs of American football  Take personalization into consideration  Belong to first scheme of video abstraction  Abstraction based on highlights that are closely related to semantical video contents  Detecting significant events like score events

Introduction How to detect events  Image analysis is very difficult  This method’s solution is to make use of external metadata, called gamestats  Linking video segments with descriptions of the gamestats Personalization  Extensively attempted in a variety of application fields  Emphasize it because the significance of scenes vary according to preferences and interests  Provide a profile to collect personal preferences

Related work – time compression Smith et al.  Extracted significant information from video such as keywords, specific objects, camera motions and scene breaks with integrating language, audio, and image analyzes Lienhart et al.  Assemble and edit scenes of significant events in action movies, focusing the on actor/actress’s closeup, text, and sound of gunfire and explosion These two method are based on surface features of the video rather than on its semantical contents.

Related work – time compression Oh et al.  Abstracting video using user selected interesting scenes  Automatically uncover the remaining interesting scenes in the video by choosing some interesting scenes Babaguchi  Video abstraction based on its semantical content in the sports domain  To select highlights of a game, an impact factor for a significant event in two-team sports was proposed He et al.  Create summaries for online audio-video presentations  Use pitch and pause in audio signals, slide transition points in the presentation, and users’ access patterns

Related work – spatial expansion Goal  Visualize the whole contents of the video Yeung and Yeo  Automatically create a set of video posters (keyframe layouts) by the dominance value of each shot Uchihashi et al.  Making video posters whose size can be changed according to the importance measure Chang et al.  Make shot-level summaries of time ordered shot sequence or hierarchical keyframe clusters, as well as program-level summaries

Detection of significant events Detect significant events in the original video stream according to the description in the gamestats

Identification of event frames An event occurs in the shot including the event frame Attempt to recognize text expressing the game time in the overlay, and then to identify an event frame To identify the event frame, an overlay model is employed

Detection of event shots A shot is defined as consecutive image frames at a single camera view Classify the event shots into four types  live-play, replay, pre-play, and post-play shot

Generation of personalized video abstract Generating video abstracts from the detected significant events Select highlights of the game from all the events, considering profile descriptions The generating rules for the video abstract:

Profile A video abstract has to be personalized because significance of events could change individually Provide a profile to collect personal preferences and interests, the items are:  Favorite teams  Favorite players  Events to want to see  Specifications Range of the video stream to be abstracted Length of the abstract

Significance degree of events The highlights of the game depend on the significance of each event, and significance can be estimated in terms of event rank, event occurrence time, and the profile Event rank  State change event (SCE): a score event can change the current state into a different state  Rank 1: SCE’s.  Rank 2: not SCE’s, but exceptional score events  Rank 3: events closely related to score events  Rank 4: all other event that are not Rank 1 to 3  Rank based significance degree of event I r : where r i denotes the rank of the ith event E i, αis a coefficient to consider how large the difference of the rank affects the significance

Significance degree of events Event occurrence time  The score events occurring at the latter or final stage of the game largely affect the result that should have great significance  Occurrence time based significance degree of event I t : where N is the number of all events, β is a coefficient to consider how large the occurrence time affects the significance Profile  Comparing the descriptions of the profile and the occurring event  Profile based significance degree of event I p : where l denotes the number of descriptions that don’t coincide with each other, γ is a coefficient to consider how large the profile affects the significance Significance degree of an event I:

Selection of highlights To determine highlights, we concentrate on both priority order of shots and significance degree of events Priority order of each shot segment  Motion live shot  Still live shot  Motion replay shot  Still replay shot  Motion pre-play shot  Still pre-play shot  Motion post-play shot  Still post-play shot

Selection of highlights

Experimental results

Two measures to evaluate the quality of the generated abstract where N denote the number of highlights included in abstracts

Experimental results – effect of personalization Inclusion ratio: the ratio of the length of shots which are concerned with the specified team to the total length

Experimental results – effect of personalization 4-symbol string in the cells of table represents each condition of the pre- play, live, replay, and post-play shots for the event

Experimental results – effect of personalization

Conclusion Based on the detected significant events by recognizing the textual overlays Link the video contents with useful external metadata by using the gamestats Three sorts of significance degrees play a central role in highlight selection Remaining problems  The method is for different two-team sports  A tailoring mechanism for shots, adjusting for the total abstract  Seek for a sophisticated way of refining the profile