Visual Analytics for Big Video Visualization Robert S. Laramee Visual and Interactive Computing Group Department of Computer Science Swansea University R.S.Laramee@swansea.ac.uk 1

Overview Data Visualization Motivation: Rugby Sport Video Analysis Visualization Using Glyphs Glyph Design and Sorting System Interface Knowledge Assisted Ranking Conclusions and Acknowledgments 2

Visualization and the Visual Cortex Visualization exploits our powerful visual system 2 million nerve fibers coming from optic nerves Several billion neurons devoted to analyzing visual information (30% cortex) 8% for touch, 3% for hearing (Discover, 1993, Ware, 2013) Enables massively parallel processing of the visual field, i.e., incoming color, motion, texture, shapes etc. 3

Benefits of Data Visualization 4

Visual Analytics ”Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces.” –Illuminating the Path: The Research and Development Agenda for Visual Analytics, Thomas and Cook, 2005 5

Motivation: Massive Multimedia Data from Videos Case Study-Sport Videos: Rugby Sport Video Analysis Watching video is time consuming Trying to find most important events Trying to look for patterns and relationships Trying to compare games 6

Visualization Using Glyphs Data Visualization Using Glyphs: a special class of visualization Discrete: The underlying data is represented pictorially Glyphs by themselves are not novel Sorting: we combine visualization and computer vision techniques, add sorting and interaction capabilities 7

Challenges of Sport Video Data 8

Glyphs For Visualizing Rugby Events 9

Glyphs For Visualizing Rugby Events 10

Visualization Pipeline 11

System Interface 12

Hierarchical Sorting 13

Glyph Sorting Results and Demo 2 matches Gain (Y) vs Event (X) Plotted Events are: Kick Reception, Restart Reception, Lineout, Turnover, Scrum, Penalty Purple highlights indicate points scored 14

Knowledge-Assisted Ranking Rugby Sport Video Analysis Watching video is time consuming (linear search) Trying to find most important events (manual search), looking for patterns, relationships, game comparisons Analysis: Trying to rank events according to their importance Challenging when incorporating several data dimensions 15

Enhanced System Overview Glyph-Based Visualization: shows events sorted by ranking and Y axis (Gain) Model Visualization: Analysis how current model parameters and accuracy correspond to ranking input Ranking Input: ranking model can be exported to primary axis in glyph-based visualization Glyph Control Panel: interaction enables control of axes within glyph-based visualization 16

Visual Analytic Pipeline Tacit Knowledge: A priori knowledge that domain experts have Partial Knowledge: Knowledge domain experts have about most important influential data attributes (or sort keys) Tacit Knowledge does not scale up to large number of events, but choosing a few representative events is easy System Knowledge: No a priori knowledge - derived from tacit knowledge + partial knowledge + sorting functions 17

Event Ranking Using Regression Approach inspired by card sorting (Rugg and McGeorge 1997) User-centered approach to categorize a set of items in to groups, e.g., symbols in cartography (Roth et al., 2011), online course sites (Doubleday, 2013) “Cards” are glyphs Let: e1,e2,…,en be events Let: es1<es2<…<esn an ordering Define: y = Eb where E is an n x m matrix, and bj are the weights or contributions of each sort key Goal: estimate weights of b and ranking function using regression 18

Event Ranking Models User provides sample event ranking (small number e.g., 9) System compares manual ranking with ranking predicted by regression model Each polyline is an event Contribution of each data attribute within model is depicted by gauges on each axis Last axis encodes ranking confidence Each regression model may discover a different set of key performance indicators Performance of each model is evaluated, best model is chosen 19

Refining Model Parameters User may rank event based on an ad hoc requirement, intuition, guesses Thus user may refine model parameters by applying weighting parameters to sort keys Enables user to explore new sorting strategies and understand impact on predicted ranking e.g. remove a sort key 20

Interactive Brushing All views are coordinated and linked. Glyphs corresponding to brushed polylines are highlighted (in focus) Context glyphs can be scaled down Users can select a glyph and replay original video of event 21

Comparing Two Matches Comparison of 2 matches: 81-7, 16-17 Expert user starts by selecting representative glyphs based on Gain Events are input into ranking input and confirm by watching corresponding video Analysts then visually assess resulting model Phases are removed from attribute ranking Large cluster of scoring events is observed in 1st match and absent in 2nd Two poor kicks + turnovers discovered quickly 22

Knowledge Assisted Ranking Results and Demo Analyst: “Using the software has enabled us to discover new key performance indicators that we wouldn’t have recognized before…” Coach: “The system here is a good way at grouping clips…” Player: “The software is useful as it allows you to break up the gaim by what you want to see…” 23

Acknowledgements Thank you for your attention! Any questions? We thank the following: David H..S. Chung, Rhodri Bown, Min Chen, Iwan W. Griffiths, Phillip A. Legg, Adrian Morris, Matthew L. Parry, The Welsh Assembly Government (WAG) For more information, please see: David Chung, Phillip Legg, Matthew Parry, Iwan Griffiths, Rhodri Bown, Robert S Laramee, and Min Chen, Knowledge-Assisted Ranking: A Visual Analytic Application for Sport Event Data, IEEE Computer Graphics & Applications (IEEE CG&A), forthcoming 24

Empirical User Study Investigate difficulty of formalizing a ranking. 5 participants Task: identify and rank set of events by most important positive outcome Task 1: 5 events Task 2: 10 events Color map emphasizes worst and best events Assess confidence Task 3: identify most important set of influential attributes Task 4: specify model 25