Transformation of an Uncertain Video Search Pipeline to a Sketch-Based Visual Analytics Loop Philip A. Legg 1,2, David H.S. Chung 2, Matthew L. Parry 2, Rhodri Bown 3, Mark W. Jones 2, Iwan W. Griffiths 2 and Min Chen 1. University of Oxford 1, Swansea University 2, Welsh Rugby Union 3

Sports Analysis – What is Rugby?

Problem Sports analysts currently study hours of video content to find clips of action for a varierty of needs… Team and Player de-briefings Team committee meetings Match highlights Coaching tactical analysis How can we support this process???

Typical Machine Learning Pipeline Annotated training data provided by the user System learns a model that supports the trained input data Unseen test data is provided by the user System applies model to new test data to obtain a resulting output

Annotated training data provided by the user System learns a model that supports the trained input data Unseen test data is provided by the user System applies model to new test data to obtain a resulting output Typical Machine Learning Pipeline Analyst do not have the time or the technical ability to manually annotate training data! What if the analyst’s requirements change? Trained model becomes useless for their task!

Typical Machine Learning Pipeline

Visual Analytics Loop User inputs their search query into the system. System compares input against current model and returns results. User accepts / rejects results returned by the system. Model is refined based on the user’s acceptance of the results.

Visual Analytics Loop User inputs their search query into the system. System compares input against current model and returns results. User accepts / rejects results returned by the system. Model is refined based on the user’s acceptance of the results.

Interface Overview Sketch Input Rejected Results Model Vis. Accepted Results Search Space Vis. Search Results

User Input for searching Rugby video Sketch-based input. Intuitive drawing tools used to define: Movement paths (red) Regions of interest (blue) Distance range (green)

Obtaining data from Rugby video Top-down view to obtain full pitch coverage. Background filtering to extract players. Team centroid calculated from player positions. Convex hull used to bound team within shape.

Visual Analytics Loop User inputs their search query into the system. System compares input against current model and returns results. User accepts / rejects results returned by the system. Model is refined based on the user’s acceptance of the results.

Interface Overview Sketch Input Rejected Results Model Vis. Accepted Results Search Space Vis. Search Results

Search Space Visualization Timeline view for browsing video. Action glyphs to summarize video content. Plot shows the overall similarity score from the analytical search model.

Interface Overview Sketch Input Rejected Results Model Vis. Accepted Results Search Space Vis. Search Results

Model Visualization Parallel co-ordinates view. Each polyline corresponds to a video segment. Value for each axis show how each video segment scores against that particular metric within the analytical search model.

Model Visualization Pie slice above each axis shows the weighted contribution that metric provides to the overall similarity score. Weights can be adjusted through acceptance / rejection of results, or manual over-ride.

Visual Analytics Loop User inputs their search query into the system. System compares input against current model and returns results. User accepts / rejects results returned by the system. Model is refined based on the user’s acceptance of the results.

Interface Overview Sketch Input Rejected Results Model Vis. Accepted Results Search Space Vis. Search Results

Model Learning Thumbnail selection highlights corresponding polyline in model visualization. Normalized weights are updated based on user selection. Acceptance: w n = w n (1 + ω) Rejection: w n = w n (1 - ω)

Thumbnail Viewer To further guide user judgement, thumbnail can be interactively explored. Thumbnail visualization Annotated top-down video showing distances and team groupings Standard match views

Visual Analytics Loop User inputs their search query into the system. System compares input against current model and returns results. User accepts / rejects results returned by the system. Model is refined based on the user’s acceptance of the results.

Video

Summary We present a novel approach to searching large video data using a visual analytics loop. Model supports learning from user input to refine results – less time- consuming than preparing training data for machine learning. Visual analytics allow the user to assess the performance of the analytical search model. Visual analytics allow the user to search and examine match video content in greater detail.

Thank you for your attention. Philip A. Legg 1,2, David H.S. Chung 2, Matthew L. Parry 2, Rhodri Bown 3, Mark W. Jones 2, Iwan W. Griffiths 2 and Min Chen 1. University of Oxford 1, Swansea University 2, Welsh Rugby Union 3