Foundations & Core in Computer Vision: A System Perspective Ce Liu Microsoft Research New England

Vision vs. Learning Computer vision: visual application of machine learning? Data  features  algorithms  data ML: design algorithms given input and output data CV: find the best input and output data given available algorithms

Theoretical vs. Experimental Theoretical analysis of a visual system – Best & worst cases – Average performance Theoretical analysis is challenging as many visual distributions are hard to model (signal processing: 2 nd order processes, machine learning: exponential families) Experimental approach: full spectrum of system performance as a function of the amount of data, annotation, number of categories, noise, and other conditions

Quality vs. Speed HD videos, billions of images to index Real time & 90% vs. one hour per frame & 95%? Mechanism to balance quality and speed in modeling

Automatic vs. semi-automatic Common review feedback: parameters are hand-tuned; not clear how to set the parameters Vision system user feedback: I don’t know how to tweak parameters! Computer-oriented vs. human-oriented representations Human-in-the-loop (collaborative) vision – How to optimally use humans (what, which and how accurate) beyond traditional active learning – Model design by crowd-sourcing – Learning by subtraction

Algorithms vs. Sensors Two approaches to solving a vision problem – Look at images, design algorithms, experiment, improve… – Look at cameras, design new/better sensors, … Cameras for full-spectrum, high res, low noise, depth, motion, occluding boundary, object, … What’s the optimal sensor/device for solving a vision problem? What’s the limit of sensors?

Thank you! Ce Liu Microsoft Research New England