Training Regimes Motivation  Allow state-of-the-art subcomponents  With “Black-box” functionality  This idea also occurs in other application areas Audio Signal ProcessingText Understanding STANFORD STANFORD Cascaded Classification Models: Combining Models for Holistic Scene Understanding Geremy Heitz, Stephen Gould, Ashutosh Saxena, and Daphne Koller Cascaded Classification Models  Independent models classify using only independent features  Further levels of “context-aware” classifiers include the original features, and features produced from the previous level’s outputs Object Detection Results Multi-class Segmentation 3D Reconstruction ROAD BUILDING CAR BUS PEOPLE BUILDING ROAD STREET SCENE Currently, the community tends to solve these problems in isolation We want a simple, flexible method for solving them jointly, allowing for the sharing of contextual cues What’s happening in this picture? To solve the scene understanding problem… …we need to know: What are the objects? What are the regions? What are the surfaces? What is the layout? What is the category? HOG DETECTOR Dalal & Triggs Base Detector: D(W) = HOG Detector Score Goal: Put a box around all the cars, people, motorcycles, boats, sheep, and cows Context Aware Detector: Goal: Label each pixel as belonging to one of 7 classes. Base Model: CRF with boosted singleton classifiers + pairwise smoothing Context Aware Model: Relative location map features Grass more likely Independent Models use only their own features Ground Models use features from the other tasks’ groundtruth labels CCM Models use features from the other tasks’ MAP outputs INDEPENDENT CCM CARPEDESTRIANMOTORBIKEBOAT CATEGORIES REGION LABELS ? Scene Categorization  From Scene Category  MAP category, marginals  From Region Labels  How much of each label is in a window adjacent to W  From Depths  Mean, variance of depths, estimate of “true” object size  Final Classifier P(Y) = Logistic(Φ(W)) Scene Type: Urban scene % of “building” above WVariance of depths in W Goal: Label each pixel as belonging to one of 7 classes. Base Model: CRF with boosted singleton classifiers + pairwise smoothing BLACK BOX GRASS SKY Find d* Reoptimize depths with new constraints: d CCM = argmin α||d - d*|| + β||d - d CONTEXT || Context Aware Model: Relative location map features Goal: Label each pixel as belonging to one of 7 classes. Base Model: Logistic over simple features RURAL Context Aware Model: Additional Features  From Detections  Number of each object type present  From Segmentation  Fraction of image of each region type DetectionCarPersonBikeBoatSheepCowDepth INDEP m 2-CCM m RegionsTreeRoadGrassWaterSkyBuildingFG INDEP CCM Boats & Water INDEPPred. RoadPred. Water True Road True Water CCMPred. RoadPred. Water True Road True Water DS1: 422 Images, 5 fold testing Categorization, Detection, Region Labeling All images have full groundtruth for all tasks DS1: 1745 Images Detection, Region Labeling, Depth Reconstruction Disjoint groundtruth = Water = Sky = Building = Tree = Road = Grass = Foreground = Car = Person = Motorcycle = Boat = Sheep = Cow INDEPENDENT CCM Independent Objects Independent Regions CCM Objects Independent Objects Independent Regions CCM Objects Independent Objects Independent Regions CCM Regions Independent Objects Independent Regions CCM Regions