CSP Examples Tamara Berg CS 560 Artificial Intelligence Many slides throughout the course adapted from Svetlana Lazebnik, Dan Klein, Stuart Russell, Andrew Moore, Percy Liang, Luke Zettlemoyer

Announcements/Reminders HW1 due date extended to Sunday, Sept 13, 11:59pm on classroom server – submit one solution per group and Ric with the location of your directory and names of your group

Review from last class

Backtracking search In CSP’s, variable assignments are commutative – For example, [WA = red then NT = green] is the same as [NT = green then WA = red] We only need to consider assignments to a single variable at each level (i.e., we fix the order of assignments) Depth-first search for CSPs with single-variable assignments is called backtracking search

Example

Red?Green?Blue? x xx x

Backtracking search algorithm

Improving Backtracking Efficiency Ordering – How should we order variables for assignment? – How should we order values from the domains?

Backtracking search algorithm

Improving Backtracking Efficiency Ordering – How should we order variables for assignment? – How should we order values from the domains? Filtering – Can we detect inevitable failures early?

Early detection of failure Apply inference to reduce the space of possible assignments and detect failure early

Early detection of failure: Forward checking Keep track of remaining legal values for unassigned variables Terminate search when any variable has no legal values

Simplest form of constraint propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y Arc consistency Consistent!

Simplest form of constraint propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y Arc consistency

Simplest form of propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y – When checking X  Y, throw out any values of X for which there isn’t an allowed value of Y If X loses a value, all pairs Z  X need to be rechecked Arc consistency

Simplest form of propagation makes each pair of variables consistent: – X  Y is consistent iff for every value of X there is some allowed value of Y – When checking X  Y, throw out any values of X for which there isn’t an allowed value of Y Arc consistency detects failure earlier than forward checking Can be run before or after each assignment Arc consistency

Improving Backtracking Efficiency Ordering – How should we order variables for assignment? – How should we order values from the domains? Filtering – Can we detect inevitable failures early? Structure – Can we exploit problem structure?

Tasmania and the mainland are separate structures. We can solve them independently and then combine their solutions. More generally, we can decompose the CSP into separate connected components (sub-problems of the CSP). Then the solution is the union of the sub- problem solutions. This can be much more efficient than solving the entire CSP as one problem.

Algorithm for tree-structured CSPs Choose one variable as root, order variables from root to leaves such that every node's parent precedes it in the ordering

Algorithm for tree-structured CSPs Choose one variable as root, order variables from root to leaves such that every node's parent precedes it in the ordering Backward removal phase: check arc consistency starting from the rightmost node and going backwards

Algorithm for tree-structured CSPs Choose one variable as root, order variables from root to leaves such that every node's parent precedes it in the ordering Backward removal phase: check arc consistency starting from the rightmost node and going backwards Forward assignment phase: select an element from the domain of each variable going left to right. We are guaranteed that there will be a valid assignment because each arc is arc consistent

Alternatives to backtracking

Local Search Methods Local Search: Improve what you have until you can’t make it better Generally much faster and more memory efficient than back-tracking search (but not necessarily complete)

Local search for CSPs Start with an initial assignment of variables to values Allow states with unsatisfied constraints Attempt to improve states by reassigning variable values Hill-climbing search: – In each iteration, randomly select any conflicted variable and choose value that violates the fewest constraints – I.e., attempt to greedily minimize total number of violated constraints h = number of conflicts

Local search for CSPs Start with “complete” states, i.e., all variables assigned Allow states with unsatisfied constraints Attempt to improve states by reassigning variable values Hill-climbing search: – In each iteration, randomly select any conflicted variable and choose value that violates the fewest constraints – I.e., attempt to greedily minimize total number of violated constraints – Problem: local minima h = 1

Origin of CSPs

CSP in computer vision: Line drawing interpretation An example polyhedron: David WaltzDavid Waltz, 1975 Desired output: Goal: Label lines in drawing as convex (+), concave (-), or boundary (>). How could we set this up as a CSP? Variables? Domains? Constraints?

CSP in computer vision: Line drawing interpretation Four vertex types: Constraints imposed by each vertex type: David WaltzDavid Waltz, 1975

CSPs in recent research!

Inferring Temporal Order of Images from 3d Structure Grant Schindler, Frank Dellaert, Sing Bing Kang IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR),

The World Changes Over Time How can we reason about time in structure from motion problems?

Ultimate Goal Produce time-varying 3d models of cities from historical photographs Requires: Building 3d models Reasoning about time

Temporal Inference Problem 1. When was each photograph taken? 2. When did each building first appear? 3. When was each building removed? Set of Photographs: Set of Objects: Buildings

Temporal Inference Problem (a, b)Time Interval tTime Temporal Inference Problem Given observations Z and scene geometry X, what are the temporal parameters T? Images Structure from Motion (x,y,z)Position (x,y,z)Translation (y,p,r)Rotation f Focal Length Structure Cameras Correspondence s 3D

Overview 3D Images + Time Structure + Time Intervals Buildings D Correspondences Images

Overview 3D Images + Time Structure + Time Intervals Buildings D Correspondences Images

Overview 3D Images + Time Structure + Time Intervals Buildings D Correspondences Images

Overview 3D Images + Time Structure + Time Intervals Buildings D Correspondences Image s

3D Reconstruction Images 3D Point Cloud Structure from Motion Bundler Software by Noah Snavely (Snavely et al SIGGRAPH 2006)

Scene Reconstruction Automatically estimate – position, orientation, and focal length of cameras – 3D positions of feature points Feature detection Pairwise feature matching Pairwise feature matching Incremental structure from motion Correspondence estimation

Input Photos

Feature detection Detect features points on objects [Lowe, IJCV 2004]

Pairwise feature matching Match feature points between each pair of images

Correspondence estimation Link up pairwise matches to form matches across several images Image 1Image 2Image 3Image 4

Structure from motion Camera 1 Camera 2 Camera 3 R 1,t 1 R 2,t 2 R 3,t 3 p1p1 p4p4 p3p3 p2p2 p5p5 p6p6 p7p7 minimize f (R, T, P)f (R, T, P)

Incremental structure from motion See: e.g. for more infohttp://

Grouping Points into Buildings 3D Point Cloud 3D Buildings Building Geometry 1. Convex hull of each group 2. Fit ground plane to camera centers 3. Extend convex hulls to ground Point Grouping Group points according to both: a. Distance < threshold in 3D b. Observed simultaneously in 1+ images

3D Reconstruction: Points vs. Objects Lower Manhattan 454 images 83,860 points960 Buildings

Reasoning About Time: From Constraint Satisfaction… Key Idea: Visibility of 3D points constrains image ordering Variables? Domains? Constraints? Disadvantage: Only relative image ordering

Visibility Reasoning Goal: reorder images (columns) to have as few violations as possible observed missing occluded Columns: images Rows: points Violates constraints: Satisfies constraints:

Temporal Ordering via CSP Goal: reorder images (columns) to have as few violations as possible Backtracking search: intractable (n! solutions for n images) Local search: start with random ordering of columns, swap columns or groups of columns to reduce the number of conflicts

Results...

Temporal Ordering via CSP Goal: reorder images (columns) to have as few violations as possible Local search: start with random ordering of columns, swap columns or groups of columns to reduce the number of conflicts Can also reorder the rows to group together points that appear and disappear at the same time – that gives you buildings

Results: Time-varying 3d model

... Results

Conclusions Authors demonstrated that constraint satisfaction problems provide a powerful framework in which to solve temporal ordering problems Presented the first known method for solving this ordering problem. Remaining challenges? – Dealing with noise in classification of points as observed/missing/occluded.