Submodularity Reading Group Matroids, Submodular Functions M. Pawan Kumar

Submodularity Reading Group Matroids, Submodular Functions M. Pawan Kumar http://www.robots.ox.ac.uk/~oval/

Submodularity Reading Group Until 1980s –Matroids, Submodularity, Polymatroids –Submodular Function Minimization is “Easy” 1990s: Convex optimization for SF Min 2000s: Combinatorial algos for SF Min 2010s: SF Max (Greedy, Multilinear) Tutorial Format

Subsequent Reading Groups One every two weeks Alternate with TVG reading group Tuesdays? 4pm-6pm? I will send out emails for “volunteers”

Website

Matroids Maximum Weight Independent Set Submodular Functions Relationship Outline

€1000 €400 €700 Steal at most 2 items Greedy Algorithm €1000

€400 €700 Steal at most 1 item Greedy Algorithm €1000 €1700

€400 Steal at most 0 items Greedy Algorithm €1700 Success

€1000 €400 €700 2 kg 1 kg 1.5 kg Steal at most 2.5 kg Greedy Algorithm (Most Expensive) €1000 2 kg

€400 €700 1 kg 1.5 kg Steal at most 0.5 kg Greedy Algorithm (Most Expensive) €1000 2 kg Failure

€1000 €400 €700 2 kg 1 kg 1.5 kg Steal at most 2.5 kg Greedy Algorithm (Best Ratio) €1000 2 kg

€400 €700 1 kg 1.5 kg Steal at most 0.5 kg Greedy Algorithm (Best Ratio) €1000 2 kg Failure Why?

Matroids –Definition –Examples –Terminology Maximum Weight Independent Set Submodular Functions Relationship Outline

Subset System Set S Non-empty collection of subsets I Property: If X  I and Y ⊆ X, then Y  I (S, I ) is a subset system

Hereditary Property Set S Non-empty collection of subsets I Property: If X  I and Y ⊆ X, then Y  I (S, I ) is a subset system

Example Set S = {1,2,…,m} I = Set of all X ⊆ S such that |X| ≤ k Is (S, I ) a subset system? Yes

Example Set S = {1,2,…,m}, w ≥ 0 I = Set of all X ⊆ S such that Σ s  X w(s) ≤ W Is (S, I ) a subset system YesNot true if w can be negative

Matroid Subset system (S, I ) Property: If X, Y  I and |X| < |Y| then there exists a s  Y\X M = (S, I ) is a matroid such that X ∪ {s}  I

Augmentation/Exchange Property Subset system (S, I ) Property: If X, Y  I and |X| < |Y| then there exists a s  Y\X M = (S, I ) is a matroid such that X ∪ {s}  I

Example Set S = {1,2,…,m} I = Set of all X ⊆ S such that |X| ≤ k Is M = (S, I ) a matroid?Yes Uniform matroid

Example Set S = {1,2,…,m}, w ≥ 0 I = Set of all X ⊆ S such that Σ s  X w(s) ≤ W Is M = (S, I ) a matroid?No Coincidence?No

Matroids (S, I ) is a matroid (S, I ) admits an optimal greedy algorithm

Matroids (S, I ) is a matroid (S, I ) admits an optimal greedy algorithm Next section

Uniform Matroid S = {1,2,…,m} X ⊆ S I = Set of all X ⊆ S such that |X| ≤ k

Graph Theory

Forest v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 G = (V, E) V = {v 1,…,v n } E = {e 1,…,e m } Subset of edges that contain no circuit

Forest v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 G = (V, E) V = {v 1,…,v n } E = {e 1,…,e m } Subset of edges that contain no circuit Forest?

Graphic Matroid G = (V, E), S = E X ⊆ S X ∈ I if X is a forest

Matching v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 G = (V, E) Matching is a set of disjoint edges. No two edges in a matching share an endpoint.

Matching v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 G = (V, E) Matching is a set of disjoint edges. No two edges in a matching share an endpoint. ✓

Matching v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 G = (V, E) Matching is a set of disjoint edges. No two edges in a matching share an endpoint. ✗

Matching Matroid v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 G = (V, E) X ⊆ S ∈ I if a matching covers X S = V (S, I ) is a matroid?Matching Matroid

Set Theory

Partition Set S Non-empty subsets {S i } {1, 2, 3, 4, 5, 6, 7, 8, 9} Mutually exclusive S i ∩ S j = ϕ, for all i ≠ j Collectively exhaustive ∪ i S i = S {{1, 2, 3}, {4, 5, 6}, {7, 8}}? Partition {S i }

Partition Set S Non-empty subsets {S i } {1, 2, 3, 4, 5, 6, 7, 8, 9} Mutually exclusive S i ∩ S j = ϕ, for all i ≠ j Collectively exhaustive ∪ i S i = S {{1, 2, 3}, {4, 5, 6, 7}, {7, 8, 9}}? Partition {S i }

Partition Set S Non-empty subsets {S i } {1, 2, 3, 4, 5, 6, 7, 8, 9} Mutually exclusive S i ∩ S j = ϕ, for all i ≠ j Collectively exhaustive ∪ i S i = S {{1, 2, 3}, {4, 5, 6, 7}, {8, 9}}? Partition {S i }

Partition Set S{1, 2, 3, 4, 5, 6, 7, 8, 9} {{1, 2, 3}, {4, 5, 6, 7}, {8, 9}} Partition {S i }

Limited Subset of Partition Set S{1, 2, 3, 4, 5, 6, 7, 8, 9} {{1, 2, 3}, {4, 5, 6, 7}, {8, 9}} Partition {S i } Limits {l i } 3 2 1 Limited Subset (LS) X ⊆ S |X ∩ S i | ≤ l i, for all i {1, 2, 4, 5, 6, 8}?

Limited Subset of Partition Set S{1, 2, 3, 4, 5, 6, 7, 8, 9} {{1, 2, 3}, {4, 5, 6, 7}, {8, 9}} Partition {S i } Limits {l i } 3 2 1 Limited Subset (LS) X ⊆ S |X ∩ S i | ≤ l i, for all i {1, 2, 4, 5, 8}?

Limited Subset of Partition Set S{1, 2, 3, 4, 5, 6, 7, 8, 9} {{1, 2, 3}, {4, 5, 6, 7}, {8, 9}} Partition {S i } Limits {l i } 3 2 1 Limited Subset (LS) X ⊆ S {1, 2, 4, 5}? |X ∩ S i | ≤ l i, for all i

Limited Subset of Partition Set S{1, 2, 3, 4, 5, 6, 7, 8, 9} {{1, 2, 3}, {4, 5, 6, 7}, {8, 9}} Partition {S i } Limits {l i } 3 2 1 Limited Subset (LS) X ⊆ S Subset of an LS is an LSSubset system |X ∩ S i | ≤ l i, for all i

Subset System Set S {S i, i = 1, 2, …, n} is a partition {l 1,l 2,…,l n } are non-negative integers X ⊆ S ∈ I if X is a limited subset of partition

Subset System {l 1,l 2,…,l n } are non-negative integers X ⊆ S ∈ I if |X ∩ S i | ≤ l i for all i ∈ {1,2,…,n} (S, I ) is a matroid? Partition Matroid Set S {S i, i = 1, 2, …, n} is a partition

Linear Algebra

12341234 24682468 11111111 22222222 33333333 12121212 24242424 12121212 24242424 Matrix ASubset of columns {a 1,a 2,…,a k } Linearly independent (LI)? There exists no α ≠ 0 such that Σ i α i a i = 0 ✗

12341234 24682468 11111111 22222222 33333333 12121212 24242424 12121212 24242424 Matrix ASubset of columns {a 1,a 2,…,a k } Linearly independent (LI)? There exists no α ≠ 0 such that Σ i α i a i = 0 ✓

Linear Matroid Matrix A of size n x m, S = {1,2,…,m} X ⊆ S, A(X) = set of columns of A indexed by X X  I if and only if A(X) are linearly independent

Independent Set Matroid M = (S, I ) X ⊆ S is independent if X  I X ⊆ S is dependent if X ∉ I

Base of a Subset Matroid M = (S, I ) X is a base of U ⊆ S if it satisfies three properties (i) X ⊆ U(ii) X ∈ I (iii) There exists no U’ ∈ I, such that X ⊂ U’ ⊆ U subset of Uindependent Inclusionwise maximal

An Interesting Property M = (S, I ) is a subset system M is a matroid For all U ⊆ S, all bases of U have same size

An Interesting Property M = (S, I ) is a subset system M is a matroid For all U ⊆ S, all bases of U have same size Proof? X∈ IX∈ I Y∈ IY∈ I Base of X ∪ Y?

An Interesting Property M = (S, I ) is a subset system M is a matroid For all U ⊆ S, all bases of U have same size An alternate definition for matroids

Rank of a Subset Matroid M = (S, I ) U ⊆ S r M (U) = Size of a base of U

Base of a Matroid Matroid M = (S, I ) X is a base S

Rank of a Matroid Matroid M = (S, I ) r M = Rank of S

Subset System? Hereditary Property? Matroid? Exchange Property? Uniform, Graphic, Linear Matroids? Base of a subset? Rank of a subset? Recap

Matroids Maximum Weight Independent Set Submodular Functions Relationship Outline

Independent Set Matroid M = (S, I ) X ⊆ S is independent if X  I X ⊆ S is dependent if X ∉ I

Weight of an Independent Set Matroid M = (S, I ) Weight function w: S → Non-negative Real Weight of an independent set X The sum of weight of its elements

Weight of an Independent Set Matroid M = (S, I ) w(X) = ∑ s ∈ X w(s) Weight of an independent set X Weight function w: S → Non-negative Real

Example (Uniform Matroid) S = {1,2,…,9} sw(s) 110 25 32 41 53 66 712 82 91 k = 4 X = {1, 3, 5}

Example (Uniform Matroid) S = {1,2,…,9} sw(s) 110 25 32 41 53 66 712 82 91 k = 4 X = {1, 3, 5} w(X)? 15

Example (Graphic Matroid) v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 2 6 2 5 3 1 3 2 4 S = E Forest X

Example (Graphic Matroid) v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 2 6 2 5 3 1 3 2 4 S = E Forest X w(X)? 10

Maximum Weight Independent Set Matroid M = (S, I ) max X ∈ I w(X) Find an independent set with maximum weight Weight function w: S → Non-negative Real

Maximum Weight Independent Set Matroid M = (S, I ) max X ∈ I ∑ s ∈ X w(s) Find an independent set with maximum weight Weight function w: S → Non-negative Real

True or False There exists an optimal solution that is a base of the matroid TRUE

Example (Uniform Matroid) S = {1,2,…,9} sw(s) 110 25 32 41 53 66 712 82 91 k = 4 Feasible Solutions?

Example (Uniform Matroid) S = {1,2,…,9} sw(s) 110 25 32 41 53 66 712 82 91 k = 4 Optimal Solution?

Example (Uniform Matroid) S = {1,2,…,9} sw(s) 110 25 32 41 53 66 712 82 91 k = 4

Example (Graphic Matroid) v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 2 6 2 5 3 1 3 2 4 S = E Feasible Solutions?

Example (Graphic Matroid) v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 2 6 2 5 3 1 3 2 4 S = E Optimal Solution?

Example (Graphic Matroid) v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 2 6 2 5 3 1 3 2 4 S = E Efficient Algorithm?

Matroids Maximum Weight Independent Set –Greedy Algorithm –Efficiency –Optimality –Extensions Submodular Functions Relationship Outline

Greedy Algorithm Start with an empty set Repeat Pick a new element with maximum weight such that the new set is independent Until no more elements can be added Add the element to the set

Greedy Algorithm X ← ϕ Repeat Pick a new element with maximum weight such that the new set is independent Until no more elements can be added Add the element to the set

Greedy Algorithm X ← ϕ Repeat s* = argmax x ∈ S\X w(s) such that the new set is independent Until no more elements can be added Add the element to the set

Greedy Algorithm X ← ϕ Repeat s* = argmax x ∈ S\X w(s) such that X ∪ {s} ∈ I Until no more elements can be added Add the element to the set

Greedy Algorithm X ← ϕ Repeat s* = argmax x ∈ S\X w(s) such that X ∪ {s} ∈ I Until no more elements can be added X ← X ∪ {s}

Example (Uniform Matroid) S = {1,2,…,9} sw(s) 110 25 32 41 53 66 712 82 91 k = 4

Example (Graphic Matroid) v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v6v6 v6v6 v4v4 v4v4 v5v5 v5v5 v3v3 v3v3 2 6 2 5 3 1 3 2 4 S = E

Efficiency of Greedy Algorithm X ← ϕ Repeat s* = argmax x ∈ S\X w(s) such that X ∪ {s} ∈ I Until no more elements can be added X ← X ∪ {s} O(|S|)

Independence Testing Oracle Efficiency depends on independence testing Uniform matroid? Graphic matroid? Brute-force is not an option

Optimality: Necessity (S, I ) is a matroid (S, I ) admits an optimal greedy algorithm Proof?

Proof Sketch Matroid M = (S, I ) Proof by induction Current solution X X is part of optimal solution B (base of M)

Proof Sketch Next step chooses to add s If s ∈ B, trivial Otherwise, consider base B’ such that B’ = (B ∪ {s})\{t} w(B’) ≥ w(B) by construction X ∪ {s} ⊆ B’ ⊂ B ∪ {s} How to construct B’?

Optimality: Sufficiency (S, I ) is a matroid (S, I ) admits an optimal greedy algorithm Proof?

Proof Sketch X ∈ I Proof by contradiction There should exist an s ∈ Y\X, X ∪ {s} ∈ I Subset system M = (S, I ) Y ∈ I k = |X| < |Y|

Proof Sketch w(s) = k+2, if s ∈ X k+1, if s ∈ Y\X 0, otherwise Greedy solution has weight k(k+2) w(Y) ≥ (k+1)(k+1) > k(k+2) Recall k = |X|

Maximum Weight Base Matroid M = (S, I ) max X ∈ B w(X) Find a base with maximum weight Does greedy work? YES Weight function w: S → Real

Minimum Weight Base Matroid M = (S, I ) min X ∈ B w(X) Find a base with minimum weight Does greedy work? YES Weight function w: S → Real

Maximum Weight Independent Set Matroid M = (S, I ) max X ∈ I w(X) Find an independent set with maximum weight Does greedy work? YES Weight function w: S → Real

Matroids Submodular Functions Relationship Outline

Submodular Function Set S Function f over power set of S f(T) + f(U) ≥ f(T ∪ U) + f(T ∩ U) for all T, U ⊆ S

Supermodular Function Set S Function f over power set of S f(T) + f(U) ≤ f(T ∪ U) + f(T ∩ U) for all T, U ⊆ S

Modular Function Set S Function f over power set of S f(T) + f(U) = f(T ∪ U) + f(T ∩ U) for all T, U ⊆ S

Modular Function f(T) = ∑ s ∈ T w(s) + K Is f modular? All modular functions have above form? YES Prove at home

Matroids Submodular Functions –Diminishing Returns –Examples Relationship Outline

Diminishing Returns Define d f (s|T) = f(T ∪ {s}) - f(T) Gain by adding s to T If f is submodular, d f (s|T) is non-increasing

Diminishing Returns f(U ∪ {s}) + f(U ∪ {t}) ≥ f(U) + f(U ∪ {s,t}) for all U ⊆ S and distinct s,t ∈ S\U Necessary condition for submodularity Gain by adding s to T Define d f (s|T) = f(T ∪ {s}) - f(T) Proof?

Diminishing Returns Sufficient condition for submodularity Prove at homeGain by adding s to T f(U ∪ {s}) + f(U ∪ {t}) ≥ f(U) + f(U ∪ {s,t}) for all U ⊆ S and distinct s,t ∈ S\U Define d f (s|T) = f(T ∪ {s}) - f(T)

Matroids Submodular Functions –Diminishing Returns –Examples Relationship Outline

Set Theory

Set Unions T 1, T 2, …, T n ⊆ T f(U) = ∑ s ∈ U’ w(s), U’ = ∪ i ∈ U T i S = {1, 2, … n} Submodular Non-negative weight w(s) of element s ∈ T Minimum of f? Is f non-decreasing? 0 YES Prove at home

Graph Theory

Directed Graph Cuts Minimum of f? Digraph G = (V, A) f(U) = ∑ a ∈ out-arcs(U) c(a) S = V Proof? Submodular Non-negative capacity c(a) of arc a ∈ A Is f non-decreasing? 0 NO

Directed Graph Cuts Minimum of f over U ⊆ S\{t} such that s ∈ U? Digraph G = (V, A) f(U) = ∑ a ∈ out-arcs(U) c(a) S = V Proof? Submodular Non-negative capacity c(a) of arc a ∈ A Minimum s-t cut = Maximum s-t flow

Matroids Submodular Functions Relationship Outline

Rank Function of Matroid Matroid M = (S, I ) Rank function r X is independent if and only if r(X) = |X|

Property of Rank Function Set S For all T, U ⊆ S Rank function of a matroid r if T ⊆ U r(T) ≤ r(U) ≤ |U| Proof?

Property of Rank Function Set S For all T, U ⊆ S Rank function of a matroid r r(T ∪ U) + r(T ∩ U) ≤ r(T) + r(U) Proof? Rank function of a matroid is submodular

Proof Sketch Matroid M = (S, I ) X ⊆ T∩U, X ∈ I is inclusionwise maximal Y ⊆ T ∪ U, X ⊆ Y, Y ∈ I is inclusionwise maximal r(T∩U) = |X|Why?

Proof Sketch r(T ∪ U) = |Y| Why? Matroid M = (S, I ) X ⊆ T∩U, X ∈ I is inclusionwise maximal Y ⊆ T ∪ U, X ⊆ Y, Y ∈ I is inclusionwise maximal

Proof Sketch r(T) ≥ |Y∩T|Why? Matroid M = (S, I ) X ⊆ T∩U, X ∈ I is inclusionwise maximal Y ⊆ T ∪ U, X ⊆ Y, Y ∈ I is inclusionwise maximal

Proof Sketch Why?r(U) ≥ |Y∩U| Matroid M = (S, I ) X ⊆ T∩U, X ∈ I is inclusionwise maximal Y ⊆ T ∪ U, X ⊆ Y, Y ∈ I is inclusionwise maximal

Proof Sketch r(T) + r(U) ≥ |Y∩T| + |Y∩U| = |Y∩(T∩U)| + |Y∩(T ∪ U)| ≥ |X| + |Y| Matroid M = (S, I ) = r(T∩U) + r(T ∪ U)

Questions? Next time … Polytopes

Submodularity Reading Group Matroids, Submodular Functions M. Pawan Kumar

Similar presentations

Presentation on theme: "Submodularity Reading Group Matroids, Submodular Functions M. Pawan Kumar"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Submodularity Reading Group Matroids, Submodular Functions M. Pawan Kumar

Similar presentations

Presentation on theme: "Submodularity Reading Group Matroids, Submodular Functions M. Pawan Kumar"— Presentation transcript:

Similar presentations

About project

Feedback