Example: Feature selection Given random variables Y, X1, … Xn Want to predict Y from subset XA = (Xi1,…,Xik) Want k most informative features: A* = argmax IG(XA; Y) s.t. |A| · k where IG(XA; Y) = H(Y) - H(Y | XA) Problem inherently combinatorial! Y “Sick” X1 “Fever” X2 “Rash” X3 “Male” Naïve Bayes Model Uncertainty before knowing XA Uncertainty after knowing XA

Key property: Diminishing returns Selection A = {} Selection B = {X2,X3} Y “Sick” Y “Sick” X2 “Rash” X3 “Male” X1 “Fever” Adding X1 will help a lot! Adding X1 doesn’t help much New feature X1 B + s Large improvement Submodularity: A + s Small improvement For Aµ B, z(A [ {s}) – z(A) ¸ z(B [ {s}) – z(B)

Submodular set functions Set function z on V is called submodular if For all A,B µ V: z(A)+z(B) ¸ z(A[B)+z(AÅB) Equivalent diminishing returns characterization: + ¸ + A A [ B B AÅB B + S Large improvement Submodularity: A + S Small improvement For AµB, sB, z(A [ {s}) – z(A) ¸ z(B [ {s}) – z(B)

Example: Set cover Want to cover floorplan with discs Place sensors in building Possible locations V For A µ V: z(A) = “area covered by sensors placed at A” Node predicts values of positions with some radius Formally: W finite set, collection of n subsets Si µ W For A µ V={1,…,n} define z(A) = |i2 A Si|

Set cover is submodular A={S1,S2} S1 S2 S’ z(A[{S’})-z(A) ¸ z(B[{S’})-z(B) S1 S2 S3 S’ S4 B = {S1,S2,S3,S4}