Ugo Montanari On the optimal approximation of descrete functions with low- dimentional tables.

Overview Introduction Approximation with a sum of low-dimensional functions Optomal approximation with a given interaction graph Optimal approximation with a fixed amount of memory

Introduction Problem of storing large high-dimentional arrays is often critical (dynamic programming optimization techniques, belief propagation etc.)‏ Montanari proposes methos of optimal approximation (in the least square sence) of the given function with a sum of lower-dimentional functions.

Advantages The decoding process is very simple (a fixed number of summations)‏ The compression ratio often is high, mean error can be small, if the interaction between separated variables is limited The approximation function has a form of a sum of terms and is therefore suitable for the dynamic programming optimization

Approximation with a sum of low- dimentional functions F – function of n discrete variables (with same domains)‏

Lettice

Approximation with a sum of low- dimentional functions

Average projection of the function:

Proper function: A function g(X i ) such that its average projections on all the subsets of X i are identically zero will be called a proper function of X i.

Theorem 2.1 The set S i of all the proper functions of X i is a vector space and is called the proper space of X i. Theorem: The proper space S i of all the elements X i of lattice L are mutually orthogonal. Proof:

Characteristic function B B:L -> 0,1 Monotonocity constraint: The meaning of the characteristic function B is to specify the form of an approximate sum of terms for function F

Example of characteristic function We want to approximate function F(x 1,x 2,x 3 ) with a sum of the form F = f 1 (x 1,x 2 )+f 2 (x 2,x 3 )‏ Function B:

Characteristic space S B :

Problem A

Algorithm A that solves Problem A Step 1. Compute the average projections of F on all elements X i of lattice L. Step 2. Let Step 3. Execute next step for all r = 1,...,n Step 4. For all elements X i of L having cardinality r, let where the summation is extended to all X j of L smaller then X i Compute function:

Theorem 2.2 Theorem 2.2 proves validity of Algorithm A

Proof of theorem 2.2 (a) For every we have: (b) and (c). We assume inductively the thesis is true for function k j (X j ) and spaces S j with cardinality X j smaller then r, and prove for r.

Proof of theorem 2.2 cont'd Prove that :  If, then From written as

Proof of theorem 2.2 cont'd is proved to be solution to Problem A

Optimal approximation with a given interaction graph Sum of terms: Interaction graph: Alternative sum of terms:

Problem B Given function F and interaction graph G find the sum such that G is the interaction graph of and the error |F- | is minimal Note: interaction graph does not define uniquely the form of approximating function, so Problem B is not trivially reducibleto Problem A

Theorem 3.1 Theorem proves that the form of the optimal approximating sum depends only on given interaction graph G, and not upon the actual values of F. Theorem: Characteristic function B of an optimal approximating sum is computable as follows. We have B( X i ) = 1 iff the set of vertices W i corresponding to the set of variables X i defines a complete subgraph of G.

Proof of theorem 3.1

Example Problem B reduces to: - Finding all complete subgraphs of graph G - Solving problem A

Optimal approximation with a fixed amount of memory Problem C: Given function F find a sum whose terms can be stored as tables in no more than M cells of memory and such that the error |F- | is minimal

2 ways of storing a sum 1) The sum reqires 2N^2 cells 2) Store 6 functions from the table, such that if any of arguments of f 1 : f 5 is zero, then the value of function is zero and it's not stored. Total storage space:

2 ways of storing a sum In general, first methos requires cells, where summation is extended to all maximal sets. Second method requires Cells, but the summation extends to all sets and is optimal, because it is exactly equal to the number of dimentions of vector space

2 ways of storing a sum

Error By definition, Thus

Translation of Problem C into integer programming problem (0,1) resticted Problem D: Determine the integer variables y i (i=1,...,m) (0,1) restricted such that with the constraints

Correnpondence between Problem C and Problem D In Problem D both the objective function and the constraints are linear. Therefore linear interger programming methods apply.