Robust inversion, dimensionality reduction, and randomized sampling (Aleksandr Aravkin Michael P. Friedlander et al) Cong Fang Dec 14 th 2014

Stochastic optimization Basic gradient-descent with a fixed step has a strong linear convergence rate if the function is strong convex and is Lipschitz continuous Remove the strong-convexity,it has a has a sublinear convergencerate of O(1/k) If we use subgradient to replace the gradient, strong-convexity, it has a has a sublinear convergencerate of O(1/sqrt(k))

Approximate gradient g is a direction to decent solving by BFGS(quasi-Newton). a is a step to ensure the function to decent.(Wolfe and Armijo) The paper use the Limited-memory BFGS to get the Hessian of the function and do Armijo backtracking linesearch.

Stochastic gradient methods Suppose: Then it holds: If the function is strong convex, then:

Incremental-gradient methods Incremental-gradient methods are a special case of stochastic approximation. Here, rather than computing the full gradient on each iteration, a function is randomly selected among i ∈ {1,...,M}, and the gradient estimate is constructed With the additional assumption of strong convexity, it converges sublinearly.

Sampling methods The sampling approach described allows us to interpolate between the one-at-a-time incremental- gradient method at one extreme, and a full gradient method at the other.

Assume: Suppose

(a) The sample size (fraction of the total population of m = 1000) required to reduce the error linearly with error constant 0.9. (b) The corresponding cumulative number of samples used

Numerical experiments G is the (sparse) Jacobian of Au with respect to x, and u and v are solutions of the linear systems The forward model for frequency-domain acoustic FWI, for a single source function q, assumes that wave propagation in the earth is described by the scalar Helmholtz equation.

Experimental a.Relative error between the true and reconstructed models for least-squares,Huber, and Student t penalties b.Convergence of different optimization strategies on the Students t penalty:Limited- memory BFGS using the full gradient (“full”), incremental gradient with constant and decreasing step sizes, and the sampling approach Sampling batch size:

Experimental Simply solve a linear reverse problem. n=20(not sparse),m=1000. Add 30% salt & peper noise. Graduately accelerating sample size seems not work. Use Limited-memory BFGS and Wolfe linear search.

Thank you !