Robust inversion, dimensionality reduction, and randomized sampling (Aleksandr Aravkin Michael P. Friedlander et al) Cong Fang Dec 14 th 2014.

Slides:



Advertisements
Similar presentations
Bayesian Belief Propagation
Advertisements

Arc-length computation and arc-length parameterization
Approximate Analytical/Numerical Solutions to the Groundwater Transport Problem CWR 6536 Stochastic Subsurface Hydrology.
Accelerated, Parallel and PROXimal coordinate descent IPAM February 2014 APPROX Peter Richtárik (Joint work with Olivier Fercoq - arXiv: )
Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.
CS479/679 Pattern Recognition Dr. George Bebis
Optimization Tutorial
Exploiting Sparse Markov and Covariance Structure in Multiresolution Models Presenter: Zhe Chen ECE / CMR Tennessee Technological University October 22,
CHAPTER 8 A NNEALING- T YPE A LGORITHMS Organization of chapter in ISSO –Introduction to simulated annealing –Simulated annealing algorithm Basic algorithm.
Middle Term Exam 03/01 (Thursday), take home, turn in at noon time of 03/02 (Friday)
More MR Fingerprinting
An Introduction to Sparse Coding, Sparse Sensing, and Optimization Speaker: Wei-Lun Chao Date: Nov. 23, 2011 DISP Lab, Graduate Institute of Communication.
Inexact SQP Methods for Equality Constrained Optimization Frank Edward Curtis Department of IE/MS, Northwestern University with Richard Byrd and Jorge.
1 L-BFGS and Delayed Dynamical Systems Approach for Unconstrained Optimization Xiaohui XIE Supervisor: Dr. Hon Wah TAM.
Function Optimization Newton’s Method. Conjugate Gradients
1cs542g-term Notes  Extra class this Friday 1-2pm  If you want to receive s about the course (and are auditing) send me .
Tutorial 12 Unconstrained optimization Conjugate gradients.
1 L-BFGS and Delayed Dynamical Systems Approach for Unconstrained Optimization Xiaohui XIE Supervisor: Dr. Hon Wah TAM.
Optimization Methods Unconstrained optimization of an objective function F Deterministic, gradient-based methods Running a PDE: will cover later in course.
Planning operation start times for the manufacture of capital products with uncertain processing times and resource constraints D.P. Song, Dr. C.Hicks.
Advanced Topics in Optimization
Semi-Stochastic Gradient Descent Methods Jakub Konečný (joint work with Peter Richtárik) University of Edinburgh SIAM Annual Meeting, Chicago July 7, 2014.
GRADIENT PROJECTION FOR SPARSE RECONSTRUCTION: APPLICATION TO COMPRESSED SENSING AND OTHER INVERSE PROBLEMS M´ARIO A. T. FIGUEIREDO ROBERT D. NOWAK STEPHEN.
Staffing Service Systems via Simulation Julius Atlason, Marina Epelman University of Michigan Shane Henderson Cornell University.
Error estimates for degenerate parabolic equation Yabin Fan CASA Seminar,
Unconstrained Optimization Rong Jin. Logistic Regression The optimization problem is to find weights w and b that maximizes the above log-likelihood How.
Compressed Sensing Compressive Sampling
3.7. O THER G AME P HYSICS A PPROACHES Overview of other game engine physics approaches.
Computational Optimization
UNCONSTRAINED MULTIVARIABLE
Collaborative Filtering Matrix Factorization Approach
Jinhui Tang †, Shuicheng Yan †, Richang Hong †, Guo-Jun Qi ‡, Tat-Seng Chua † † National University of Singapore ‡ University of Illinois at Urbana-Champaign.
1 Hybrid methods for solving large-scale parameter estimation problems Carlos A. Quintero 1 Miguel Argáez 1 Hector Klie 2 Leticia Velázquez 1 Mary Wheeler.
CHAPTER 4 S TOCHASTIC A PPROXIMATION FOR R OOT F INDING IN N ONLINEAR M ODELS Organization of chapter in ISSO –Introduction and potpourri of examples Sample.
1 Jorge Nocedal Northwestern University With S. Hansen, R. Byrd and Y. Singer IPAM, UCLA, Feb 2014 A Stochastic Quasi-Newton Method for Large-Scale Learning.
Computing a posteriori covariance in variational DA I.Gejadze, F.-X. Le Dimet, V.Shutyaev.
Qualifier Exam in HPC February 10 th, Quasi-Newton methods Alexandru Cioaca.
1 Numerical Shape Optimisation in Blow Moulding Hans Groot.
Probabilistic Robotics Bayes Filter Implementations.
Tbilisi, GGSWBS'14 Optimization for inverse modelling Ketevan Kasradze 1 Hendrik Elbern 1,2
Center for Evolutionary Functional Genomics Large-Scale Sparse Logistic Regression Jieping Ye Arizona State University Joint work with Jun Liu and Jianhui.
Chin-Yu Huang Department of Computer Science National Tsing Hua University Hsinchu, Taiwan Optimal Allocation of Testing-Resource Considering Cost, Reliability,
Efficient computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data using the L 1 Norm Anders Eriksson and Anton van den Hengel.
Research Vignette: The TransCom3 Time-Dependent Global CO 2 Flux Inversion … and More David F. Baker NCAR 12 July 2007 David F. Baker NCAR 12 July 2007.
Sensitivity derivatives Can obtain sensitivity derivatives of structural response at several levels Finite difference sensitivity (section 7.1) Analytical.
Quasi-Newton Methods of Optimization Lecture 2. General Algorithm n A Baseline Scenario Algorithm U (Model algorithm for n- dimensional unconstrained.
Data Modeling Patrice Koehl Department of Biological Sciences National University of Singapore
Exact Differentiable Exterior Penalty for Linear Programming Olvi Mangasarian UW Madison & UCSD La Jolla Edward Wild UW Madison December 20, 2015 TexPoint.
Inexact SQP methods for equality constrained optimization Frank Edward Curtis Department of IE/MS, Northwestern University with Richard Byrd and Jorge.
Machine learning optimization Usman Roshan. Machine learning Two components: – Modeling – Optimization Modeling – Generative: we assume a probabilistic.
INTRO TO OPTIMIZATION MATH-415 Numerical Analysis 1.
Using Neumann Series to Solve Inverse Problems in Imaging Christopher Kumar Anand.
Exact Differentiable Exterior Penalty for Linear Programming
Computational Optimization
Kabanikhin S. I., Krivorotko O. I.
A Simple Artificial Neuron
Generalization and adaptivity in stochastic convex optimization
3.7. Other Game Physics Approaches
Probabilistic Models for Linear Regression
CS5321 Numerical Optimization
Chapter 6. Large Scale Optimization
Collaborative Filtering Matrix Factorization Approach
Estimating Networks With Jumps
Chapter 10. Numerical Solutions of Nonlinear Systems of Equations
Slides for Introduction to Stochastic Search and Optimization (ISSO) by J. C. Spall CHAPTER 15 SIMULATION-BASED OPTIMIZATION II: STOCHASTIC GRADIENT AND.
CS5321 Numerical Optimization
Neural Network Training
Computer Animation Algorithms and Techniques
Chapter 6. Large Scale Optimization
Presentation transcript:

Robust inversion, dimensionality reduction, and randomized sampling (Aleksandr Aravkin Michael P. Friedlander et al) Cong Fang Dec 14 th 2014

Stochastic optimization Basic gradient-descent with a fixed step has a strong linear convergence rate if the function is strong convex and is Lipschitz continuous Remove the strong-convexity,it has a has a sublinear convergencerate of O(1/k) If we use subgradient to replace the gradient, strong-convexity, it has a has a sublinear convergencerate of O(1/sqrt(k))

Approximate gradient g is a direction to decent solving by BFGS(quasi-Newton). a is a step to ensure the function to decent.(Wolfe and Armijo) The paper use the Limited-memory BFGS to get the Hessian of the function and do Armijo backtracking linesearch.

Stochastic gradient methods Suppose: Then it holds: If the function is strong convex, then:

Incremental-gradient methods Incremental-gradient methods are a special case of stochastic approximation. Here, rather than computing the full gradient on each iteration, a function is randomly selected among i ∈ {1,...,M}, and the gradient estimate is constructed With the additional assumption of strong convexity, it converges sublinearly.

Sampling methods The sampling approach described allows us to interpolate between the one-at-a-time incremental- gradient method at one extreme, and a full gradient method at the other.

Assume: Suppose

(a) The sample size (fraction of the total population of m = 1000) required to reduce the error linearly with error constant 0.9. (b) The corresponding cumulative number of samples used

Numerical experiments G is the (sparse) Jacobian of Au with respect to x, and u and v are solutions of the linear systems The forward model for frequency-domain acoustic FWI, for a single source function q, assumes that wave propagation in the earth is described by the scalar Helmholtz equation.

Experimental a.Relative error between the true and reconstructed models for least-squares,Huber, and Student t penalties b.Convergence of different optimization strategies on the Students t penalty:Limited- memory BFGS using the full gradient (“full”), incremental gradient with constant and decreasing step sizes, and the sampling approach Sampling batch size:

Experimental Simply solve a linear reverse problem. n=20(not sparse),m=1000. Add 30% salt & peper noise. Graduately accelerating sample size seems not work. Use Limited-memory BFGS and Wolfe linear search.

Thank you !