Derivatives and Gradients

Slides:

Advertisements

Similar presentations

Courant and all that Consistency, Convergence Stability Numerical Dispersion Computational grids and numerical anisotropy The goal of this lecture is to.

Advertisements

Multi-Layer Perceptron (MLP)

Yi Heng Second Order Differentiation Bommerholz – Summer School 2006.

The Chain Rule Section 3.6c.

Partial Differential Equations

Using a tangent line approximation of the function, find an approximate value for.

Optimization of thermal processes

Lecture 18 - Numerical Differentiation

Engineering Optimization – Concepts and Applications Engineering Optimization Concepts and Applications Fred van Keulen Matthijs Langelaar CLA H21.1

Empirical Maximum Likelihood and Stochastic Process Lecture VIII.

Supervised learning 1.Early learning algorithms 2.First order gradient methods 3.Second order gradient methods.

Lecture 2: Numerical Differentiation. Derivative as a gradient

Computer-Aided Analysis on Energy and Thermofluid Sciences Y.C. Shih Fall 2011 Chapter 6: Basics of Finite Difference Chapter 6 Basics of Finite Difference.

Chapter 1 Introduction The solutions of engineering problems can be obtained using analytical methods or numerical methods. Analytical differentiation.

Copyright © 2005 Pearson Education, Inc. Publishing as Pearson Addison-Wesley.

© 2010 Pearson Education, Inc. All rights reserved.

AppxA_01fig_PChem.jpg Complex Numbers i. AppxA_02fig_PChem.jpg Complex Conjugate.

Unconstrained Optimization Problem

CVEN Exam 1 Review. Matlab Basic commands and syntax Basic commands and syntax Creation of functions and programs Creation of functions and programs.

ECIV 301 Programming & Graphics Numerical Methods for Engineers Lecture 5 Approximations, Errors and The Taylor Series.

Advanced Topics in Optimization

Dr. Jie Zou PHY Chapter 7 Numerical Differentiation: 1 Lecture (I) 1 Ref: “Applied Numerical Methods with MATLAB for Engineers and Scientists”, Steven.

The Islamic University of Gaza Faculty of Engineering Civil Engineering Department Numerical Analysis ECIV 3306 Chapter 23 Numerical Differentiation.

(MTH 250) Lecture 24 Calculus. Previous Lecture’s Summary Multivariable functions Limits along smooth curves Limits of multivariable functions Continuity.

Biointelligence Laboratory, Seoul National University

Backpropagation An efficient way to compute the gradient Hung-yi Lee.

Rules of differentiation REVIEW:. The Chain Rule.

Review Taylor Series and Error Analysis Roots of Equations

Neural Network Introduction Hung-yi Lee. Review: Supervised Learning Training: Pick the “best” Function f * Training Data Model Testing: Hypothesis Function.

Block 4 Nonlinear Systems Lesson 14 – The Methods of Differential Calculus The world is not only nonlinear but is changing as well 1 Narrator: Charles.

1 EEE 431 Computational Methods in Electrodynamics Lecture 4 By Dr. Rasime Uyguroglu

MA Day 25- February 11, 2013 Review of last week’s material Section 11.5: The Chain Rule Section 11.6: The Directional Derivative.

Chapter 4 Techniques of Differentiation Sections 4.1, 4.2, and 4.3.

Derivatives In modern structural analysis we calculate response using fairly complex equations. We often need to solve many thousands of simultaneous equations.

Matrix Differential Calculus By Dr. Md. Nurul Haque Mollah, Professor, Dept. of Statistics, University of Rajshahi, Bangladesh Dr. M. N. H. MOLLAH.

Linear Discrimination Reading: Chapter 2 of textbook.

Akram Bitar and Larry Manevitz Department of Computer Science

11 1 Backpropagation Multilayer Perceptron R – S 1 – S 2 – S 3 Network.

Finite Difference Methods Definitions. Finite Difference Methods Approximate derivatives ** difference between exact derivative and its approximation.

Today’s class Numerical Differentiation Finite Difference Methods Numerical Methods Lecture 14 Prof. Jinbo Bi CSE, UConn 1.

Sensitivity derivatives Can obtain sensitivity derivatives of structural response at several levels Finite difference sensitivity (section 7.1) Analytical.

Engineering Analysis – Computational Fluid Dynamics –

© Imperial College London Numerical Solution of Differential Equations 3 rd year JMC group project ● Summer Term 2004 Supervisor: Prof. Jeff Cash Saeed.

Exam 1 Oct 3, closed book Place ITE 119, Time:12:30-1:45pm

Numerical Methods Multidimensional Gradient Methods in Optimization- Theory

Neural Networks Vladimir Pleskonjić 3188/ /20 Vladimir Pleskonjić General Feedforward neural networks Inputs are numeric features Outputs are in.

Richard D. Neidinger Davidson College MOSAIC Kick-Off Workshop, IMA, MN Summer 2010 Automatic Differentiation using MATLAB OOP.

Signal & Weight Vector Spaces

Dr. Mujahed AlDhaifallah ( Term 342)

MA Day 34- February 22, 2013 Review for test #2 Chapter 11: Differential Multivariable Calculus.

Derivative Shortcuts -Power Rule -Product Rule -Quotient Rule -Chain Rule.

NUMERICAL DIFFERENTIATION or DIFFERENCE APPROXIMATION Used to evaluate derivatives of a function using the functional values at grid points. They are.

D Nagesh Kumar, IISc Water Resources Systems Planning and Management: M2L2 Introduction to Optimization (ii) Constrained and Unconstrained Optimization.

6.5.4 Back-Propagation Computation in Fully-Connected MLP.

§ 4.2 The Exponential Function e x.

Chapter 14 Partial Derivatives.

Numerical Differentiation

MTH1170 Implicit Differentiation

Computing Gradient Hung-yi Lee 李宏毅

The taylor series and Numerical Differentiation

Java Implementation of Optimal Brain Surgeon

L5 Optimal Design concepts pt A

Backpropagation.

SE301: Numerical Methods Topic 8 Ordinary Differential Equations (ODEs) Lecture KFUPM (Term 101) Section 04 Read , 26-2, 27-1 CISE301_Topic8L2.

The taylor series and Numerical Differentiation

Backpropagation.

NN & Optimization Yue Wu.

More with Rules for Differentiation

Dhruv Batra Georgia Tech

Akram Bitar and Larry Manevitz Department of Computer Science

Presentation transcript:

Derivatives and Gradients

Derivatives Derivatives tell us which direction to search for a solution

Derivatives Slope of Tangent Line

Derivatives

Derivatives in Multiple Dimensions Gradient Vector Hessian Matrix

Derivatives in Multiple Dimensions Directional Derivative

Numerical Differentiation Finite Difference Methods Complex Step Method

Numerical Differentiation: Finite Difference Derivation from Taylor series expansion

Numerical Differentiation: Finite Difference Neighboring points are used to approximate the derivative h too small causes numerical cancellation errors

Numerical Differentiation: Finite Difference Error Analysis Forward Difference: O(h) Central Difference: O(h2)

Numerical Differentiation: Complex Step Taylor series expansion using imaginary step

Numerical Differentiation Error Comparison

Automatic Differentiation Evaluate a function and compute partial derivatives simultaneously using the chain rule of differentiation

Automatic Differentiation Forward Accumulation is equivalent to expanding a function using the chain rule and computing the derivatives inside-out Requires n-passes to compute n-dimensional gradient Example

Automatic Differentiation Forward Accumulation

Automatic Differentiation Forward Accumulation

Automatic Differentiation Forward Accumulation

Automatic Differentiation Reverse accumulation is performed in single run using two passes over an n-dimensional function (forward and back) Note: this is central to the backpropagation algorithm used to train neural networks Many open-source software implementations are available

Summary Derivatives are useful in optimization because they provide information about how to change a given point in order to improve the objective function For multivariate functions, various derivative-based concepts are useful for directing the search for an optimum, including the gradient, the Hessian, and the directional derivative One approach to numerical differentiation includes finite difference approximations

Summary Complex step method can eliminate the effect of subtractive cancellation error when taking small steps Analytic differentiation methods include forward and reverse accumulation on computational graphs