1. Vector Calculus and Quadratic function
10/13/2018
By Nafees Ahamad

2. Gradient of a function Consider a multivariable, scalar function f(x)
f(x)=f(xnx1)=f(x1,x2…xn) where 𝑥=𝑥 𝑛×1 = 𝑥 1 𝑥 𝑥 𝑛 𝑖𝑠 𝑎 𝑣𝑒𝑐𝑡𝑜𝑟
Gradient of f(x)
𝛻𝑓 𝑥 = 𝑔𝑟𝑎𝑑 𝑓(𝑥) 𝑥=𝑥∗ = 𝜕𝑓(𝑥) 𝜕𝑥 𝑥=𝑥∗
where 𝑥=𝑥 𝑛×1 =𝑥 𝑛×1 ∗ = 𝑥 1 ∗ 𝑥 2 ∗ 𝑥 𝑛 ∗ 𝑖𝑠 𝑎 𝑣𝑒𝑐𝑡𝑜𝑟
10/13/2018
By Nafees Ahamad

3. Gradient of a function…
𝛻𝑓 𝑥 = 𝜕𝑓(𝑥) 𝜕 𝑥 1 𝜕𝑓(𝑥) 𝜕 𝑥 𝜕𝑓(𝑥) 𝜕 𝑥 𝑛 𝑥=𝑥∗ = 𝜕𝑓(𝑥) 𝜕 𝑥 1 𝜕𝑓(𝑥) 𝜕 𝑥 𝜕𝑓(𝑥) 𝜕 𝑥 𝑛 𝑥=𝑥∗ 𝑇
So gradient of a scalar function is a column vector
(Gradient=> derivative of a scalar function w.r.t vector)
T→Transpose
10/13/2018
By Nafees Ahamad

4. Example 1: Calculate the gradient vector for the function
𝑓 𝑥 = 𝑥 𝑥 1 𝑥 2 + 3𝑥 𝑥 1 𝑥 2 𝑥 3 + 𝑥 3 3
at 𝑥 3×1 = 𝑇
Solution
𝜕𝑓(𝑥) 𝜕 𝑥 1 =2 𝑥 1 +2 𝑥 2 +4 𝑥 2 𝑥 3
𝜕𝑓(𝑥) 𝜕 𝑥 2 =2 𝑥 1 +6 𝑥 2 +4 𝑥 1 𝑥 3
𝜕𝑓(𝑥) 𝜕 𝑥 3 =4 𝑥 1 𝑥 2 +3 𝑥 3 2
10/13/2018
By Nafees Ahamad

5. Example 1…
So gradient
𝛻𝑓 𝑥 = 𝜕𝑓(𝑥) 𝜕 𝑥 1 𝜕𝑓(𝑥) 𝜕 𝑥 2 𝜕𝑓(𝑥) 𝜕 𝑥 𝑥=𝑥∗ = 2 𝑥 1 +2 𝑥 2 +4 𝑥 2 𝑥 3 2 𝑥 1 +6 𝑥 2 +4 𝑥 1 𝑥 3 4 𝑥 1 𝑥 2 +3 𝑥 𝑥= 𝑥 ∗ = =
Geometrically , the gradient of a function is normal to the tangent plane at the point x=x*
10/13/2018
By Nafees Ahamad

6. 2nd derivative of a real valued function:
First take the derivative of a function (scalar) that means gradient of a function. Which results into a vector. Now again take the derivate of this vector w.r.t vector which will result into a matrix.
𝛻 2 𝑓 𝑥 = 𝜕 𝜕𝑓(𝑥) 𝜕𝑥 𝜕𝑥 𝑥=𝑥∗ = 𝜕 2 𝑓(𝑥) 𝜕𝑥 2 𝑥=𝑥∗
𝛻 2 𝑓 𝑥 = 𝜕 𝜕𝑥 𝜕𝑓(𝑥) 𝜕 𝑥 1 𝜕𝑓(𝑥) 𝜕 𝑥 𝜕𝑓(𝑥) 𝜕 𝑥 𝑛 𝑥=𝑥∗ 𝑇
10/13/2018
By Nafees Ahamad

7. 2nd derivative of a real valued function…
𝛻 2 𝑓 𝑥 = 𝜕 2 𝑓(𝑥) 𝜕 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝜕𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝑛 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 𝜕𝑓(𝑥) 𝜕 𝑥 2 𝑥 𝑛 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝑛 𝜕 2 𝑓(𝑥) 𝜕 𝑥 2 𝑥 𝑛 𝜕 2 𝑓(𝑥) 𝜕 𝑥 𝑛 𝑥= 𝑥 ∗
It is a symmetrical nxn matrix (aij=aji i≠j ) and known Hessian Matrix
10/13/2018
By Nafees Ahamad

8. Example 2 Calculate 2nd derivative of function
𝑓 𝑥 = 𝑥 𝑥 1 𝑥 2 + 3𝑥 𝑥 1 𝑥 2 𝑥 3 + 𝑥 3 3
at 𝑥 3×1 = 𝑇
Solution:
𝜕𝑓(𝑥) 𝜕 𝑥 1 =2 𝑥 1 +2 𝑥 2 +4 𝑥 2 𝑥 3
𝜕𝑓(𝑥) 𝜕 𝑥 2 =2 𝑥 1 +6 𝑥 2 +4 𝑥 1 𝑥 3
𝜕𝑓(𝑥) 𝜕 𝑥 3 =4 𝑥 1 𝑥 2 +3 𝑥 3 2
10/13/2018
By Nafees Ahamad

9. Example 2…
𝛻 2 𝑓 𝑥 = 𝜕 2 𝑓(𝑥) 𝜕 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 2 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 2 𝑥 𝜕 2 𝑓(𝑥) 𝜕 𝑥 𝑥= 𝑥 ∗
10/13/2018
By Nafees Ahamad

10. Example 2… 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 2 =2 𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 2 =2+4 𝑥 3
𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 2 =2
𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 2 =2+4 𝑥 3
𝜕 2 𝑓(𝑥) 𝜕 𝑥 1 𝑥 3 =4 𝑥 2
𝜕 2 𝑓(𝑥) 𝜕 𝑥 2 2 =6
𝜕 2 𝑓(𝑥) 𝜕 𝑥 2 𝑥 3 =4 𝑥 1
𝜕 2 𝑓(𝑥) 𝜕 𝑥 3 2 =6 𝑥 3
𝛻 2 𝑓 𝑥 = 𝑥 3 4 𝑥 𝑥 𝑥 1 4 𝑥 2 4 𝑥 1 6 𝑥 3 𝑥= 𝑥 ∗ = 𝑇
𝛻 2 𝑓 𝑥 =
10/13/2018
By Nafees Ahamad

11. Quadratic functions
Suppose we have a function of two variables x1 & x2
The quadratic function for these variables may be
𝑓 𝑥 1 , 𝑥 2 =𝑎 𝑥 1 2 +𝑏 𝑥 1 𝑥 2 +𝑐 𝑥 2 2 −−−(1)
Above equation (1) may be written as
𝑓 𝑥 1 , 𝑥 2 = 𝑥 1 𝑥 𝑎 𝑏 0 𝑐 𝑥 1 𝑥 2 = 𝑥 𝑇 𝑃𝑥 Or
𝑓 𝑥 1 , 𝑥 2 = 𝑥 1 𝑥 𝑎 0 𝑏 𝑐 𝑥 1 𝑥 2 = 𝑥 𝑇 𝑃𝑥
We can change above diagonal elements and get many (infinite) solution
Say Matrix P
10/13/2018
By Nafees Ahamad

12. Quadratic functions … 𝑓 𝑥 1 , 𝑥 2 = 𝑥 1 𝑥 2 𝑎 −𝑏 2𝑏 𝑐 𝑥 1 𝑥 2 = 𝑥 𝑇 𝑃𝑥
𝑓 𝑥 1 , 𝑥 2 = 𝑥 1 𝑥 𝑎 −𝑏 2𝑏 𝑐 𝑥 1 𝑥 2 = 𝑥 𝑇 𝑃𝑥
𝑓 𝑥 1 , 𝑥 2 = 𝑥 1 𝑥 𝑎 𝑏 2 𝑏 2 𝑐 𝑥 1 𝑥 2 = 𝑥 𝑇 𝑃𝑥
So Matrix P is not unique
P=Symmetrical matrix
10/13/2018
By Nafees Ahamad

13. General Quadratic function
A quadratic function of ‘n’ variables
𝑓 𝑥 1 , 𝑥 2 , 𝑥 3 ,… 𝑥 𝑛 =𝑓 𝑥 𝑛×1 = 𝑥 𝑇 𝑃𝑥
Where 𝑥= 𝑥 1 𝑥 𝑥 𝑛 𝑛×1 and Pnxn symmetric matrix
10/13/2018
By Nafees Ahamad

14. General Quadratic function…
Suppose P is not a symmetric matrix then
𝑓 𝑥 = 𝑥 𝑇 𝑃𝑥= 1 2 𝑥 𝑇 𝑃𝑥 𝑥 𝑇 𝑃𝑥
𝑓 𝑥 = 𝑥 𝑇 𝑃𝑥= 1 2 𝑥 𝑇 𝑃𝑥 𝑥 𝑇 𝑃𝑥 𝑇
𝑓 𝑥 = 𝑥 𝑇 𝑃𝑥= 1 2 𝑥 𝑇 𝑃𝑥 𝑥 𝑇 𝑃 𝑇 𝑥 ∵ 𝐴𝐵𝐶 𝑇 = 𝐶 𝑇 𝐵 𝑇 𝐴 𝑇 & 𝑥 𝑇 𝑇 =𝑥
𝑓 𝑥 = 𝑥 𝑇 𝑃𝑥= 𝑥 𝑇 𝑃+ 𝑃 𝑇 2 𝑥= 𝑥 𝑇 Qx
f(x) will be scalar function i.e f(x)=10 say, so it may be written as F(x)=5+5
1 2 𝑥 𝑇 𝑃𝑥 is scalar so we can take transpose of it
Always Symmetrical matrix
10/13/2018
By Nafees Ahamad

15. Next:
Now some other concepts like + definite, + semi definite, - definite & - sem definite.
10/13/2018
By Nafees Ahamad