1. Slides for Introduction to Stochastic Search and Optimization (ISSO) by J. C. Spall
CHAPTER 15 SIMULATION-BASED OPTIMIZATION II: STOCHASTIC GRADIENT AND SAMPLE PATH METHODS
Organization of chapter in ISSO
Introduction to gradient estimation
Interchange of derivative and integral
Gradient estimation techniques
Likelihood ratio/score function (LR/SF)
Infinitesimal perturbation analysis (IPA)
Optimization with gradient estimates
Sample path method

2. Issues in Gradient Estimation
Estimate the gradient of the loss function with respect to parameters for optimization from simulation outputs
where L(q) is a scalar-valued loss function to minimize and q is a p-dimensional vector of parameters
Essential properties of gradient estimates
Unbiased:
Small variance

3. Two Types of Parameters
where V is the random effect in the system, is the probability density function of V
Distributional parameters qD: Elements of q that enter via their effect on probability distribution of V. For example, if scalar V has distribution N(m,s2), then m and s2 are distributional parameters
Structural parameters qS: Elements of q that have effects directly on the loss function (via Q)
Distinction not always obvious

4. Interchange of Derivative and Integral
Unbiased gradient estimations using only one simulation require the interchange of derivative and integral:
Above generally not true. Technical conditions needed for validity:
Q ·pV and are continuous
Above has implications in practical applications

5. A General Form of Gradient Estimate
Assume that all the conditions required for the exchange of derivative and integral are satisfied,
Hence, an unbiased gradient estimate can be obtained as
Output from one simulation!

6. Two Gradient Estimates: LR/SF and IPA
pure LR/SF
pure IPA
Likelihood Ratio/ Score Function (LR/SF): only distributional parameters
Infinitestimal Perturbation Analysis (IPA): only structural parameters

7. Comparison of Pure LR/SF and IPA
In practice, neither extreme (LR/SF or IPA) may provide a framework for reasonable implementation:
LR/SF may require deriving a complex distribution function starting from U(0,1)
IPA may lead to intractable Q/q with a complex Q(q,V)
Pure LR/SF gradient estimate tend to suffer from large variance (variance can grow with the number of components in V)
Pure IPA may result in a Q(q,V) that fails to meet the conditions for valid interchange of derivative and integral. Hence can lead to biased gradient estimate.
In many cases where IPA is feasible, it leads to low variance gradient estimate

8. A Simple Example: Exponential Distribution
Let Z be exponential random variable with mean q. That is
. Define L = E(Z) = q. Then L/q = 1.
LR/SF estimate: V = Z; Q(q,V) = V.
IPA estimate: V = U(0,1); Q(q,V) = -qlogV (Z = -qlogV).
Both of LR/SF and IPA estimators are unbiased

9. Stochastic Optimization with Gradient Estimate
Use the gradient estimates in the root-finding stochastic approximation (SA) algorithm to minimize the loss function L(q) = E[Q(q,V)]: Find q* such that g(q*) = 0 based on simulation outputs
A general root-finding SA algorithm:
where ak is the step size with
If Yk is unbiased and has bounded variance (and other appropriate assumptions hold), then (a.s.)
an estimate of

10. Simulation-Based Optimization
Use gradient estimate derived from one simulation run in the iteration of SA:
where Vk is the realization of V from a simulation run with parameter q set at
run one simulation
with q = to obtain Vk
derive gradient
estimate from Vk
iterate SA with the
gradient estimate

11. Example: Experimental Response (Examples 15.4 and 15.5 in ISSO)
Let {Vk} be i.i.d. randomly generated binary (on-off) stimuli with “on” probability l. Assume Q(l,b,Vk) represents negative of specimen response, where b is design parameter. Objective is to design experiment to maximize the response (i.e., minimize Q) by selecting values for l and b.
Gradient estimate: q = [l, b]T;
where and denotes derivative w.r.t. x

12. Experimental Response (continued)
Specific response function:
where b is a structural parameter, but l is both a distributional and structural parameter. Then:

13. Search Path in Experimental Response Problem

14. Sample Path Method
Sample path method based on reusing a fixed set of simulation runs
Method based on minimizing rather than L()
represents sample mean of N simulation runs
If N is large, then minimum of is close to minimum of L() (under conditions)
Optimization problem with is effectively deterministic
Can use standard nonlinear programming
IPA and/or LR/SF methods of gradient estimation still relevant
Generally need to choose a fixed value of  (reference value) to produce the N simulation runs
Choice of reference value has impact on for finite N

15. Accuracy of Sample Path Method
Interested in accuracy of sample path method in seeking true optimal  (minimum of L())
Let represent minimum of surrogate loss
Let denote final solution from nonlinear programming method
Hence, error in estimate is due to two sources:
Error in nonlinear programming solution to finding
Difference in  and
Triangle inequality can be used to provide bound to overall error:
Sometimes numerical values can be assigned to two right-hand terms in triangle inequality