1. Derivative-free Methods using Linesearch Techniques Stefano Lucidi

2. P. Tseng L. Grippo joint works with (the father linesearch approach) M. Sciandrone G. Liuzzi F. Lampariello V. Piccialli (in order of appearance in this research activity) F. Rinaldi G. Fasano

3. PROBLEM DEFINITION: are not available

4. MOTIVATIONS: In many engineering problems the objective and constraint function values are obtained by  direct measurements  complex simulation programs first order derivatives can be often neither explicitly calculated nor approximated

5. MOTIVATIONS: In fact  the mathematical representations of the objective function and the constraints are not available  the source codes of the programs are not available  the values of the objective function and the constraints can be affected by the presence of noise  the evaluations of the objective function and the constraints can be very expensive

6. MOTIVATIONS: the mathematical representations of the objective function and the constraints are not available the first order derivatives the objective function and the constraints can not be computed analytically

7. MOTIVATIONS: the source codes of the programs are not available the automatic differentiation techniques can not be applied

8. MOTIVATIONS: the evaluations of the objective function and the constraints can be very expensive the finite difference approximations can be too expensive (they need n function evaluations at least)

9. MOTIVATIONS: finite difference approximations can produce very wrong estimates of the first order derivatives the values of the objective function and the constraints can be affected by the presence of noise

10. NUMERICAL EXPERIENCE: we considered 41 box constrained standard test problems we perturbed such problems in the following way: where denotes a Gaussian distributed random number with zero mean and variance

11. NUMERICAL EXPERIENCE: we considered two codes: Number of Failures DF_box = derivative-free method E04UCF = NAG subroutine using finite-differences gradients DF_box E04UCF

12. GLOBALLY CONVERGENT DF METHODS Direct search methods use only function values - pattern search methods where the function is evaluated on specified geometric patterns - line search methods which use one-dimensional minimization along suitable search directions Modelling methods approximate the functions by suitable models which are progressively built and updated

13. UNCONSTRAINED MINIMIZATION PROBLEMS is not available is compact

14. THE ROLE OF THE GRADIENT characterizes accurately the local behaviour of f allows us to determine an "efficient" descent direction to determine a "good" step length along the direction

15. THE ROLE OF THE GRADIENT is the directional derivatives of along provides the rates of change of along the 2n directions characterizes accurately the local behaviour of

16. HOW TO OVERCOME THE LACK OF GRADIENT the local behaviour of along should be indicative of the whole local behaviour of a set of directions can be associated at each

17. ASSUMPTION D Given, the bounded sequences are such that

18. EXAMPLES OF SETS OF DIRECTIONS are linearly independent and bounded

19. EXAMPLES OF SETS OF DIRECTIONS (Lewis,Torczon) are bounded

20. EXAMPLES OF SETS OF DIRECTIONS

21. UNCONSTRAINED MINIMIZATION PROBLEMS Assumption D ensures that, performing finer and finer sampling of along it is possible: - either to realize that the point is a good approximation of a stationary point of - or to find a point where is decreased

22. GLOBAL CONVERGENCE By Assumption D we have:

23. GLOBAL CONVERGENCE By using satisfying Assumption D it is possible: to characterize the global convergence of a sequence of points by means the existence of suitable sequences of failures in decreasing the objective function along the directions

24. GLOBAL CONVERGENCE By Assumption D we have:

25. PROPOSITION Let and be such that: - - satisfy Assumption D then - there exist sequences of points and scalars such that

26. GLOBAL CONVERGENCE the sampling of along all the directions can be distributed along the iterations of the algorithm the Proposition characterizes in “some sense” the requirements on the accettable samplings of along the directions that guarantee the global convergence it is not necessary to perform at each point a sampling of along all the directions

27. GLOBAL CONVERGENCE The use of directions satisfying Condition D and the result of producing sequences of points satisfying the hypothesis of the Proposition are the common elements of all the globally convergent direct search methods The direct search methods can divided in - pattern search methods - line search methods

28. PATTERN SEARCH METHODS Cons: all the points produced must lie in a suitable lattice this implies - additional assumptions on the search directions - restrictions on the choiches of the steplenghts Pros: they require that the new point produces a simple decrease of (in the line search methods the new point must guarantees a “sufficient” decrease of ) (in the line search methods no additional requiriments respect to Assumption D and the assumptions of the Proposition)

29. LINESEARCH TECHNIQUES

31. ALGORITHM DF STEP 1 Compute satisfying Assumption D Minimization of along STEP 2 STEP 3 Compute and set k=k+1

32. STEP 2 The aim of this step is: - to detect the “promising” directions, the direction along which the function decreases “sufficiently” - to compute steplenghts along these directions which guarantee both a “sufficiently” decrease of the function and a “sufficient” moving from the previous point

33. LINESEARCH TECHNIQUE

35. STEP 2 The value of the initial step along the i-th direction derives from the linesearch performed along the i-th direction at the previuos iteration If the set of search directions does not depend on the iteration namely the scalar should be representative of the behaviour of the objective function along the i-th direction

36. STEP 3 set k=k+1 and go to Step 1 Find such thatotherwise set At Step 3, every approximation technique can be used to produce a new better point

37. GLOBAL CONVERGENCE THEOREM Let be the sequence of points produced by DF Algorithm then there exists an accomulation point of and every accumulation points of is a stationary point of the objective function

38. LINEARLY CONSTRAINED MINIMIZATION PROBLEMS is not available is compact (LCP)

39. LINEARLY CONSTRAINED MINIMIZATION PROBLEMS Given a feasible point it is possible to define the set of the indeces of the active constraints the set of the feasible directions

40. LINEARLY CONSTRAINED MINIMIZATION PROBLEMS is a stationary point for Problem (LCP) is a stationary point for Problem (LCP)

41. LINEARLY CONSTRAINED MINIMIZATION PROBLEMS is a stationary point for Problem (LCP)

42. LINEARLY CONSTRAINED MINIMIZATION PROBLEMS an estimate of the set of the indeces of the active constraints an estimate of the set of the feasible directions Given and it is possible to define has good properties which allow us to define globally convergent algorithms

43. ASSUMPTION D2 (an example) Given and the set of directions with satisfies: is uniformly bounded

44. ALGORITHM DFL STEP 1 Compute satisfying Assumption D2 Minimization of along STEP 2 STEP 3 Compute the new point and set k=k+1

45. GLOBAL CONVERGENCE THEOREM Let be the sequence of points produced by DFL Algorithm then there exists an accomulation point of and every accumulation points of is a stationary point for Problem (LCP)

46. BOX CONSTRAINED MINIMIZATION PROBLEMS is not available is compact (BCP) satisfies Assumption D2 the set

47. NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS (NCP) is not available are not available

48. NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS and given a point We define

49. NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS ASSUMPTION A1 The set is compact For every ASSUMPTION A2 there exists a vector such that Assumption A1 boundeness of the iterates Assumption A2 existence and boundeness of the Lagrange multipliers

50. NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS We consider the following continuously differentiable penalty function: where (penalty parameter)

51. NONLINEARLY CONSTRAINED MINIMIZATION PROBLEMS ? ?

52. ALGORITHM DFN STEP 1 Compute satisfying Assumption D2 Minimization of along STEP 2 STEP 3 Compute the new point and set k=k+1

53. new STEP 3 ( )  set k=k+1 and go to Step 1  Find such that otherwise set if set otherwise set andthen 

54. new STEP 3 is reduced whenever a better approximation of a stationary point of the penalty function has been obtained can be viewed as stationarity measure

55. GLOBAL CONVERGENCE THEOREM Let be the sequence of points produced by DFN Algorithm then there exists an accomulation point of which is a stationary point for Problem (LCP)

56. MIXED NONLINEARLY MINIMIZATION PROBLEMS (MNCP) We define n. discr. var. n. cont. var.

57. MIXED NONLINEARLY MINIMIZATION PROBLEMS We define is a stationary point of Problem MNLP if there exists such that:

58. ALGORITHM MDFN STEP 1 Compute Mixed Minimization of along STEP 2 STEP 3 Compute the new point and set k=k+1

59. ALGORITHM MDFN STEP 1 Compute STEP 2 STEP 3 Compute the new point and set k=k+1 a continuous linesearch along If perform a discrete linesearch along If perform

60. Continuous linesearch Continuous linesearch of MDFN = linesearch of DFN it produces the point

61. LINESEARCH TECHNIQUE

64. MIXED NONLINEARLY MINIMIZATION PROBLEMS or every accumulation point of the sequence produced by the algorithm, satisfies: ASSUMPTION A3 Either the nonlinear constraints functions do not depend on the integer variables are such that

65. GLOBAL CONVERGENCE THEOREM Let be the sequence of points produced by MDFN Algorithm then there exists an accomulation point of which is a stationary point for Problem (MNCP)

66. MIXED NONLINEARLY MINIMIZATION PROBLEMS More complex (and expensive) derivative-free algorithms allows us  to determine “better” stationary points  to tackle “more difficult” mixed nonlinear optimization problems

67. MIXED NONLINEARLY MINIMIZATION PROBLEMS  to determine “better” stationary points for Problem (MNCP) satisfies the KKT conditions w.r.t.

68. discrete general variables continuous variables discrete dimensional variables Discrete dimensional variables z : Vector of discrete variables which determine the number of continuous and discrete variables Three different sets of variables:  to tackle “more difficult” mixed nonlinear optimization problems

69. HARD MIXED NONLINEARLY MINIMIZATION PROBLEMS The feasible set of y depends on the dimensional variables z The feasible set of x depends on the discrete variables y and on the dimensional variables z (Hard-MNCP)

70. NONSMOOTH MINIMIZATION PROBLEMS

71. the cone of descent directions can be made arbitrarily narrow

72. NONSMOOTH MINIMIZATION PROBLEMS Possible approaches:  smoothing techniques  “larger” set of search directions

73. NONSMOOTH MINIMIZATION PROBLEMS  smoothing techniques

74. NONSMOOTH MINIMIZATION PROBLEMS

75. ALGORITHM DFN STEP 1 Compute satisfying Assumption D2 Minimization of along STEP 2 STEP 3 Compute the new point and set k=k+1

76. new STEP 3  set k=k+1 and go to Step 1  Find such that otherwise set set 

77. new STEP 3 is reduced whenever a better approximation of a stationary point of the penalty function has been obtained can be viewed as stationarity measure

78. GLOBAL CONVERGENCE THEOREM Let be the sequence of points produced by the Algorithm then there exists an accomulation point of which is a stationary point for the MinMax Problem

79. NONSMOOTH MINIMIZATION PROBLEMS  “larger” set of search directions (NCP) locally Lipschitz-continuous

80. We consider the following nonsmooth penalty function: where (penalty parameter) NONSMOOTH MINIMIZATION PROBLEMS

81. ASSUMPTION A1 The set is compact For every ASSUMPTION A2 there exists a vector such that

82. NONSMOOTH MINIMIZATION PROBLEMS

83. set of search directions which are asintotically dense in the unit sphere It is possible to define algorithms globally convergent towards NONSMOOTH MINIMIZATION PROBLEMS stationary points (in the Clarke sense) by assuming that the algorithms use

84. NONSMOOTH MINIMIZATION PROBLEMS Multiobjective optimization problem (working in progress) locally Lipschitz-continuous

85. NONSMOOTH MINIMIZATION PROBLEMS Bilevel optimization problem (working in progress)

86. Our DF-codes are available at: http://www.dis.uniroma1.it/~lucidi/DFL

87. Thank your for your attention

88. n. magnets=3 n. rings =6 n.magnets =4 half magnet Optimal Design of Magnetic Resonance apparatus

89. Design Variables Positions of the rings along the X-axis X x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 Angular positions of each row of small magnets

90. Design Variables Offsets of the 4 outermost rings w.r.t. the 2 innermost ones X b1b1 b2b2 b3b3 b4b4 Radius of magnets (integer values)

91. Objective Function The objective function measures the non-uniformity of the magnetic field within a specified target region which is Magnetic field as uniform as possible and directed along the Z axis

92. nr=5 nm=3 r=22 f=51 ppm Starting point (commercial devices) nr=7 nm=3 r=27 f=18 ppm Final point

93. Magnetic Resonance Results Behavior of on the ZY plane 51ppm configuration 18ppm configuration