1. Graphical optimization
Some problems are cheap to simulate or test.
Even if they are not, we may fit a surrogate that is cheap to evaluate.
Relying on optimization software to find the optimum is foolhardy.
It is better to thoroughly explore manually.
In two dimensions, graphical optimization is a good way to go.
For problems with more variables, it may still be good to identify two important variables and proceed graphically.
In higher dimensions, two dimensional cuts can help understand the relationship between local optima.

2. Example Plot and estimate minimum of In the range
Can do contour plot or mesh plot
Two alternatives for visualizing an objective function or constraints are contour plots and mesh plot. Here these are illustrated for the function
𝑓( 𝑥 1 , 𝑥 2 )=2+ 𝑥 1 − 𝑥 2 +2 𝑥 𝑥 1 𝑥 2 + 𝑥 2 2
The contour plot on the right was generated by the following Matlab sequence:
x=linspace(-2,0,40); y=linspace(0,3,40);
[X,Y]=meshgrid(x,y); Z=2+X-Y+2*X.^2+2*X.*Y+Y.^2;
cs=contour(X,Y,Z);
clabel(cs); xlabel('x_1');ylabel('x_2');
From the contour plot it appears that the minimum is near (-1,1.5), and the value of the function at the minimum is lower than 1.0.
Instead the mesh plot on the left was generated by
cs=surfc(X,Y,Z);
xlabel('x_1'); ylabel('x_2'); zlabel('f(x_1,x_2)');
The contour plot is more convenient for locating the approximate position of the minimum of the function, while the mesh plot gives more readily its shape.

3. Then zoom
Zooming by reducing the range does not change the estimate of the position (-1,1.5), but the value is now below 0.8.
Once we know the approximate position of the minimum we can zoom on a narrower range using the Matlab sequence:
x=linspace(-1.5,-0.5,40);
y=linspace(1,2,40);
[X,Y]=meshgrid(x,y);
Z=2+X-Y+2*X.^2+2*X.*Y+Y.^2
The figures do not change our estimate of the position of the minimum (-1,1.5), but update our estimate of the value of the minimum (below 0.8).

4. In higher dimensions
Often different solutions of optimization problems are obtained from multiple starting points or even from same starting point.
It is tempting to ascribe this to local optima.
Often it is the result of algorithmic or software failure.
Generate line connecting two solutions
In higher dimensions we cannot use graphical optimization, but we can use graphical techniques to check for problems with our solution, or to gain understanding about the relation between alternative solutions. The most common applications is when we get multiple solutions by performing the optimization several times. The solution may change because we use a different starting point, or it may change because the optimization algorithm has a random component, as in genetic algorithms or particle swarm optimization.
Given two solutions, 𝐱 1 and 𝐱 2 , the equation of the line connecting the two points is
𝒙 𝜶 =𝛼 𝐱 1 +(1−𝛼) 𝐱 ≤𝛼<1
Where 𝛼 is now a one dimensional coordinate, and we can plot the function versus 𝛼
f 𝜶 =𝒇[𝒙 𝜶 ]

5. Example Consider function
And consider the two candidate local optima (0,0,0) and (0.94,0.94,0.94).
Line connecting first and second point indicates that they may be local optima.
As an example we consider the function
𝑓=|sin2𝜋𝑥sin2𝜋𝑦sin𝜋𝑧|+𝑥 𝑦 𝑧 2 0≤𝑥,𝑦,𝑧≤1
An optimization routine gave us two different answers (0,0,0), and (0.94,0.94,0.94) in two different runs. We plot the function on the line connecting the two points using the following Matlab sequence:
alpha=linspace(0,1,101);
x1=[0 0 0]; x2=[ ];
x=x2(1)*alpha+x1(1)*(1-alpha); y=x2(2)*alpha+x1(2)*(1-alpha);
z=x2(3)*alpha+x1(3)*(1-alpha);
f=abs(sin(2*pi.*x).*sin(2*pi.*y).*sin(2*pi.*z))+x.*sqrt(y).*z.^2;
plot(alpha,f);
xlabel('alpha');ylabel('f');
The plot of the function between the two points indicates that the two candidate points appear to be local optima along that line. This is a necessary condition for the two points to really be local optima, but it is not sufficient. Here, it is easy to check that the second point is not a local optimum, by reducing z. However, usually when the point is not a local optimum, it would not appear to be so on such a plot.

6. Two-dimensional cut
We can add a third point 𝐱 3 (not necessarily a minimum) to visualize the function in the plane defined by these three points
Now 𝐱 1 corresponds to 𝛼=1, 𝛽=0, 𝐱 2 to 𝛼=0, 𝛽=1, and 𝐱 3 to 𝛼=0, 𝛽=0

7. Example Now consider the function
Using numerical optimization we obtained 𝐱 1 =[0.25,0.25,-0.25], 𝐱 2 =[0.25,0.75,0.25] We add 𝐱 3 =[0,0,0] Contour plot confirms that 𝐱 1 and 𝐱 2 are likely optima, but 𝐱 3 is not
For an example consider the function
𝑓=sin2𝜋𝑥sin2𝜋𝑦sin𝜋𝑧+𝑥 𝑦 𝑧 2 0≤𝑥,𝑦,𝑧≤1
It has multiple optima, and by numerical optimization we obtained two at 𝐱 1 =[0.25,0.25,0.25], 𝐱 2 =[0.25,0.75,0.25]. We cannot plot the function in 3D space, but we can add another points, [0,0,0] and plot the function in the plane defined by these three points. The first candidate optimum corresponds to 𝛼=1, 𝛽=0 and the second one to 𝛼=0, 𝛽=1, with the added point corresponding to 𝛼=0, 𝛽=0.
The plot shown in the figure confirms that the two points are likely to be minima (no guarantee because we do not see how they behave perpendicular to the plane). The plot was produced by the following Matlab sequence:
alpha=linspace(0,1,101); beta=linspace(0,1,101);
[A,B]=meshgrid(alpha,beta);
x1=[ ]; x2=[ ]; x3=[0 0 0];
x=A.*x1(1)+B.*x2(1)+(1-A-B).*x3(1);
y=A.*x1(2)+B.*x2(2)+(1-A-B).*x3(2);
z=A.*x1(3)+B.*x2(3)+(1-A-B).*x3(3);
f=sin(2*pi.*x).*sin(2*pi.*y).*sin(2*pi.*z)+x.*sqrt(y).*z.^2;
cs=contour(A,B,f); Clabel(cs); xlabel('alpha'); ylabel('beta');
The function was optimized using Matlab fminsearch. For example, the first optimum was obtained with
(sin(2*pi*x(1))*sin(2*pi*x(2))*sin(2*pi*x(3)))+x(1)*sqrt(x(2))*x(3)^2;
g(x), [0,0,0])

8. 10 dimensional constrained example
Aerodynamic design of supersonic transport.
Objective function, take-off gross weight is a cheap and simple function of design variables.
Design variables define geometry of wing and fuselage.
Constraints include range, take-off and landing constraints, and maneuvrability constraints.
The constraints are all uni-modal functions, but in combination, they create a complex, non-convex feasible domain.
To complement the previous simple example we also look at an example where multiple optima are not due to waviness of the objective functions but due to the fact that the feasible domain (the domain where all constraints are satisfied) is not convex (that is two points in the domain can be connected by a line that passes outside the domain). The example is taken from: Knill, D.L., Giunta, A.A., Baker, C.A., Grossman, B., Mason, W.H., Haftka, R.T., and Watson, L.T., “Response Surface Models Combining Linear and Euler Aerodynamics for Supersonic Transport Design,” Journal of Aircraft, 36(1), pp , 1999
The objective function is the take-off weight of a supersonic transport, the design variables define the shape of the wing and fuselage, and there is a large number of constraints such as range, take-off clearance constraints, and maneuverability constraints.
All the constraints are approximated by quadratic polynomials, so they are not wavy, and the objective function is also unimodal (not wavy). However the constraints still create different local optima at vertices of the feasible domain.

9. Two local optima and a third point
Simple constraints create non-convex design space.
Can that
happen with
linear
constraints?
Using optimization software with different starting points, two local optima were found, and a third point was added to define a plane. The area defined by the three points can e limited to the triangle, by adding the requirement that 𝛼+𝛽≤1. Here however we let 𝛼 and 𝛽 and their sum to vary up to 1.2 to visualize better the constraints.
In the figure, the color indicates the objective function (weight) the lines indicate the constraint boundaries, empty circles indicate feasible points , which satisfy all the constraints, and filled circles indicate infeasible points (violating at leas t one constraint). In the plane of the figure the feasible domain looks disjoint, with one optimum in one island of feasible design and the other optimum in another island.
It is possible that the two island are connected outside the plane, but they may be connected, for example, only near the place where the two islands are very close. In that case, in order to go from one optimum to another one may have to go up in weight and down again, which a gradient based optimizer may not be able to do.
The figure establishes that the feasible domain is non-convex, because it is possible to connect two feasible points with a straight line that goes outside the feasible domain. This cannot happen with linear constraints, because they are guaranteed to produce a feasible domain.

10. Problems
Find graphically the minimum of the function in Slide 5 in the plane x=1.
Find another minimum of the function in Slide 7 that is not along the line connecting the two given optima and plot the function in the triangle connecting the three optima. Limit the figure only to the triangle, by requiring 𝛼+𝛽≤1.