1. AIAA 2002-5531 OBSERVATIONS ON CFD SIMULATION UNCERTAINTIES
Serhat Hosder, Bernard Grossman, William H. Mason, and
Layne T. Watson
Virginia Polytechnic Institute and State University
Blacksburg, VA
Raphael T. Haftka
University of Florida
Gainesville, FL
9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization
4-6 September 2002
Atlanta, GA

2. Introduction
Computational fluid dynamics (CFD) as an aero/hydrodynamic analysis and design tool
CFD being used increasingly in multidisciplinary design and optimization (MDO) problems
CFD results have an associated uncertainty, originating from different sources
Sources and magnitudes of the uncertainty important to assess the accuracy of the results

3. Motivation
Design uncertainties due to computational simulation error
Optimization is an iterative procedure subject to convergence error.
Estimating convergence error may require expensive accurate optimization runs
Many simulation runs are performed in engineering design. (e.g., design of experiments)
Statistical analysis of error

4. Objectives
Find error characteristics of a structural optimization of a high speed civil transport
Estimate error level of the optimization procedure
Identify probabilistic distribution model of the optimization error
Estimate mean and standard deviation of errors without expensive accurate runs
Improve response surface approximation against erroneous simulation runs via robust regression

5. High Speed Civil Transport (HSCT)
250 passenger aircraft, 5500 nm range, cruise at Mach 2.4
Take off gross weight (WTOGW) is minimized
Up to 29 configuration design variables including wing, nacelle and fuselage geometry, fuel weight, and flight altitude
For this study, a simplified 5DV version is used

6. Structural optimization of HSCT
For each HSCT configuration, wing structural weight (Ws) is minimized by structural optimization (GENESIS) with 40 design variables
Structural optimization is performed a priori to build a response surface approximation of Ws

7. Effects of initial design point on optimization error
For Case 2, the initial design point was perturbed from that of Case 1, by factors between 0.1 ~ 1.9
In average, Case 2 has the same level of error as Case 1
The errors were calculated with respect to higher fidelity runs with tightened convergence criteria

8. Estimating error of incomplete optimization
To get optimization error we need to know true optimum, which is rarely known for practical engineering optimization
Optimization error = OBJ* - OBJ*true
To estimate OBJ*true
Find convergence setting to achieve very accurate optimization
This approach can be expensive

9. The Weibull distribution model
Widely used in reliability models (e.g., lifetime of devices)
Characterized by a shape parameter  and a scale parameter 
PDF function according to 

10. Estimation of distribution parameters
Maximum likelihood estimation (MLE) to estimate distribution parameters of the assumed distribution.
Find  to maximize the likelihood function.
2 goodness of fit test
Comparison of histograms between data and fit
p-value indicates quality of the fit

11. Difference fit to estimate statistics of optimization errors
Errors of two optimization runs of different initial design point
s = W1 – Wt
t = W2 – Wt , (Wt is unknown true optimum.)
The difference of s and t is equal to W1 – W2
x = s – t = (W1 – Wt) – (W2 – Wt ) = W1 – W2
We can fit distribution to x instead of s or t.
g(s; 1)
PDF
s, t
h(t; 2)
x=s-t
s and t are independent
Joint distribution of g(s) and h(t)
MLE fit for optimization difference x using f(x)
No need to estimate Wt

12. Comparison between error fit and difference fit
Difference fit is applied to the differences between Cases 1 and 2, assuming the Weibull model for both cases
Even when only inaccurate optimizations are available, difference fit can give reasonable estimates of uncertainty of the optimization

13. Estimated distribution parameters
Another set of optimization with a difference initial design point was enough, which is straightforward and no more expensive
The difference fit gave error statistics as accurate as the error fits involving expensive higher fidelity runs

14. Outliers and robust regression
Robust regression techniques
Alternative to the least squares fit that may be greatly affected by a few very bad data (outliers)
Iteratively reweighted least squares (IRLS)
Inaccurate response surface approximation is another error source in design process
Improve response surface approximation by repairing outliers

15. Iteratively Reweighted Least Squares
Weighted least squares: Initially all points have same weight of 1.0.
At each iteration, the weight is reduced for points that are far from the response surface.
This gradually moves the response surface away from outliers.
IRLS approximation leaves out detected outliers
1 dimensional example

16. Nonsymmetric weighting function
Regular weighting functions
Symmetrical
Huber’s function gives non-zero weighting to big outliers
Biweight rejects big outliers completely
Nonsymmetric weighting function
make use of one-sidedness of error by penalizing points of positive residual more severely
Weighting functions of IRLS
Aggressive search by reducing tuning constant B

17. Improvements of RS approximation by outlier repair
Comparison of RS before and after repair

18. Drag polar results from 1st AIAA Drag Prediction Workshop (Hemsch, 2001)

19. Objective of the Paper
Finding the magnitude of CFD simulation uncertainties that a well informed user may encounter and analyzing their sources
We study 2-D, turbulent, transonic flow in a converging-diverging channel
complex fluid dynamics problem
affordable for making multiple runs
known as “Sajben Transonic Diffuser” in CFD validation studies

20. Transonic Diffuser Problem
Weak shock case (Pe/P0i=0.82)
experiment
P/P0i
CFD
Strong shock case (Pe/P0i=0.72)
P/P0i
streamlines
Separation bubble
Contour variable: velocity magnitude

21. Uncertainty Sources (following Oberkampf and Blottner)
Physical Modeling Uncertainty
PDEs describing the flow
Euler, Thin-Layer N-S, Full N-S, etc.
boundary conditions and initial conditions
geometry representation
auxiliary physical models
turbulence models, thermodynamic models, etc.
Discretization Error
Iterative Convergence Error
Programming Errors
We show that uncertainties from different sources interact

22. Computational Modeling
General Aerodynamic Simulation Program (GASP)
A commercial, Reynolds-averaged, 3-D, finite volume Navier-Stokes (N-S) code
Has different solution and modeling options. An informed CFD user still “uncertain” about which one to choose
For inviscid fluxes (commonly used options in CFD)
Upwind-biased 3rd order accurate Roe-Flux scheme
Flux-limiters: Min-Mod and Van Albada
Turbulence models (typical for turbulent flows)
Spalart-Allmaras (Sp-Al)
k- (Wilcox, 1998 version) with Sarkar’s compressibility correction

23. Grids Used in the Computations
y/ht
A single solution on grid 5 requires approximately 1170 hours of total node CPU time on a SGI Origin2000 with six processors (10000 cycles)
Grid level
Mesh Size (number of cells) 1
40 x 25 2
80 x 50 3
160 x 100 4
320 x 200 5
640 x 400
Grid 2 is the typical grid level used in CFD applications

24. Nozzle efficiency
Nozzle efficiency (neff ), a global indicator of CFD results:
H0i : Total enthalpy at the inlet
He : Enthalpy at the exit
Hes : Exit enthalpy at the state that would be reached by isentropic expansion to the actual pressure at the exit

25. Uncertainty in Nozzle Efficiency

26. Discretization Error by Richardson’s Extrapolation
error coefficient
order of the method
grid level
a measure of grid spacing
Turbulence model
Pe/P0i
estimate of p (observed order of accuracy)
estimate of (neff)exact
Grid level
Discretization error (%)
Sp-Al
0.72
(strong shock)
1.322 1
14.298 2
6.790 3
2.716 4
1.086
0.82
(weak shock)
1.578
8.005
3.539
1.185
0.397
k- 
1.656
4.432
1.452
0.461
0.146

27. Error in Geometry Representation

28. Error in Geometry Representation
Upstream of the shock, discrepancy between the CFD results of original geometry and the experiment is due to the error in geometry representation.
Downstream of the shock, wall pressure distributions are the same regardless of the geometry used.

29. Downstream Boundary Condition

30. Downstream Boundary Condition
Extending the geometry or changing the exit pressure ratio affect:
location and strength of the shock
size of the separation bubble

31. Uncertainty Comparison in Nozzle Efficiency
Strong Shock
Weak Shock
Maximum value of
the total variation in nozzle efficiency
10% 4%
the difference between grid level 2 and grid level 4 6%
(Sp-Al)
3.5%
the relative uncertainty due to the selection of turbulence model 9%
(grid 4) 2%
(grid 2)
the uncertainty due to the error in geometry representation
0.5%
(grid 3, k-)
1.4%
the uncertainty due to the change in exit boundary location
0.8%
(grid 3, Sp-Al)
1.1%
(grid 2, Sp-Al)

32. Conclusions
Based on the results obtained from this study,
For attached flows without shocks (or with weak shocks), informed users may obtain reasonably accurate results
They may get large errors for the cases with strong shocks and substantial separation
Grid convergence is not achieved with grid levels that have moderate mesh sizes (especially for separated flows)
Difficult to isolate physical modeling uncertainties from numerical errors

33. Conclusions
Uncertainties from different sources interact, especially in the simulation of flows with separation
The magnitudes of numerical errors are influenced by the physical models (turbulence models) used
Discretization error and turbulence models are dominant sources of uncertainty in nozzle efficiency and they are larger for the strong shock case
We should asses the contribution of CFD uncertainties to the MDO problems that include the simulation of complex flows