Stochastic Hydrology Hydrological Frequency Analysis (I) Fundamentals of HFA Prof. Ke-Sheng Cheng Department of Bioenvironmental Systems Engineering National Taiwan University

General interpretation of hydrological frequency analysis Hydrological frequency analysis is the work of determining the magnitude of hydrological variables that corresponds to a given exceedance probability. Frequency analysis can be conducted for many hydrological variables including floods, rainfalls, and droughts. The work can be better perceived by treating the interested variable as a random variable. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Let X represent the hydrological (random) variable under investigation Let X represent the hydrological (random) variable under investigation. A value xc associating to some event is chosen such that if X assumes a value exceeding xc the event is said to occur. Every time when a random experiment (or a trial) is conducted the event may or may not occur. We are interested in the number of Bernoulli trials in which the first success occur. This can be described by the geometric distribution. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Geometric distribution Geometric distribution represents the probability of obtaining the first success in x independent and identical Bernoulli trials. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Recurrence interval vs return period Average number of trials to achieve the first success. Recurrence interval vs return period 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

The general equation of frequency analysis 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Collecting required data. Estimating the mean, standard deviation and coefficient of skewness. Determining appropriate distribution. Calculating xT using the general eq. It is apparent that calculation of involves determining the type of distribution for X and estimation of its mean and standard deviation. The former can be done by GOF tests and the latter is accomplished by parametric point estimation. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Data series for frequency analysis Complete duration series A complete duration series consists of all the observed data. Partial duration series A partial duration series is a series of data which are selected so that their magnitude is greater than a predefined base value. If the base value is selected so that the number of values in the series is equal to the number of years of the record, the series is called an “annual exceedance series”. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Extreme value series Data independency An extreme value series is a data series that includes the largest or smallest values occurring in each of the equally-long time intervals of the record. If the time interval is taken as one year and the largest values are used, then we have an “annual maximum series”. Data independency Why is it important? Annual exceedance series and annual maximum series are different. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Parameter estimation Method of moments Maximum likelihood method Method of L-moments (Gaining more attention in recent years) Depending on the distribution types, parameter estimation may involve estimation of the mean, standard deviation and/or coefficient of skewness. Parameter estimation exemplified by the gamma distribution. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Gamma distribution parameter estimation Gamma distribution is a special case of the Pearson type III distribution (with zero location parameter). Gamma density where , , and  are the mean, standard deviation, and coefficient of skewness of X (or Y), respectively, and  and  are respectively the scale and shape parameters of the gamma distribution. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

MOM estimators 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Maximum likelihood estimator 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Evaluating bias of different estimators of coefficient of skewness 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Evaluating mean square error of different estimators of coefficient of skewness 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Techniques for goodness-of-fit test A good reference for detailed discussion about GOF test is: Goodness-of-fit Techniques. Edited by R.B. D’Agostino and M.A. Stephens, 1986. Probability plotting Chi-square test Kolmogorov-Smirnov Test Moment-ratios diagram method L-moments based GOF tests 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Probability plotting Fundamental concept Probability papers Empirical CDF vs theoretical CDF Example of misuse of probability plotting 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Suppose the true underlying distribution depends on a location parameter  and a scale parameter  (they need not to be the mean and standard deviation, respectively). The CDF of such a distribution can be written as where Z is referred to as the standardized variable and G(z) is the CDF of Z. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

where x represents the observed values of the random variable X. Also let Fn(X) represents the empirical cumulative distribution function (ECDF) of X based on a random sample of size n. A probability plot is a plot of on x where x represents the observed values of the random variable X. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Most of the plotting position methods are empirical Most of the plotting position methods are empirical. If n is the total number of values to be plotted and m is the rank of a value in a list ordered by descending magnitude, the exceedence probability of the mth largest value, xm, is , for large n, shown in the following table. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Misuse of probability plotting Log Pearson Type III ? 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Misuse of probability plotting 48-hr rainfall depth Log Pearson Type III ? 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

How do we fit a probability distribution to a random sample? Fitting a probability distribution to annual maximum series (Non-parametric GOF tests) How do we fit a probability distribution to a random sample? What type of distribution should be adopted? What are the parameter values for the distribution? How good is our fit? 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Chi-square GOF test 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Kolmogorov-Smirnov GOF test The chi-square test compares the empirical histogram against the theoretical histogram. In contrast, the K-S test compares the empirical cumulative distribution function (ECDF) against the theoretical CDF. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

In order to measure the difference between Fn(X) and F(X), ECDF statistics based on the vertical distances between Fn(X) and F(X) have been proposed. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Hypothesis test using Dn 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Values of for the Kolmogorov-Smirnov test 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

IDF curve fitting using the Horner’s equation The intensity-duration-frequency (IDF) relationship of the design storm depths 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

DDF curves 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

IDF curves 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Alternative IDF fitting (Return-period specific) 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Further discussions on frequency analysis Extracting annual maximum series Probabilistic interpretation of the design total depth Joint distribution of duration and total depth Selection of the best-fit distribution 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Annual maximum series Data in an annual maximum series are considered IID and therefore form a random sample. For a given design duration tr, we continuously move a window of size tr along the time axis and select the maximum total values within the window in each year. Determination of the annual maximum rainfall is NOT based on the real storm duration; instead, a design duration which is artificially picked is used for this purpose. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Random sample for estimation of design storm depth The design storm depth of a specified duration with return period T is the value of D(tr) with the probability of exceedance equals  /T. Estimation of the design storm depth requires collecting a random sample of size n, i.e., {x1, x2, …, xn}. A random sample is a collection of independently observed and identically distributed (IID) data. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Probabilistic interpretation of the design storm depth 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

It should also be noted that since the total depth in the depth-duration-frequency relationship only represents the total amount of rainfall of the design duration (not the real storm duration), the probability distributions in the preceding figure do not represent distributions of total depth of real storm events. Or, more specifically, the preceding figure does not represent the bivariate distribution of duration and total depth of real storm events. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

The usage of annual maximum series for rainfall frequency analysis is more of an intelligent and convenient engineering practice and the annual maximum data do not provide much information about the characteristics of the duration and total depth of real storm events. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Joint distribution of the total depth and duration Total rainfall depth of a storm event varies with its storm duration. [A bivariate distribution for (D, tr).] For a given storm duration tr, the total depth D(tr) is considered as a random variable and its magnitudes corresponding to specific exceedance probabilities are estimated. [Conditional distribution] In general, 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Selection of the best-fit distribution Methods of model selection based on loss of information. Akaike information criterion (AIC) Schwarz's Bayesian information criterion (BIC) Hannan-Quinn (HQIC) information criterion Common practices of WRA-Taiwan SE and U SSE and SE Can the p-value be used for selection of the best-fit distribution? 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

Information-criteria-based model selection where is the log-likelihood function for the parameter  associated with the model, n is the sample size, and p is the dimension of the parametric space. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

WRA Practice p: Number of distribution parameters Weibull plotting position formula is used for calculation of cumulative probability. 12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU

12/4/2018 Lab for Remote Sensing Hydrology and Spatial Modeling, Dept. of Bioenvironmental Systems Eng., NTU