Lwando Kondlo Supervisor: Prof. Chris Koen University of the Western Cape 12/3/2008 SKA SA Postgraduate Bursary Conference Estimation of the parameters of a truncated Pareto distribution when the sample is contaminated by measurement errors

Introduction 12/3/2008 SKA SA Postgraduate Bursary Conference  The Pareto distribution is a simple model for positive data.  The truncated version has a wide range of application in several field in data analysis [1].  In astronomy and many physical and social sciences, the parameters of this truncated Pareto are estimated to draw inference about the processes underlying the phenomena: 1. To scale up the local observations to global patterns 2. To test theoretical models

Introduction(Cont’d) 12/3/2008 SKA SA Postgraduate Bursary Conference Therefore, it is essential that these parameters be estimated accurately. Unfortunately, the binning-based method traditionally used in astronomy and other fields perform quite poorly [2]. In this presentation, we discuss a more sophisticated method for fitting these parameters based on MLE.

Measurement error model 12/3/2008 SKA SA Postgraduate Bursary Conference The model for a variable measured with error is Where the measurement error is assumed to be independent of X. X is true value, but X is not directly observed, Y is observed instead.

Measurement error model (Cont’d) 12/3/2008 SKA SA Postgraduate Bursary Conference  The PDF or parameters of X are of interest when the objective is to estimate characteristics of the population excluding within-variability.  The estimation of PDF or parameters in the presence of measurement error is also known as deconvolution.

Objectives 12/3/2008 SKA SA Postgraduate Bursary Conference Develop a numerical methodology for deconvolution when the distribution is of Pareto form. Apply the methodology to the real data. Data of cloud masses (GMC) in various galaxies.

Convolution 12/3/2008 SKA SA Postgraduate Bursary Conference If X has the PDF g(.) and has the PDF h(.) Then, the sum Y has the PDF given by the convolution integral The forms of the densities g(.) and h(.) are assumed known, in this case.

Convolution (Cont’d) 12/3/2008 SKA SA Postgraduate Bursary Conference g(.) has a power-law form (i.e., truncated Pareto distribution)

Convolution (Cont’d) 12/3/2008 SKA SA Postgraduate Bursary Conference h(.) has a normal distribution with zero mean

Error-contaminated distribution 12/3/2008 SKA SA Postgraduate Bursary Conference

Non-contaminated distribution 12/3/2008 SKA SA Postgraduate Bursary Conference

Convolution 12/3/2008 SKA SA Postgraduate Bursary Conference The density f(.) is given by This is called an error-contaminated truncated Pareto density function. The only unknowns are the specific parameter values

Parameter Estimation 12/3/2008 SKA SA Postgraduate Bursary Conference MLE – preferred method for estimating parameter values. MLE determines the parameter values that maximise the likelihood of the observed data given the model Specifically, MLE finds the value of L, U, a and that maximise the product of the probabilities of each observed values Y

Parameter Estimation (Cont’d) 12/3/2008 SKA SA Postgraduate Bursary Conference The log-likelihood is The best values of the parameters are obtained by maximising. Optimisation – iterative procedure.

Simulation 12/3/2008 SKA SA Postgraduate Bursary Conference To validate our method, we generated data-sets of sizes n = 200, 400, 600 random points drawn from a truncated Pareto distribution with added normal errors to simulate the effects of measurement error.

Simulation Results 12/3/2008 SKA SA Postgraduate Bursary Conference True values Estimated values LUa nLUa Bias error2.87%1.77%24.08%25.00% Bias error0.35%0.53%21.70%28.45% Bias error0.94%0.88%22.54%5.20%

Application 12/3/2008 SKA SA Postgraduate Bursary Conference We apply the method to the statistical analysis of cloud masses from a survey in various galaxies obtained by radio telescope somewhere. It is known that cloud masses follows a power-law (Pareto) distribution. But the methods used to measure cloud masses are subject to measurement error [5] Instrumental error – chemical evolution, temperature, etc.

Second survey of Molecular clouds, by Fukui et. al., /3/2008 SKA SA Postgraduate Bursary Conference The figure shows the M33 region

Frequency Distribution 12/3/2008 SKA SA Postgraduate Bursary Conference

Results 12/3/2008 SKA SA Postgraduate Bursary Conference L Ua MLE Std errors info. Matrix Std errors Jackknife The lowest mass L we measure for a GMC in M33 is The highest mass U for a GMC in M33 is The masses with power law exponent for a GMC in M33 is The std errors are used to provide an indication of the size of the uncertainty, but its formal use is to provide confidence intervals

Assessing quality of fit 12/3/2008 SKA SA Postgraduate Bursary Conference The cumulative distribution function (CDF) of Y is given as: Then

Assessing quality of fit Graphical AssessmentsGoodness of Fit test 12/3/2008 SKA SA Postgraduate Bursary Conference Probability-Probability (P- P) Plots Kolmogorov-Smirnov (K- S) test

P-P Plots P-P plot compares the theoretical and empirical CDF in terms of their probabilities 12/3/2008 SKA SA Postgraduate Bursary Conference The coordinates of a point on a P- P plot are

K-S GoF test 12/3/2008 SKA SA Postgraduate Bursary Conference The K-S test statistic is based on the maximum distance between the theoretical CDF and the empirical CDF. The K- S statistic

K-S GoF Test 12/3/2008 SKA SA Postgraduate Bursary Conference

Conclusion 12/3/2008 SKA SA Postgraduate Bursary Conference The deconvolution method recovers the properties of the truncated Pareto distribution with very little/no bias. Produces reasonable error estimates from inverse Fisher information matrix and the Jackknife. The probability plot is approximately linear, indicating that the sample comes from the postulated distribution.

Future objectives 12/3/2008 SKA SA Postgraduate Bursary Conference Other distributions Truncated data Comparison with results for other methods.

Acknowledgments 12/3/2008 SKA SA Postgraduate Bursary Conference Acknowledge all people who contributed to the work presented and the funding sources SKA.

References 12/3/2008 SKA SA Postgraduate Bursary Conference [1] Zeninetti, L. and Ferraro, M.(2008). On the truncated Pareto distribution with applications. Central European Journal of Physics. Vol. 6 (1). 1-6 [2] White, E. P., et. al. (2008). “On estimating the exponent of power-law frequency distributions. Ecology. Vol [3] Engargiola, G., et. al., "Giant molecular clouds in M33. I. BIMA all-disk survey, [4] Cordy, C. B. and Thomas, D. ". ApJS 149 (1997). “ Deconvolution of a distribution function”. American Statistical Association. Vol [5] Rosolowsky, E (2005). “The Mass Spectra of Giant Molecular Clouds in the Local Group ” The Astronomical Society of the Pacific, Vol. 117,

12/3/2008 SKA SA Postgraduate Bursary Conference Thank you