IMPUTING MISSING VALUES FOR HIERARCHICAL POPULATION DATA Overview of Database Research Muhammad Aurangzeb Ahmad Nupur Bhatnagar

Background Along with harmonizing U.S. and international census data ; Minnesota Population Center improves data quality for historical U.S. census samples. Modern samples, the U.S. Census has allocated any missing data. Older census years data had to be converted into machine readable form. Errors and omissions were coded as Missing. If the records with missing data are not representative of the dataset as a whole, throwing out the missing cases can effect results of calculations in an undesirable way.

Problem Definition Given : 1850,1860,1870, Datasets with set of variables. Missing variable of interest is the RELATE variable. This particular variable describes an individual's relationship to the head of household. Constraints: A person is not traceable down multiple years. Position of people within a household holds significance. Different household structure within a given year with minimum number of people being 1 and maximum 17. Importance : The relate variable is of significant importance since it is very vital in trend analysis of the household structure for the Researchers. RelateCode Head/Househol der 01 Missing Available Spouse02 Missing Available Child03 Missing Available Child-in-law04 Missing Available To Find : Predict the relate variable for years where it is missing

Existing Approach : Hot Deck Allocation FIRST PASS: Substitute the missing values on the basis of simple rules that are hard coded explicitly. Ex: Relate code 101: head of household Relate Code 201:spouse of head of the household Relate Code 301:Child of the head of the household. These relationship if missing are generated by simple rules. Result : Almost 75% of the missing values are IMPUTED. SECOND PASS: The remaining 25% cases are assigned using the following process: Persons having a relate code of 101,201 or 301 are removed. The remaining persons are known as “donors”. For each donor a temporary table is created that comprises of the predictor variables of the missing relate code of the record. A temporary table for each qualifying donor record is created.

Approach-contd THIRD PASS The predictor variables of each record with a missing relate label is compared against the temporary table of the donors. If the first characteristic in the temporary table matches the value of the predictor variable of the current record a score is assigned to that donor. If the first characteristic does not match; the donor is ignored. The process goes on interactively comparing the characteristic of donors with the predictor variables of missing record and increasing the score of the donors. Result The donor with maximum score qualifies for substituting the missing relate label of the recipient record. This version of algorithm was implemented in Fortran and is now being converted into Java code.

Results : Traditional Approach Fortran Results: Num of all persons: # of imputed 01: , percent correct: 99.37% # of imputed 02: 80256, percent correct: 98.6% # of imputed 03: , percent correct: 99.01% # of imputed 04: 1822, percent correct: 79.04% # of imputed 05: 3343, percent correct: 87.28% # of imputed 06: 2123, percent correct: 87.58% # of imputed 07: 5335, percent correct: 85.77% # of imputed 08: 2566, percent correct: 89.59% # of imputed 09: 6800, percent correct: 87.6% # of imputed 10: 4449, percent correct: 79.56% # of imputed 11: 0, percent correct: 0% # of imputed 12: 37281, percent correct: 95.31% # of imputed 13: 2289, percent correct: 87.83%

Results : Traditional Approach Fortran Results: Num of all persons: # of imputed 01: , percent correct: 99.37% # of imputed 02: 80256, percent correct: 98.6% # of imputed 03: , percent correct: 99.01% # of imputed 04: 1822, percent correct: 79.04% # of imputed 05: 3343, percent correct: 87.28% # of imputed 06: 2123, percent correct: 87.58% # of imputed 07: 5335, percent correct: 85.77% # of imputed 08: 2566, percent correct: 89.59% # of imputed 09: 6800, percent correct: 87.6% # of imputed 10: 4449, percent correct: 79.56% # of imputed 11: 0, percent correct: 0% # of imputed 12: 37281, percent correct: 95.31% # of imputed 13: 2289, percent correct: 87.83%

Proposed Approach : Data Reformatting H_idP_idNUMPRECRelateSexAge M F M10 H_idRelate_1 Relate_2 Relate_3Sex_1Sex_2Sex_3Age_1Age_2Age_3NUMPREC MFM3 Data Reformatting : In order to capture the entire Household in a single row; the data was reformatted with each row entailing the characteristics of every person in a household.

Proposed Approach :Classification classifier F(relate_charcaterstics) P(x) F(family_characterstics) Relationship Code Ex: 04,05,06 Classification : Task of assigning objects to one of the predefined categories. P(x): Position Vector of the person whose relationship is being imputed in the family F(family_characterstics): Attributes of person belonging to the same household F(relate_charcaterstics): Attributes of person whose relationship is being imputed

In a 4 person Household If I am a child in law and 4 th person in the household compared to if I am again the fourth person but this time in a five person household would the set of my characteristics overlap?

Assigning categories : Segregation Segregation is a process of separating complex structures into smaller more meaningful clusters such that each cluster represents a part of the complex data independently. In the census data there is a variable NUMPREC that reports the number of person records that are included in a household. Sibling_in _law Sibling Child_in_ law Age Child_in law Sibling_in _law Sibling Age Number Of person 0405

Proposed Approach : Classification 4 Person Household Age_04Sex_04Occ_04CLASS 24M F F M76604 Does not captures the dependence of a person with respect to the household. Classification is the task of learning a target function f that maps each attribute set to one of the predefined class labels.

Proposed Approach : Classification 4 Person Household _age04_ position Prev _age Next_ age Head_ age Age_ diff_ head Label Takes into account the complete household and the position vector of the person in the household.

Data analysis Bar graph : Household Size

Results : MataBagging using Decision trees and Rule Based classifier : PrecisionRecallF-MeasureClass PrecisionRecallF-MeasureClass Ada Boost

Results : Exisiting vs Data Mining Recall Measure: Number of correctly classified instances out of the number of relevant instances

Validation Metrics used for comparison: Recall: Recall measures the fraction of positive examples correctly predicted by the classifier. recall(X) = number of correctly classified instances of class X / number of instances in class X 1880,1910 1% sample data set to train and test the classifier. Sql Server 2005 for data pre processing, analysis and data reformatting. Weka for building and testing the classifier.

Summary and Future Work Higher accuracy when predicting relationship codes as compared to the existing hot deck allocation method for year Tested our model on 1910 dataset where the labels were known and it worked with the desired accuracy. Reduced time complexity for the execution. Extend it to larger number of households. Extend it to the detailed version of Relationship.

Questions ?

Background