1. Chapter 17 Read Raw Data in Fixed Format using Formatted Input Objectives Distinguish between standard and nonstandard numeric data Read standard fixed-field data Read nonstandard fixed-field data

2. Review of Column Input General Syntax of Column Input: INPUT var start_col – end_col ………. ; – Var is the variable name – $ is for character variable. – Start_col, end _clo specify the starting and ending col # for reading the variable. Ex. INPUT L_Name $ 1 – 15 F_name $ 17-25 age 26-28 Choles 30-35;

3. Important features and usages of Column Input It can read character variables of the data values have embedded blanks. Missing data values will be read as missing from the defined columns (Blank for character and ‘.’ for numeric). Columns can be re-read. ex. INPUT supplier $ 5-20 ItemNum 15-20 amount 22-30; Columns can be read backwards or forwards. Ex. INPUT F_name $ 20-30 L_Name $ 1-15 age 16-18;

4. Raw data that can not be read by Column Input When data values are not standard numeric data: Ex. Data values having $ sign, having comma, having %, etc. Data values are not organized in a fixed columns for each variable. Numeric data values having decimal places, the decimal is not recorded in the data. Date, Time, Datetime data that are not recorded in numeric values, instead, recorded as commonly used date, such as 11/14/2010. Such as date requires a special format to read it. If it is read using Column Input, it much be read as a character data. Data values are not recorded in fixed format, for example, data values are recorded by using delimiters, such as blank /, ; tab, and so on.

5. A review of Standard Vs. Nonstandard Numeric Data Standard numeric data can contain only Numbers Decimal places Numbers in scientific or E-notation (ex, 4.2E3) Plus or minus signs Nonstandard numeric data includes Values contain special characters, such as %, $, comma (,), etc. Date and time values Data in fractions, integer binary, real binary, hexadecimal forms, etc.

6. Determine if each of the following numeric data standard or nonstandard data 345.12 Standard $345.12 Nonstandard 3,456.12Nonstandard 20DEC2010Nonstandard date 12/20/2010Nonstandard date

7. A review of Fixed Format Vs. Free format Fixed format means a variable occupies in a fixed range of columns from observation to observation. Free format means the data values are not in a fixed range of columns. Ex:Fixed formatFree format12345678901234567890-------------------- HIGH 340 12.5 FHIGH 340 12.5 F LOW 5630 7.5 F MEDIAN 674 26.73 M

8. A Review of basic statements for reading External Raw data in a Data Step General form for the complete DATA step without FILEMANE statement: DATA SAS_data_set_name; INFILE ‘input-raw-data-file’; INPUT variable $ start - end...; RUN; General form for the complete DATA step with Filename statement: DATA SAS_data_set_name; FILENAME Fileref ‘input-raw-data-file’; INFILE Fielref ; INPUT variable $ start - end...; RUN;

9. Example: Review of Reading External Raw Data Using Column Input Read External Data salesdata.dat with FILENAME statement FILENAME sal_dat ‘C:\math707\RawData\RawData_dat\salesdata.dat’ ; DATA saleslib.sales_sasdata; INFILE sal_dat; INPUT last_name $ 1-7 sale_date $ 9-11 residential 13 -21 commercial 23 – 31; Read External Data salesdata.dat without FILENAME statement DATA saleslib.Sales_sasdata; INFILE ‘C:\math707\RawData\RawData_dat\salesdata.dat’ ; INPUT last_name $ 1-7 sale_date $ 9-11 residential 13 -21 commercial 23 – 31;

10. Reading External Raw Data using Formatted Input General Syntax: INPUT variable Informat.; Pointer-control: pointers to control the position of the column. Variable: variable name to be created. Informat: the format to input the variable. NOTE: Two pointer-controls to control the column position are: @n : moves the pointer to the specified column. This is the absolute column of the data record. +n : move the pointer forward n columns beginning from the current position.

11. Informat in the INPUT statement Informat is the SAS format used in the Input statement to READ data values. It is used in the INPUT statement, and it is called INformat. The SAS format we discussed in Chapter four DISPLAYING data values can be used as Informat in the INPUT statement as fixed formatted input.

12. Recall: SAS Format A format is an instruction that SAS uses to write data values. SAS formats have the following form: 12 format. Format name Total width (including decimal places and special characters) Number of decimal places Required delimiter Indicates a character format

13. SAS INFormats 13

14. SAS INFormats for Date, Time 14 Recall that a SAS date is stored as the number of days between 01JAN1960 and the specified date. If the date or time values are created using ex, 10/16/2001, 16OCT2001, in order to read the date properly from the external data set, we need to use the Informat: Date in External data set InformatData value read 10/16/2001mmddyy10.101601 16OCT2001Date9.101601

15. Some commonly used Informat PERCENTw.dDATE9.NENGOw. $BINARYw.DATETIMEw.PDw.d $VARYINGw.HEXw.PERCENTw. $w.JULIANw.TIMEw. COMMAw.dMMDDYYw.W.d

16. The Informat COMMAw.d COMMAw.d informat reads nonstandard numeric data and removes the embedded Blanks, commas, dashes, dollar signs, percent signs, right parenthesis, left parenthesis, which are converted to negative sign. Actual Data valueCOMMAw.dData value read 12,345.67COMMA9.212345.67 $12,345.67COMMA10.212345.67 COMMA9.212345.67 12-345.67COMMA9.212345.67 (12,345.67)COMMA11.2-12345.67

17. Exercise Practice Informat in Input statement: The data set aug99n.dat is posted on the class website. Three observations of the data set are shown below : 100 02AUG1999 R 6 180 90% 103 03AUG1999 C 20 80 100% 102 04AUG1999 T 4 800 85% Write a SAS program to read this data, pay special attention to the use of Informat to read non-standard numeric values. Print the data set using proper display formats for non-standard variables. Field NameStart ColumnEnd ColumnMaximum WidthData Type ID133numeric Date5139character Item15 1character Quantity17193numeric Price21244numeric Percentsale26297numeric

18. Answer: There are many ways to accomplish the same goal. Here is an example data orders; infile 'C:\math707\RawData\RawData_dat\aug99n.dat'; input ID 1-3 @5 date date9. item $ 15 quantity 17- 19 totalcost 21-24 @26 percentsale percent4. ; proc print; format date MMDDYY8. percentsale percent6. ; run;

19. Salesdata.dat 1234567890123456789012345678901234567890 ---------------------------------------- SMITH 10JAN2010 140097.98 440356.70 DAVIS 15JAN2010 385873.00 178234.00 JOHNSON 20JAN2010 98654.32 339485.00 SMITH 01FEB2010 225983.09 12250.00 DAVIS 12FEB2010 88456.23 55564.00 JOHNSON 22FEB2010 135837.34 32423.00 SMITH 10MAR2010.000000 584593.00 DAVIS 18MAR2010 89342.00 173222.00 JOHNSON 26MAR2010 398573.90 193567.00

20. Read Salesdata.dat data using Formatted Input Data work.sale; INFILE ‘C:\math707\RawData\RawData_dat \Salesdata.dat’; input last_name $7. @9 month date9. @19 residential 9.2 +1 commercial 9.2; run; Proc print; Run; NOTE: INFILE defines the location of the data set. $w. is the format for character variable. @n: moves the pointer to the column n. +n: move forward n columns from the current position. The pointer starts at column 1. After reading a variable, the pointer move the next column as the current position. Ex: After reading last_name with 7 columns, the pointer moves to column 8 as the current position. After reading residential (starting at 19, reading 9 columns), that is, residential is from 19 to 27. The pointer moves to column 28 as the current column. Hence, +1 asks the pointer move one column forward from col 28 to col 29, then, read 9 columns for commercial.

21. Example: Read the following data using Formatted Input The following is the scores of quizzes, test1, test2 and final of a class. Name Q1 Q2 Q3 Q4 Q5 T1 T2 Final ----+----1----+----2----+----3----+----4 CSA 17 18 15 19 18 85 92 145 DB. 16 14 18 16 72 76 120 QC 20 18 19 16 20 92 95 143 DC 18 15. 15 20 82 79 125 E 20 18 15 15 18 80 82 135 F 16 16 15 15 16 72 75 116 GC 20 16 17 16 17. 87 139 HD 18 15 15. 19 85 79 115 IM 17 18 19 20 20 95 92 145 WB 13 16 14 15 16 72 66 110 Write a SAS program to read the data by having the data included in the SAS program.

22. /*Program Statements */ DATA scores; /*Column Input */ INPUTName $ 1-5 Q1 6-7 Q2 10-11 Q3 14-15 Q4 18-19 Q5 22-23TEST1 25-27 TEST2 29-31 Final 33-36; /*Formatted Input */ INPUT NAME $5. Q1 2. @10 Q2 2. @14 Q3 2. @18 Q4 2. @22 Q5 2. @25 TEST1 3. @29 TEST2 3. @33 FINAL 4.; 1234567890123456789012345678901234567890 CSA 17 18 15 19 18 85 92 145 DB. 16 14 18 16 72 76 120 QC 20 18 19 16 20 92 95 143 DC 18 15. 15 20 82 79 125 E 20 18 15 15 18 80 82 135 F 16 16 15 15 16 72 75 116 GC 20 16 17 16 17. 87 139 HD 18 15 15. 19 85 79 115 IM 17 18 19 20 20 95 92 145 WB 13 16 14 15 16 72 66 110 ; RUN;

23. Different formatted inputs to read the same data /*Column Input */ INPUTName $ 1-5 Q1 6-7 Q2 10-11 Q3 14-15 Q4 18-19 Q5 22-23 TEST1 25-27 TEST2 29-31 Final 33-36; /*Formatted Input */ INPUT NAME $5. Q1 2. @10 Q2 2. @14 Q3 2. @18 Q4 2. @22 Q5 2. @25 TEST1 3. @29 TEST2 3. @33 FINAL 4.; INPUT NAME $5. Q1 2. +2 Q2 2. +2 Q3 2. +2 Q4 2. @22 Q5 2. @25 TEST1 3. +1 TEST2 3. @33 FINAL 4.; INPUT NAME $5. (Q1-Q5 TEST1 TEST2)(2. +2) @33 FINAL 4. ;

24. Exercise Open the program c5_colInp And change the Column INPUT statement using Formatted Input.

25. Fixed Record Length Vs. Variable Record Length In reading an external data set, the record length is the size of each record. Usually a record consists of the variables of an observation. NOTE: It is possible one record can consists of multiple observations. This will be discussed later The size of each record is usually ‘FIXED’, that is the same record size for every record. However, it may not be the case in data recording. That is, the record size may differ. When the record lengths differ, Formatted input may not read the data values correctly due to the fact that Formatted input will look for the # of columns specified for each variable. When the record lengths vary, the pointer may continue to the next record in order to read the specified # of columns for last variable (usually) in the INPUT statement. An error will occur when this situation happens.

26. Formatted Input when Reading Records with Variable Record Lengths Using the PAD option in the INFILE Statement One way to fix the problem is to add the blank spaces to the existing records that are short of the record length to change the record length to be ‘FIXED’. The other way is to inform SAS to ‘PAD’ the blanks to those records which are too short. Suppose the record length for the Salesdata.dat is not fixed. Example: Data work.sale; INFILE ‘C:\math707\RawData\RawData_dat \ Salesdata.dat’ pad; input last_name $7. @9 sale_date date9. @19 residential 9.2 +1 commercial 9.2; run;

27. Formatted PUT to created External Data Set Similar to formatted Input to read external raw data set, one can create external data set using formatted PUT statement. FILENAME fileref ‘file-location’; FILE fileref; PUT var format ……… ; RUN;

28. Example: Create External Data Set using Formatted PUT To create an external data for the salesdata that consists of only MARCH. Data work.sale; INFILE ‘C:\math707\RawData\RawData_dat\ Salesdata.dat’; input last_name $7. @9 sale_date date9. @19 residential 9.2 +1 commercial 9.2; Run; Data marchsale; set work.sale; FILE ‘C:\math707\RawData\RawData_dat\ Sales_March.dat’; IF MONTH(Sale_date) = 3; PUT l_name $7. @10 sale_date MMDDYY10. @20 residential 10.2 @35 commercial 10.2; run;

29. Exercise The following is a finance data. Variables are SSN, Name, Salary, Nyear, Birthday 029-46-9261 Rudelich 55,000.00 2 4595 074-53-9892 Vincent 65,063.28 5 228-88-9649 Benito 78,386.54 9 6780 442-21-8075 Sirignano $5,890.06 1 9295 446-93-2122 Harbinger 73,925.37 7 7896 776-84-5391 Phillipon $49,750.92 12 6596 929-75-0218 Gunter 57,445.32 4 6097 Write a SAS program to read this data using formatted format. Practice using PAD option in the Infile statement and make sure you see and understand the difference between with PAD and without PAD

30. An answer /* program (b) to read variable length records - This program has error. Carefully check the errors */ data financeb; infile 'C:\math707\RawData\RawData_dat\finance3_recordlength.dat' ; input SSN $ 1-11 Name $ 13-22 @24 salary comma10. +2 Nyear 3. +2 birthdate 5.; proc print; format birthdate date9.; title 'Errors in reading variable-length records'; run; /* Program c: use PAD option in the INFLIE statement */ data financec; infile 'C:\math707\RawData\RawData_dat\finance3_recordlength.dat' pad; input SSN $ 1-11 Name $ 13-22 @24 salary comma10. +2 Nyear 3. +2 birthdate 5.; proc print; format birthdate date9.; title 'Use PAD option to read variable-length records'; run;