Timothy Forsyth Ashok Viswanathan Debbie McCullough Donn Garvert

 Some operations are more convenient when there is one observation per subject  Example: Regression analysis  Some operations are more convenient when there are several observations per subject  Example: Plot individual observations as a time series Participant # Day Since Injury Ave. Positive Mood Ave. Negative Mood

First method: Using arrays – Can use arrays to transpose the data set – Gives user more control Original data:

Transposing the entire data set

Transposing the entire data set back to original form

Key commands: RETAIN: since these variables are not in the dataset rainfall1 we must retain these values as SAS will set to missing if we do not CALL MISSING: this will set any number of numeric values or character values to missing all at once In our example, if there was a missing data point somewhere in between month 1 and month 5 for each subject the CALL MISSING command will set these values to missing all at once. We use the OF command so that SAS looks at all 5 values.

Without adding options to PROC TRANSPOSE we will get a confusing output, but the data will be correct:

When we add options to PROC TRANSPOSE we can get a cleaner looking data set Renamed _name_ to “month” Renamed col1 to “rainfall” Dropped subjects with missing data

Transposing the rainfall data back to original form

 PREFIX= option: this was not used in the example, but can be used as an option in this conversion  Useful when your id variable is a number  For example if month was listed as 1, 2, 3, 4, 5 we could use PREFIX=Month ▪This will result in Month 1, Month 2, …, Month 5

Data Sets Often have two or more observations per subject (or other groupings)  Patients who have repeated visits to the Doctor’s office or clinic  Sales at a store on a given day.  Inches of rainfall over many months in a given city (Current Example)  This is known as Longitudinal Data SAS processes data one observation at a time so special techniques are needed to perform calculations across observations I will cover : 1)Identifying the first and last observation in a group. 2)A few different ways to count the number of occurrences of a subject (or other grouping.). In our example we will count, for a given city, the number of months that had their average rainfall recorded.

Listing of SAS Data Set Rainfall Ave Average City Rainfall Temp Sunlight Alameda Fremont Fremont Fremont Fremont Fremont Hayward Hayward Hayward Hayward Hayward Oakland Oakland Sunnyvale Sunnyvale Sunnyvale Sunnyvale Three Step process: Step 1) Sort the data first by the grouping variable (City) and then by the counting variable(Rainfall). Step 2) Create First and Last Variables. Step 3) Counting the months of recorded rainfall using either a data step or proc sql.

city Rainfall Alameda 5.8 Fremont 5.2 Fremont 4.5 Fremont 3.3 Fremont 1.8 Hayward 1.4 Hayward 3.6 Hayward 2.0 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 5.7 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 proc sort data=rainfall5; by city rainfall; run; proc print data=rainfall5; var city rainfall; run; city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 Dataset Rainfall has been sorted first by city and second by rainfall

city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 data rainfall_last rainfall_first; set rainfall5; by city; if last.city then output rainfall_last; else if first.city then output rainfall_first; run; proc print data=rainfall_first; run; Listing of First_City city Rainfall Fremont 1.8 Hayward 1.4 Oakland 3.1 Sunnyvale 3.4 Using the set dataset; by var(city); creates two temporary SAS variables, first.var and last.var. These two are logical variables; They equal 1 if true and 0 if false. In our case we have generated variables first.city and last.city.

city=Alameda Rainfall=5.8 FIRST.city=1 LAST.city=1 city=Fremont Rainfall=1.8 FIRST.city=1 LAST.city=0 city=Fremont Rainfall=2.2 FIRST.city=0 LAST.city=0 city=Fremont Rainfall=3.3 FIRST.city=0 LAST.city=0 city=Fremont Rainfall=4.5 FIRST.city=0 LAST.city=0 city=Fremont Rainfall=5.2 FIRST.city=0 LAST.city=1 city=Hayward Rainfall=1.4 FIRST.city=1 LAST.city=0 city=Hayward Rainfall=2 FIRST.city=0 LAST.city=0 city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 Observation 1 for Alameda is both the first and the last variable so first.city =1(true )and Last.city=1(true). Obs 1 for Fremont first.city = 1 last.city =0, etc., etc. data rainfall_last rainfall_first; set rainfall5; by city; put city= rainfall= first.city= last.city=; if last.city then output rainfall_last; else if first.city then output rainfall_first; run; Contents from the log

Listing of Counts # of Months City Rainfall Recorded Alameda Fremont Hayward Oakland Sunnyvale city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 data months_of_rec_rainfall; set rainfall5; by city; if first.city then N_months_rec = 0; N_months_rec +1; if last.city then output; run; title 'Listing of Counts'; proc print data=months_of_rec_rainfall label noobs; label N_months_rec = '# of Months Recorded'; var City Rainfall N_months_rec;; run; Prediction(actual) ; Alameda 1(1), Fremont 5(5), Hayward 5(5) Oakland 2(2), Sunnyvale 4(4).

data months_of_rec_rainfall; set rainfall5; by city; if first.city then N_months_rec= 0; 1) N_months_rec +1; 2) if last.city then output; 3) run; title 'Listing of Counts'; proc print data=months_of_rec_rainfall label noobs; label N_months_rec = '# of Months Recorded'; var City Rainfall N_months_rec;; run; 1) Initialize the counter at zero 2) Sum Statement 3) Conditional Statement

city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 months_rec_ city rain Alameda 1 Fremont 5 Hayward 5 Oakland 2 Sunnyvale 4 proc sql; create table Months_Rec_Rain as select city, count(city) as months_rec_rain from rainfall5 group by city; quit; The proc sql gives the same result as the data step.

It was shown, via a three step process, how to create a variable to count the number of occurrences of a grouping variable. The example shown dealt with the number of months of recorded rainfall in a given city. These techniques have utility in the medical or clinical setting.

SAS CODE /*eliminate missing values*/ /*create longitudinal data*/ data rainfall1; set rainfall; array rainfall_array{5} month_1-month_5; array temp_array{5} temp_1-temp_5; array hours_array{5} hours_1-hours_5; do month = 1 to 5; if missing(rainfall_array{month}) then leave; Rain = rainfall_array{month}; AveTemp = temp_array{month}; AverageSunlight = hours_array{month}; Output; end; keep city rain month AveTem AverageSunlight; run; proc print data=rainfall1; run; SAS OUTPUT Output: Average Obs city month Rain AveTemp Sunlight 1 Hayward Hayward Hayward Hayward Hayward Oakland Oakland Alameda Sunnyval Sunnyval Sunnyval Sunnyval Fremont Fremont Fremont Fremont Fremont

A FEW POINTS TO NOTE:  A proc freq approach can be used in addition to a proc means approach  We present here the proc means approach  Both ways can give us the same result/output CODE proc means data=rainfall1 nway noprint; class city; output out=counts (rename=(_freq_ = N_Recorded) drop = _type_); run; proc print data=counts; run; Things to notice:  nway  nprint  rename

The SAS System 14:46 Sunday, May 30, Average Obs city N_Recorded _STAT_ month Rain AveTemp Sunlight 1 Alameda 1 N Alameda 1 MIN Alameda 1 MAX Alameda 1 MEAN Alameda 1 STD Fremont 5 N Fremont 5 MIN Fremont 5 MAX Fremont 5 MEAN Fremont 5 STD Hayward 5 N Hayward 5 MIN Hayward 5 MAX Hayward 5 MEAN Hayward 5 STD Oakland 2 N Oakland 2 MIN Oakland 2 MAX Oakland 2 MEAN Oakland 2 STD Sunnyval 4 N Sunnyval 4 MIN Sunnyval 4 MAX Sunnyval 4 MEAN Sunnyval 4 STD

 We would like to see the differences in values by city and by month for the following variables:  The average rainfall recorded (given by ‘rain’)  The average temperature recorded (given by ‘avetemp’)  The average sunlight recorded (given by ‘avesunlight’)

proc sort data=rainfall1 out=rainfall1; by city month; run; data last; set rainfall1; by city; put city=month=first.city = last.city=; if last.city; run;

log file: NOTE: There were 17 observations read from the data set WORK.RAINFALL1. NOTE: The data set WORK.RAINFALL1 has 17 observations and 5 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.01 seconds cpu time 0.00 seconds data last; 63 set rainfall1; 64 by city; 65 put city=month=first.city = last.city=; 66 if last.city; 67 run; city=Alameda month=1 FIRST.city=1 LAST.city=1 city=Fremont month=1 FIRST.city=1 LAST.city=0 city=Fremont month=2 FIRST.city=0 LAST.city=0 city=Fremont month=3 FIRST.city=0 LAST.city=0 city=Fremont month=4 FIRST.city=0 LAST.city=0 city=Fremont month=5 FIRST.city=0 LAST.city=1 city=Hayward month=1 FIRST.city=1 LAST.city=0 city=Hayward month=2 FIRST.city=0 LAST.city=0 city=Hayward month=3 FIRST.city=0 LAST.city=0 city=Hayward month=4 FIRST.city=0 LAST.city=0 city=Hayward month=5 FIRST.city=0 LAST.city=1 city=Oakland month=1 FIRST.city=1 LAST.city=0 city=Oakland month=2 FIRST.city=0 LAST.city=1 city=Sunnyval month=1 FIRST.city=1 LAST.city=0 city=Sunnyval month=2 FIRST.city=0 LAST.city=0 city=Sunnyval month=3 FIRST.city=0 LAST.city=0 city=Sunnyval month=4 FIRST.city=0 LAST.city=1 NOTE: There were 17 observations read from the data set WORK.RAINFALL1. NOTE: The data set WORK.LAST has 5 observations and 5 variables. NOTE: DATA statement used (Total process time): real time 0.01 seconds cpu time 0.01 seconds

data difference; set rainfall1; by city; if first.city and last.city then delete; Diff_Rain = rain - lag(rain); Diff_AveTemp = AveTemp - lag(AveTemp); Diff_AverageSunlight = AverageSunlight - lag(AverageSunlight); if not first.city then output; run; proc print data=difference; run;

The SAS System 14:46 Sunday, May 30, Ave Average Diff_ Diff_ Diff_ Average  Obs city month Rain Temp Sunlight Rain AveTemp Sunlight  1 Fremont  2 Fremont  3 Fremont  4 Fremont  5 Hayward  6 Hayward  7 Hayward  8 Hayward  9 Oakland  10 Sunnyval  11 Sunnyval  12 Sunnyval

data first_last; set rainfall1; by city; if first.city and last.city then delete; if first.city or last.city then do; diff_rain = rain - lag(rain); diff_temp = avetemp - lag(avetemp); diff_sunlight = averagesunlight - lag(averagesunlight); end; if last.city then output; run; proc print data=first_last; run;

Ave Average Average diff_ diff_ diff_ Obs city month Rain Temp Sunlight rain temp Sunlight 1 Fremont Hayward Oakland Sunnyval

Using RETAIN statement is one of the best ways to “remember” values from previous observations Variables that do not come from SAS data sets are set to a missing value during each iteration of the DATA step A RETAIN statement allows you to tell SAS not to do this

data new_data_set; set old_data_set; by ID; if first.ID and last.ID then delete; Retain First_Var_1 First_Var_2 … First_Var_3; If first.ID then do; First_Var_1 = Var_1 First_Var_2 = Var_2 … First_Var_n = Var_n If last.ID then do; Diff_Var_1 = Var_1 - First_Var_1; Diff_Var_2 = Var_2 - First_Var_2; … Diff_Var_n = Var_n - First_Var_n; End; Drop Frist_: ; Run; Need to sort by the ID, or which ever variable you are grouping by! The RETAIN statement ensures these variables are not set back to missing values during iteration. The Variables named within the RETAIN Statement are not replaced with missing values during the iterations.

 The RETAIN statement ensures your variables are not set back to missing values.  When processing first observation, the retained variables are set to respective variable values.  The last iteration subtracts the retained first values from the respective last variable values.

proc sort data=rainfall; by City Month; run; data last; set rainfall; by City; put City= Month= First.City= Last.City=; if last.City; run; data first_last; set rainfall; by City; if first.city and last.city then delete; retain First_Rainfall First_Temp First_Hours; if first.City then do; First_Rainfall = Rainfall; First_Temp = Temp; First_Hours = Hours; end; if last.City then do; Diff_Rainfall = Rainfall - First_Rainfall; Diff_Temp = Temp - First_Temp; Diff_Hours = Hours - First_Hours; output; end; drop First_: ; run;

Displaying the RETAIN Statement City Month Rainfall Temp Hours Diff_ Rainfall Diff_ Temp Diff_Hours Fremont Hayward Oakland Sunnyvale

Output from LAG statement: city month Rain Temp Sunlight diff_rain diff_temp diff_Sunlight Fremont Hayward Oakland Sunnyvale Output from RETAIN statement City Month Rainfall Temp Hours Diff_ Rainfall Diff_ Temp Diff_Hours Fremont Hayward Oakland Sunnyvale

 Suppose you want to know if a certain variable value is the maximum of all of you observations  The RETAIN Statement allows us to easily find this while preserving a variable’s value from previous iteration

data Maximums; Set rainfall; Retain Max_Rainfall Max_Temp Max_Hours; Max_Rainfall = Max(Max_Rainfall, Rainfall); Max_Temp = Max(Max_Temp, Temp); Max_Hours = Max(Max_Hours, Hours); run; title "Displaying the RETAIN Statement- Finding Maximums“; proc print data=Maximums noobs; run;

Displaying the RETAIN Statement- Finding Maximums city Month Rainfall Temp Hours Max_Rainfall Max_Temp Max_Hours Alameda Fremont Fremont Fremont Fremont Fremont Hayward Hayward Hayward Hayward Hayward Oakland Oakland Oakland Sunnyvale Sunnyvale Sunnyvale Sunnyvale