1. Evaluators Scott Novak, Ph.D. RTI International 1 July 2011

2. Overview Demonstrate how to use different Dashboard Reports for Evaluation Demonstrate how to use different Dashboard Reports for Evaluation Demonstrate new Custom Reports Demonstrate new Custom ReportsAgenda Introductions Introductions Reports Reports Questions Questions

3. What are the New Reports? Performance Management Dashboard Interface (Dashboards) Custom Dashboard Reports (Custom Reports) SAS/SPSS

5. Dashboard Reports

8. Click on any graph to see a report with additional information for that measure.

9. [Your Grant Number and Name will be here] Blue line is your grant. Gold line is comparison (GFA)

11. Table will show follow- ups due, follow-ups received, and follow-up rate by month and year

13. For each outcome, data table shows number, percent at intake, percent at 6 month, and the rate of change.

15. Employment for grant is being compared to State

16. Dashboard and Custom Reports Navigation Tips (cont.) View reports in HTML, PDF, EXCEL or XML by clicking the Earth icon in the upper right corner of the report screen. After clicking a view, choose HTML to return to the Dashboard display View reports in HTML, PDF, EXCEL or XML by clicking the Earth icon in the upper right corner of the report screen. After clicking a view, choose HTML to return to the Dashboard display Dashboards only Dashboards only Change the comparisons in each display by clicking the radio buttons in the list to the left of the graphic display Change the comparisons in each display by clicking the radio buttons in the list to the left of the graphic display Dashboards include only intakes matched with 6 month follow-up interviews (except Intake Target Report) Dashboards include only intakes matched with 6 month follow-up interviews (except Intake Target Report)

18. Custom Dashboard Reports

20. For specific date rage, click “Specific Date Range” button and then click “Apply” button

21. Must select an interview type to run any outcome change report

22. To run a custom age range, choose “Specific Age Range” and click the “Apply” button Use Ctrl button to choose multiple options within a demographic group

23. Custom age range

24. Breadcrumb heading allows you to see what selections you have made on previous pages Run report by clients who were homeless at intake

27. Custom Dashboard Reports

34. To save report, click “Keep this version”, then click “Save as Report View”

35. To save report, choose “Select My Folders” and type the name of your report. Then click “Ok”.

37. External Data Sources Help contexualize performance Help contexualize performance Understand trends in factors that influence access/capacity/treatment (e.g., area unemployment, drug epidemics) Understand trends in factors that influence access/capacity/treatment (e.g., area unemployment, drug epidemics) National Survey on Drug Use and Health, Treatment Episode Data Set, Others National Survey on Drug Use and Health, Treatment Episode Data Set, Others Benchmarking facilitate comparisons Benchmarking facilitate comparisons Similar organizations based on population, resources, and community Similar organizations based on population, resources, and community

38. Evaluation Analyses: Bringing It All Together Preliminary Analyses Preliminary Analyses Missing Data Missing Data Trends Trends Process and Outcome Analyses Process and Outcome Analyses Process: Identify areas of success/improvement Process: Identify areas of success/improvement Outcome: Client outcomes Outcome: Client outcomes Special Analytic Topics Special Analytic Topics Analysis of Change over time Analysis of Change over time

39. Preliminary Analyses Missing Data Missing Data Attrition/Drop Out Attrition/Drop Out Item Missingness Item Missingness Problem: Those who are retained may be different than those who drop/out or fail t answer questions Problem: Those who are retained may be different than those who drop/out or fail t answer questions

40. Alphabet Soup MCAR MCAR MAR MAR NMAR NMAR Ignorable/non- ignorable Ignorable/non- ignorable GEE GEE RC/RE RC/RE HLM/MLM HLM/MLM PMM PMM DID DID

41. Missing Completely At Random (MCAR) Missing Completely At Random (MCAR) Likelihood of missing data (item or assessment) are due to “chance” factors. Missingness unrelated to any specific survey item (observable) Likelihood of missing data (item or assessment) are due to “chance” factors. Missingness unrelated to any specific survey item (observable) Sick on day of test administration (missed assessment) Sick on day of test administration (missed assessment) Miss item on survey because not “paying attention” Miss item on survey because not “paying attention” No differences on any factor that is not observed (unobservable) in the study, but is related to study outcome (e.g., no differences in the likelihood of being sick between smokers and non-smokers) No differences on any factor that is not observed (unobservable) in the study, but is related to study outcome (e.g., no differences in the likelihood of being sick between smokers and non-smokers) Ignorable in that results will not be biased Ignorable in that results will not be biased

42. Missing At Random (MAR) Likelihood of missingness (item or assessment) is due to a respondent characteristic(s) collected in study. Likelihood of missingness (item or assessment) is due to a respondent characteristic(s) collected in study. Smokers are more likely to be absent on day of data collection, and information on smoking status is collected. Smokers are more likely to be absent on day of data collection, and information on smoking status is collected. People who smoke are less likely to report income, and smokers have lower incomes, in general. Smoking status is collected in study. People who smoke are less likely to report income, and smokers have lower incomes, in general. Smoking status is collected in study. Analyses will be biased unless appropriate procedures are used, perhaps including information on smoking. Analyses will be biased unless appropriate procedures are used, perhaps including information on smoking. Non-ignorable, unless appropriate procedures are in place. Non-ignorable, unless appropriate procedures are in place.

43. Not Missing at Random (NMAR) Likelihood of or assessment) are due to factors not observed in study. Likelihood of or assessment) are due to factors not observed in study. Smokers more likely to be missing on income variable and smokers have lower income. But do not have information on smoking. Smokers more likely to be missing on income variable and smokers have lower income. But do not have information on smoking. Unverifiable assumption. Can distinguish MCAR and MAR by whether a factor is related to an observed covariate. Cannot distinguish between MCAR and NMAR because it is purely a hypothesis as to why data are missing. Unverifiable assumption. Can distinguish MCAR and MAR by whether a factor is related to an observed covariate. Cannot distinguish between MCAR and NMAR because it is purely a hypothesis as to why data are missing. Two procedures: Selection Models and Pattern Mixture Models Two procedures: Selection Models and Pattern Mixture Models Selection: Model the joint response of outcome and predictor Selection: Model the joint response of outcome and predictor Pattern Mixture: Average effect based on unique pattern of missingness Pattern Mixture: Average effect based on unique pattern of missingness

44. Item-Level Missing Data Data on respondents, but items missing within survey Data on respondents, but items missing within survey Typically ignored in practice Typically ignored in practice Exclude data Exclude data Reduce likelihood of detecting significant effects Reduce likelihood of detecting significant effects Bias results depending upon the nature of missing data Bias results depending upon the nature of missing data

45. Frequency of Item-Missing Data: Random

46. Frequency of Item-Missing Data: Non-Random

47. Missing Data in Longitudinal Studies Missing: Drop-out at any given wave Missing: Drop-out at any given wave Pattern: Structure of Drop-out Pattern: Structure of Drop-out Monotonic Missing Data: Monotonic Missing Data: A data row of variables Y1, Y2,..., Yp (in that order) is said to have a monotone missing pattern when the event that a variable Yj is missing for a particular individual implies that all subsequent variables Yk, k > j, are missing for that individual. Alternatively, when a variable Yj is observed for a particular individual, it is assumed that all previous variables Yk, k j, are missing for that individual. Alternatively, when a variable Yj is observed for a particular individual, it is assumed that all previous variables Yk, k < j, are also observed for that individual (defines attriter)

48. Understanding Drop-Out Drop-out: Only presenting data on clients who were successfully followed up may cloud interpretation of data Drop-out: Only presenting data on clients who were successfully followed up may cloud interpretation of data Hard-to-Treat cases often drop-out Hard-to-Treat cases often drop-out If near 100% follow-up rate, then more confidence in follow-up data If near 100% follow-up rate, then more confidence in follow-up data Identifying cases lost-to follow-up can help targeting/recruitment efforts Identifying cases lost-to follow-up can help targeting/recruitment efforts Need to understanding “coding” of drop-out (administrative discharge) Need to understanding “coding” of drop-out (administrative discharge)

49. Missing Data Patterns for 3 Observations value pattern 1='0 0 0' value pattern 1='0 0 0' 2='M 0 0' 2='M 0 0' 3='0 M 0' 3='0 M 0' 4='M M 0' 4='M M 0' 5='0 0 M' 5='0 0 M' 6='M 0 M' 6='M 0 M' 7='0 M M' 7='0 M M' 8='M M M'; 8='M M M'; value mono 0='complete data' value mono 0='complete data' 1='drop after baseline' 1='drop after baseline' 2='drop after eot' 2='drop after eot' 3='drop after 2m' 3='drop after 2m' 4='missing data at any other 4='missing data at any other

50. Types of Analyses for Drop-Out Identify patterns of drop-out Identify patterns of drop-out Understand assumptions for methods used to analyze longitudinal data Understand assumptions for methods used to analyze longitudinal data GEE: MCAR GEE: MCAR RE: MAR RE: MAR REGRESSION:/ANOVA MCAR REGRESSION:/ANOVA MCAR Determine which variables may be related to missingness and attrition (monotonic) Determine which variables may be related to missingness and attrition (monotonic) Conduct sensitivity (stability) analyses under different assumptions Conduct sensitivity (stability) analyses under different assumptions

51. Outcome and Process Data Challenge—no control group, unlike randomized control trial (RCT) Challenge—no control group, unlike randomized control trial (RCT) Variation in Clients: Variation in Clients: Treatment Setting: Treatment Setting: Detox Detox Inpatient (long-term/short-term) Inpatient (long-term/short-term) Outpatient (long-term/short-term) Outpatient (long-term/short-term) Opioid Opioid Demographics: Demographics: Age (adolescent)/race/gender/homeless Age (adolescent)/race/gender/homeless Co-occurring disorders: Co-occurring disorders: Substance abuse/mental health Substance abuse/mental health Variation in MH (personality disorders versus mood/anxiety/other) Variation in MH (personality disorders versus mood/anxiety/other) HIV status and risk behaviors HIV status and risk behaviors

52. Understanding Retention (PROCSS) as Missing Data/Treatment Exposure Understanding Retention (PROCSS) as Missing Data/Treatment Exposure Length of Stay —Number of days on service Length of Stay —Number of days on service More treatment contact is better, but at a price More treatment contact is better, but at a price Stratify analyses by type of service received when presenting length of stay data Stratify analyses by type of service received when presenting length of stay data Include types of treatment received and substance use characteristics Include types of treatment received and substance use characteristics

53. Assessing Program Impact Always pair with retention data Always pair with retention data Consider intake and discharge together Consider intake and discharge together Consider positive and negative rates of change Consider positive and negative rates of change Specify different types of models Specify different types of models Mediator Mediator Moderator Moderator

54. Mediator Identifies ways in which intervention “works” Identifies ways in which intervention “works” Baron and Kenny (1986) Baron and Kenny (1986) Causal Steps A  B  C Causal Steps A  B  C A  B, B  C, Effect of A  weakened upon introduction of “C” A  B, B  C, Effect of A  weakened upon introduction of “C” Compute Total Effect, Direct Effect, and Indirect Effect Compute Total Effect, Direct Effect, and Indirect Effect

55. 55 Key Points in Presenting Follow-up Data on Outcomes Intake6- Month Follow-Up 1. Positive to positive 2. Negative to positive 3. Positive to negative 4. Negative to negative (Abstinent, employed, etc.) (Not abstinent, unemployed, etc.)

56. 56 Positive outcomes to negative outcomes Intake6- Month Follow-Up 1. Positive to positive 2. Negative to positive 3. Positive to negative 4. Negative to negative (Abstinent, employed, etc.) (Not abstinent, unemployed, etc.)

57. 57 Negative to Negative Intake6- Month Follow-Up 1. Positive to positive 2. Negative to positive 3. Positive to negative 4. Negative to negative (Abstinent, employed, etc.) (Not abstinent, unemployed, etc.)

58. Societal Macrosystems Ecological Model Proximal Social Contexts CloseInterpersonalRelationships Individual Factors Factors

59. Strategies for Longitudinal Data: Differ on Ways to Handle Missingness Lagged/DID Model (T1/T2) Lagged/DID Model (T1/T2) MCAR/MAR: Use procedures for item missingness MCAR/MAR: Use procedures for item missingness NMAR: No sound adjustments available NMAR: No sound adjustments available Repeated Measures ANOVA: Repeated Measures ANOVA: MCAR MCAR Covariate control: Add single dummy variable Covariate control: Add single dummy variable

60. Strategies (continued) Random Effects Random Effects MCAR MCAR No adjustment needed No adjustment needed NMAR NMAR Pattern Mixture Models Pattern Mixture Models Selection models Selection models MAR MAR Covariate control Covariate control Propensity weights Propensity weights GEE GEE MCAR-No adjustment needed MCAR-No adjustment needed MAR-Imputation MAR-Imputation

61. Lagged Model Research Question: Are there differences in 6 month outcome of abstinence from drinking by treatment exposure? Research Question: Are there differences in 6 month outcome of abstinence from drinking by treatment exposure? Model: Logistic regression (lagged): Model: Logistic regression (lagged): Outcome: Alcohol Use (binary) at 6 month wave Outcome: Alcohol Use (binary) at 6 month wave Predictors: Alcohol use, treatment exposure Predictors: Alcohol use, treatment exposure Assumes complete data Assumes complete data If missing 6 month or baseline, identify pattern of missing data If missing 6 month or baseline, identify pattern of missing data Identify assumption of structure (MAR/MCAR/NMAR) Identify assumption of structure (MAR/MCAR/NMAR) Impute using Proc MI is assumptions hold Impute using Proc MI is assumptions hold If no adjustment, then results may be biased if MAR or NMAR and loss of power If no adjustment, then results may be biased if MAR or NMAR and loss of power

62. Difference-in-Difference (DID)Model Research Question: Are there differences in 6 month outcome of Changes in Reading Achievement by treatment condition? Research Question: Are there differences in 6 month outcome of Changes in Reading Achievement by treatment condition? Model: OLS regression Model: OLS regression Outcome: Changes in reading levels between baseline and 6 month waves Outcome: Changes in reading levels between baseline and 6 month waves Predictors: baseline reading levels, treatment condition Predictors: baseline reading levels, treatment condition Assumes complete data Assumes complete data If missing 6 month or baseline, identify pattern of missing data If missing 6 month or baseline, identify pattern of missing data Identify assumption of structure (MAR/MCAR/NMAR) Identify assumption of structure (MAR/MCAR/NMAR) Impute using Proc MI is assumptions hold Impute using Proc MI is assumptions hold If no adjustment, then results may be biased if MAR or NMAR, and loss of power If no adjustment, then results may be biased if MAR or NMAR, and loss of power

63. Repeated Measures (RM) ANOVA Research Question: Are there differences in mean- levels of reading Achievement by treatment condition? Research Question: Are there differences in mean- levels of reading Achievement by treatment condition? Model: ANOVA (RM) Model: ANOVA (RM) Outcome: Mean reading levels at baseline, eot, 2m and 6m Outcome: Mean reading levels at baseline, eot, 2m and 6m Predictors: survey wave, treatment condition, wave*cond Predictors: survey wave, treatment condition, wave*cond Assumes complete data Assumes complete data Identify assumption of structure (MAR/MCAR/NMAR) Identify assumption of structure (MAR/MCAR/NMAR) Impute using Proc MI is assumptions hold Impute using Proc MI is assumptions hold If no adjustment, then results may be biased if MAR or NMAR, and loss of power If no adjustment, then results may be biased if MAR or NMAR, and loss of power

64. Random Effects Research Question: Do the levels of a subject’s reading achievement change over time and by condition? Research Question: Do the levels of a subject’s reading achievement change over time and by condition? Model: Random Effects Model: Random Effects Outcome: Ss reading levels at baseline, eot, 2m and 6m Outcome: Ss reading levels at baseline, eot, 2m and 6m Predictors: survey wave, treatment condition, wave*cond Predictors: survey wave, treatment condition, wave*cond Assumes complete data within wave, but missing between wave Assumes complete data within wave, but missing between wave Identify assumption of drop-out structure (MAR/MCAR/NMAR) Identify assumption of drop-out structure (MAR/MCAR/NMAR) Allows for MAR and MCAR Allows for MAR and MCAR

65. Random Effects: MCAR and MAR Random Effects using a full information maximum likelihood procedure (FIML) allows for drop-out pattern of missing data, not item missingness within survey wave Random Effects using a full information maximum likelihood procedure (FIML) allows for drop-out pattern of missing data, not item missingness within survey wave Change default in SAS from restricted maximum likelihood (REML) because more robust than full information (FIML) Change default in SAS from restricted maximum likelihood (REML) because more robust than full information (FIML) Canned algorithms in SAS (e.g., Expectation Maximization) used to derive estimates for analyses. All performed within single procedure (e.g., Proc MIXED/Proc NLMIXED) Canned algorithms in SAS (e.g., Expectation Maximization) used to derive estimates for analyses. All performed within single procedure (e.g., Proc MIXED/Proc NLMIXED) If MAR, then include covariates to improve fitting algorithm If MAR, then include covariates to improve fitting algorithm

66. Random Effects NMAR Pattern Mixture Models: Pattern Mixture Models: Hedeker, D., & Gibbons, R.D. (1997). Application of random-effects pattern-mixture models for missing data in longitudinal studies. Psychological Methods, 2, 64-78. Hedeker, D., & Gibbons, R.D. (1997). Application of random-effects pattern-mixture models for missing data in longitudinal studies. Psychological Methods, 2, 64-78. Estimate different models based on drop-out patterns Estimate different models based on drop-out patterns Any-missing data versus completers Any-missing data versus completers Attriters versus completers versus missing Attriters versus completers versus missing Obtain pattern-mixture average results Obtain pattern-mixture average results Average over set of different patterns of missingness Average over set of different patterns of missingness Adjusted for proportion of drop-out Adjusted for proportion of drop-out

67. Generalized Estimating Equations Research Question: Are there differences in population-averaged levels of Educational Change by treatment condition? Research Question: Are there differences in population-averaged levels of Educational Change by treatment condition? Model: GEE Model: GEE Outcome: Reading levels at baseline, discharge and 6m Outcome: Reading levels at baseline, discharge and 6m Predictors: survey wave, treatment condition, wave*cond Predictors: survey wave, treatment condition, wave*cond Allows for missing data (MCAR only) Allows for missing data (MCAR only) Identify assumption of structure (MAR/MCAR/NMAR) Identify assumption of structure (MAR/MCAR/NMAR) Impute using Proc MI is assumptions hold Impute using Proc MI is assumptions hold If no adjustment, then results may be biased if MAR or NMAR, and loss of power If no adjustment, then results may be biased if MAR or NMAR, and loss of power

68. Concluding Thoughts Conduct analyses under different strategies to interrogate the influence of missing data (item or drop-out) on inferences from models Conduct analyses under different strategies to interrogate the influence of missing data (item or drop-out) on inferences from models Especially given that cannot verify MCAR and NMAR Especially given that cannot verify MCAR and NMAR Convergence in approaches will strengthen confidence in conclusions Convergence in approaches will strengthen confidence in conclusions Ignoring missing data can lead to loss of power due by excluding cases Ignoring missing data can lead to loss of power due by excluding cases Always report results of sensitivity analyses so reviewers of your work can more thoroughly evaluate your findings Always report results of sensitivity analyses so reviewers of your work can more thoroughly evaluate your findings