S519: Evaluation of Information Systems Week 14: April 7, 2008

2 Announcement Guest lecture next week Director Jonathan Plucker from the Center for Evaluation and Education Policy

3 Data Analysis Report Report the results and interpretation of the analysis of the data you collected last week Explain what kind of analysis was conducted and why The analysis may require statistical analysis or may involve qualitative methods, such as analytic induction or content analysis Include the raw data set in an appendix Do not include all the tests you run, but do include ones that you find important 2-3 single-spaced pages

4 Inferential Statistics Descriptive statistics = summarize characteristics of sets of data & relationships between various sets of data Inferential statistics = make estimates of how likely it is that such characteristics or relationships exist in the universe (generalization)

5 Inferential Statistics Inferential statistics differ from descriptive statistics in two fundamental ways: They are usually concerned with the strength of relationships (associations or differences) among two or more sets of measured variables They have accompanying tests of statistical significance (See Salkind Ch 8) which indicate how likely it is that a particular relationship would occur by chance, given a certain sized sample

6 Relationships Relationships are important to examine because: answering research questions to examine, e.g., relationships between independent variables and dependent variables Suggesting new research hypotheses and/or Qs

7 Relationships We are looking for answers to the two questions, when asking about associations and/or differences in sample data: How strong are the associations and/or differences? How likely is it that these associations and/or differences exist in the universe (population) from which the sample was drawn?

8 Statistically Significant Statistically “significant” means that any difference between the attitudes of the two groups is due to some systematic influence and not due to chance

9 Significance Level Significant findings occurred at the 0.05 level (p <.05)  there is 1 chance in 20 (or 5%) that any differences found were not due to the hypothesized reason, but to some other 5% of all values 95% of all values Do not reject nullReject null Critical value

10 How Inference Works 1. Select representative samples from two different groups (e.g., mac users vs. pc users) 2. Each user is administered a survey to assess his/her attitude. The mean scores for groups are computed and compared using some tests 3. A conclusion is reached as to whether or not the difference between the scores is the result of chance or the result of “true” and statistically different differences between the groups 4. Based on the results of analysis of the sample data, an inference is made about the entire population

11 Relationships

12 Relationships: T-test Difference??? Treat b

13 T-test: Tests between the Means of Different Groups t = (n -1)s +(n -1)s n + n - 2 n n + n X - X X is the mean for group 1 n is the # of participants in group 1 s is the variance for group The difference between the means The amount of variation within & between each of the two groups

14 T-test: Tests between the Means of Different Groups Compare two groups: one using PDA and the other using paper-based in a hospital These two populations do not overlap The variable is normally distributed in each of the two populations The variances of the normally distributed test variable for the two populations are equal Use the “TTEST” function or Data Analysis ToolPak  select “Two-Sample Assuming Equal Variances”

15 T-test: Tests between the Means of Related Groups Compare means of related groups E.g., pre- and post- IT implementation Comparison of means from each group Focus on the differences between the scores t =  D n  D - (  D) (n-1) 2 2  D is the sum of all the differences between groups n is the # of pairs of observations

16 T-test: Tests between the Means of the Same Group Use the “TTEST” function or Data Analysis ToolPak  select “Paired Two Sample for Means”

17 T-test: One Sample Use it when you only have one sample Useful for comparing with a hypothetical mean or previous evaluation score Unfortunately, there is no easy way to calculate this, so if you want, you can use SPSS

18 T-test: One Sample

19 T-test: One Sample

20 T-test: One Sample

21 More Than Two Groups: Analysis of Variance (ANOVA) Simple ANOVA One factor or one treatment variable (e.g. group membership) being explored More than two groups within this factor Simple ANOVA is used where: There is only one dimension or treatment There are more than 2 levels of the grouping factor One is looking at differences across groups in averages scores Excel: Data Analysis ToolPak  select “Anova: Single Factor”

22 Determining Importance (Davidson, 2005) Dimensional evaluation E.g., DeLone & McLean’s IS Effectiveness Model Component evaluation E.g., policies, programs (e.g., Teen program in a library) Holistic evaluation Personnel, product, service

23 6 Strategies for Determining Importance (Davidson, 2005) Have stakeholders or consumers “vote” on importance Draw on the knowledge of selected stakeholders Use evidence from the literature Use specialist judgment Use evidence from the needs and values assessments Use program theory and evidence of causal linkages What are pros and cons of these strategies?

24 Basic Steps in Data Analysis (adapted from LeCompte & Preissle, 1993) 1. Review proposal and research question 2. Scan 1. Check data for completeness [go back to the original informants] 2. Jot notes 3. Write summaries, e.g., research log, post-it 4. Speculate 5. Determine categories/ identify themes 6. Notice relationships 7. Theorize and eliminate rival theories (interpretations). Begin, often with typologies 8. Try to confirm and to disconfirm interpretations