1. IE 486 Work Analysis & Design II Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007

2. People Observed Score (for each question) Total (Yi.) p176 13 p235 8 p333 6 p443 7 p55510 p653 8 p76612 p853 8 p95510 p1012 3 p117613 p1222 4 p1345 9 Total ( Y.j ) 57 54 111 (Y..) Layout of Data Sheet

3. Cronbach’s Alpha as a measure of Internal Consistency k is the number of items in the group. s 2 res is the variance of residual components, which can not be controlled. s 2 p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance.

4. Cronbach’s Alpha Coefficient Example: Number of people : N=13 Number of people : N=13 Number of questions: k=2 Number of questions: k=2 i =1..a, j = 1..b, where a=13, b=2. i =1..a, j = 1..b, where a=13, b=2. SS(total) =  (Yij 2 ) – Y.. 2 / ab SS(total) =  (Yij 2 ) – Y.. 2 / ab = 7 2 +6 2 +3 2 +…+5 2 – (111 2 )/26 = 67.1154 = 7 2 +6 2 +3 2 +…+5 2 – (111 2 )/26 = 67.1154 SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab = (13 2 +8 2 +….+9 2 )/2 – (111 2 )/26 = 58.6154 = (13 2 +8 2 +….+9 2 )/2 – (111 2 )/26 = 58.6154 SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab = (57 2 +54 2 )/13 – (111 2 )/26 = 0.3462 = (57 2 +54 2 )/13 – (111 2 )/26 = 0.3462 SS(residual) = SS(total) – SS(people) – SS(question) SS(residual) = SS(total) – SS(people) – SS(question) = 67.1154 – 58.6154 – 0.3462 = 8.1538 = 67.1154 – 58.6154 – 0.3462 = 8.1538 Cronbach’s Alpha as a measure of Internal Consistency

5. Cronbach’s Alpha Coefficient Example (cont’) MS p = SS p /(a-1) = 58.6154/12 = 4.8846 MS r = SS r /(ab-a-b+1) = 8.1538/12 = 0.6795 The estimated variance of person effect : S 2 p = [MS p – MS r ] / k = (4.8846 – 0.6795) / 2 = 2.1026 (variance component for person) = (4.8846 – 0.6795) / 2 = 2.1026 (variance component for person) The estimated variance of residual effect: s 2 r = MS r = 0.6795 (variance component for residual) s 2 r = MS r = 0.6795 (variance component for residual) The Alpha coefficient is calculated as: (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 p ) (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 p ) = 4*2.1026/(4*2.1026 + 2*0.6795) = 0.86089

6. Exercise Suppose we collected additional data as follows: What is Cronbach’s alpha coefficient? –Can we conclude that our questionnaire has internal consistency. Subject #Question 1Question 2 167 232 357 445

7. IE 486 Work Analysis & Design II Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007

8. Cronbach’s Alpha as a measure of Internal Consistency k is the number of items in the group. s 2 res is the variance of residual components, which can not be controlled. s 2 p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance.

9. Cronbach’s Alpha as a measure of Internal Consistency For example, suppose 13 people were asked to rate a pair of questions on a 7-point scale. The pair of questions look different but they are testing the same item. For example –How much do you like the weather today? –How do you feel about the weather today?

10. People Observed Score (for each question) Total (Yi.) p176 13 p235 8 p333 6 p443 7 p55510 p653 8 p76612 p853 8 p95510 p1012 3 p117613 p1222 4 p1345 9 Total ( Y.j ) 57 54 111 (Y..) Layout of Data Sheet

11. Cronbach’s Alpha Coefficient Example: Number of people : N=13 Number of people : N=13 Number of questions: k=2 Number of questions: k=2 i =1..a, j = 1..b, where a=13, b=2. i =1..a, j = 1..b, where a=13, b=2. SS(total) =  (Yij 2 ) – Y.. 2 / ab SS(total) =  (Yij 2 ) – Y.. 2 / ab = 7 2 +6 2 +3 2 +…+5 2 – (111 2 )/26 = 67.1154 = 7 2 +6 2 +3 2 +…+5 2 – (111 2 )/26 = 67.1154 SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab = (13 2 +8 2 +….+9 2 )/2 – (111 2 )/26 = 58.6154 = (13 2 +8 2 +….+9 2 )/2 – (111 2 )/26 = 58.6154 SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab = (57 2 +54 2 )/13 – (111 2 )/26 = 0.3462 = (57 2 +54 2 )/13 – (111 2 )/26 = 0.3462 SS(residual) = SS(total) – SS(people) – SS(question) SS(residual) = SS(total) – SS(people) – SS(question) = 67.1154 – 58.6154 – 0.3462 = 8.1538 = 67.1154 – 58.6154 – 0.3462 = 8.1538 Cronbach’s Alpha as a measure of Internal Consistency

12. Cronbach’s Alpha Coefficient Example (cont’) MS p = SS p /(a-1) = 58.6154/12 = 4.8846 MS r = SS r /(ab-a-b+1) = 8.1538/12 = 0.6795 The estimated variance of person effect : S 2 p = [MS p – MS r ] / k = (4.8846 – 0.6795) / 2 = 2.1026 (variance component for person) = (4.8846 – 0.6795) / 2 = 2.1026 (variance component for person) The estimated variance of residual effect: s 2 r = MS r = 0.6795 (variance component for residual) s 2 r = MS r = 0.6795 (variance component for residual) The Alpha coefficient is calculated as: (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 res ) (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 res ) = 4*2.1026/(4*2.1026 + 2*0.6795) = 0.86089

13. SAS code for Cronbach’s alpha data one; input q1 q2; cards; 7 6 3 5 3 : 4 5 ; proc corr alpha; var q1 q2; run; Cronbach’s Alpha as a measure of Internal Consistency Output from SAS: Raw value of Coefficient : 0.860892 Same as the result from hand calculation.

14. Interpreting Cronbach’s Alpha results How to interpret the result? The higher the correlation coefficient, the higher the internal consistency of the test. The acceptable range for Cronbach ’ s alpha coefficient is usually between 0.7 – 1.0.

15. Interpreting Cronbach’s Alpha results What to do if the coefficient is low, such as 0.5? Check the following: –Are the questions ambiguous or not? –Are the scales sensitive enough to detect difference? Assumption: Question pairs are asked in the same direction

16. Exercise We wanted to measure a concept: ‘the user satisfaction about a web site’. To measure that concept, we made one pair of questions. –How do you like this web site? 1) strongly dislike … 4) neutral … 7) strongly like –I’m very pleased with this web site when using it. 1) strongly disagree … 4) neutral … 7) strongly agree

17. Exercise Suppose we collected additional data as follows: What is Cronbach’s alpha coefficient? Subject #Question 1Question 2 167 232 357 445

18. Exercise Subject #Question 1Question 2Total (Yi.) 16713 2325 35712 4459 Total (Y. j)182139 (Y..) Step 1: calculate total (Yi., Y.j, Y..)

19. Exercise Step 2: calculate SS (a = 4, b = 2) –SS(total) = (6 2 +7 2 +3 2 +2 2 +5 2 +7 2 +4 2 +5 2) – (39 2 )/8 = 22.875 –SS(people) = (13 2 +5 2 +12 2 +9 2 )/2 – (39 2 )/8 = 19.375 –SS(question) = (18 2 +21 2 )/4 – (39 2 )/8 = 1.125 –SS(residual) = 22.875 – 19.375 – 1.125 = 2.375 Step 3: calculate MS –MS(people) = SS(people) / (a-1) = 19.375/(4-1) = 6.458 = 19.375/(4-1) = 6.458 –MS(residual) = SS(residual)/ (ab-a-b+1) = 2.375/(8-4-2+1) = 0.792 = 2.375/(8-4-2+1) = 0.792

20. Exercise Step 4: calculate the estimated variance of person effect and residual effect (k = b = 2) S 2 (people) = [MS(people) – MS(residual)] / k = (6.458 - 0.792) / 2 = 2.833 = (6.458 - 0.792) / 2 = 2.833  variance component for person  variance component for person S 2 (residual) = MS(residual) = 0.792 = 0.792  variance component for residual  variance component for residual

21. Exercise Step 5: calculate the Cronbach’s alpha coefficient –Alpha coefficient = [ k 2 S 2 (people) ] / [k 2 S 2 (people) + k S 2 (residual) ] = [ k 2 S 2 (people) ] / [k 2 S 2 (people) + k S 2 (residual) ] = (2 2 * 2.833) / (2 2 * 2.833 + 2 * 0.792) = (2 2 * 2.833) / (2 2 * 2.833 + 2 * 0.792) = 0.877 = 0.877 –Alpha coefficient > 0.7 –We may conclude that our questionnaire has internal c\onsistency.