IE 486 Work Analysis & Design II Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007

People Observed Score (for each question) Total (Yi.) p p235 8 p333 6 p443 7 p55510 p653 8 p76612 p853 8 p95510 p p p p Total ( Y.j ) (Y..) Layout of Data Sheet

Cronbach’s Alpha as a measure of Internal Consistency k is the number of items in the group. s 2 res is the variance of residual components, which can not be controlled. s 2 p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance.

Cronbach’s Alpha Coefficient Example: Number of people : N=13 Number of people : N=13 Number of questions: k=2 Number of questions: k=2 i =1..a, j = 1..b, where a=13, b=2. i =1..a, j = 1..b, where a=13, b=2. SS(total) =  (Yij 2 ) – Y.. 2 / ab SS(total) =  (Yij 2 ) – Y.. 2 / ab = …+5 2 – (111 2 )/26 = = …+5 2 – (111 2 )/26 = SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab = ( ….+9 2 )/2 – (111 2 )/26 = = ( ….+9 2 )/2 – (111 2 )/26 = SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab = ( )/13 – (111 2 )/26 = = ( )/13 – (111 2 )/26 = SS(residual) = SS(total) – SS(people) – SS(question) SS(residual) = SS(total) – SS(people) – SS(question) = – – = = – – = Cronbach’s Alpha as a measure of Internal Consistency

Cronbach’s Alpha Coefficient Example (cont’) MS p = SS p /(a-1) = /12 = MS r = SS r /(ab-a-b+1) = /12 = The estimated variance of person effect : S 2 p = [MS p – MS r ] / k = ( – ) / 2 = (variance component for person) = ( – ) / 2 = (variance component for person) The estimated variance of residual effect: s 2 r = MS r = (variance component for residual) s 2 r = MS r = (variance component for residual) The Alpha coefficient is calculated as: (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 p ) (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 p ) = 4*2.1026/(4* *0.6795) =

Exercise Suppose we collected additional data as follows: What is Cronbach’s alpha coefficient? –Can we conclude that our questionnaire has internal consistency. Subject #Question 1Question

IE 486 Work Analysis & Design II Instructor: Vincent Duffy, Ph.D. School of Industrial Eng. & Ag.& Bio Eng. Lab 3 – Evaluation of Questionnaire Data Friday, February 23, 2007

Cronbach’s Alpha as a measure of Internal Consistency k is the number of items in the group. s 2 res is the variance of residual components, which can not be controlled. s 2 p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance.

Cronbach’s Alpha as a measure of Internal Consistency For example, suppose 13 people were asked to rate a pair of questions on a 7-point scale. The pair of questions look different but they are testing the same item. For example –How much do you like the weather today? –How do you feel about the weather today?

People Observed Score (for each question) Total (Yi.) p p235 8 p333 6 p443 7 p55510 p653 8 p76612 p853 8 p95510 p p p p Total ( Y.j ) (Y..) Layout of Data Sheet

Cronbach’s Alpha Coefficient Example: Number of people : N=13 Number of people : N=13 Number of questions: k=2 Number of questions: k=2 i =1..a, j = 1..b, where a=13, b=2. i =1..a, j = 1..b, where a=13, b=2. SS(total) =  (Yij 2 ) – Y.. 2 / ab SS(total) =  (Yij 2 ) – Y.. 2 / ab = …+5 2 – (111 2 )/26 = = …+5 2 – (111 2 )/26 = SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab SS(people) =  (Yi. 2 ) / b – Y.. 2 / ab = ( ….+9 2 )/2 – (111 2 )/26 = = ( ….+9 2 )/2 – (111 2 )/26 = SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab SS(question) =  (Y.j 2 ) /a– Y.. 2 / ab = ( )/13 – (111 2 )/26 = = ( )/13 – (111 2 )/26 = SS(residual) = SS(total) – SS(people) – SS(question) SS(residual) = SS(total) – SS(people) – SS(question) = – – = = – – = Cronbach’s Alpha as a measure of Internal Consistency

Cronbach’s Alpha Coefficient Example (cont’) MS p = SS p /(a-1) = /12 = MS r = SS r /(ab-a-b+1) = /12 = The estimated variance of person effect : S 2 p = [MS p – MS r ] / k = ( – ) / 2 = (variance component for person) = ( – ) / 2 = (variance component for person) The estimated variance of residual effect: s 2 r = MS r = (variance component for residual) s 2 r = MS r = (variance component for residual) The Alpha coefficient is calculated as: (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 res ) (k 2 * s 2 p )) / (k 2 * s 2 p + k * s 2 res ) = 4*2.1026/(4* *0.6795) =

SAS code for Cronbach’s alpha data one; input q1 q2; cards; : 4 5 ; proc corr alpha; var q1 q2; run; Cronbach’s Alpha as a measure of Internal Consistency Output from SAS: Raw value of Coefficient : Same as the result from hand calculation.

Interpreting Cronbach’s Alpha results How to interpret the result? The higher the correlation coefficient, the higher the internal consistency of the test. The acceptable range for Cronbach ’ s alpha coefficient is usually between 0.7 – 1.0.

Interpreting Cronbach’s Alpha results What to do if the coefficient is low, such as 0.5? Check the following: –Are the questions ambiguous or not? –Are the scales sensitive enough to detect difference? Assumption: Question pairs are asked in the same direction

Exercise We wanted to measure a concept: ‘the user satisfaction about a web site’. To measure that concept, we made one pair of questions. –How do you like this web site? 1) strongly dislike … 4) neutral … 7) strongly like –I’m very pleased with this web site when using it. 1) strongly disagree … 4) neutral … 7) strongly agree

Exercise Suppose we collected additional data as follows: What is Cronbach’s alpha coefficient? Subject #Question 1Question

Exercise Subject #Question 1Question 2Total (Yi.) Total (Y. j) (Y..) Step 1: calculate total (Yi., Y.j, Y..)

Exercise Step 2: calculate SS (a = 4, b = 2) –SS(total) = ( ) – (39 2 )/8 = –SS(people) = ( )/2 – (39 2 )/8 = –SS(question) = ( )/4 – (39 2 )/8 = –SS(residual) = – – = Step 3: calculate MS –MS(people) = SS(people) / (a-1) = /(4-1) = = /(4-1) = –MS(residual) = SS(residual)/ (ab-a-b+1) = 2.375/( ) = = 2.375/( ) = 0.792

Exercise Step 4: calculate the estimated variance of person effect and residual effect (k = b = 2) S 2 (people) = [MS(people) – MS(residual)] / k = ( ) / 2 = = ( ) / 2 =  variance component for person  variance component for person S 2 (residual) = MS(residual) = =  variance component for residual  variance component for residual

Exercise Step 5: calculate the Cronbach’s alpha coefficient –Alpha coefficient = [ k 2 S 2 (people) ] / [k 2 S 2 (people) + k S 2 (residual) ] = [ k 2 S 2 (people) ] / [k 2 S 2 (people) + k S 2 (residual) ] = (2 2 * 2.833) / (2 2 * * 0.792) = (2 2 * 2.833) / (2 2 * * 0.792) = = –Alpha coefficient > 0.7 –We may conclude that our questionnaire has internal c\onsistency.