Instructor: Vincent Duffy, Ph.D. Associate Professor of IE Lecture 11 – Questionnaire Design & Evaluation Thurs. Feb. 22, 2007 IE 486 Work Analysis & Design II

Administrative Briefly review QOTD answer sheet from L10 Questionnaire design & analysis –(in preparation for Lab 3 beginning Friday) Lab 2 due in class next Tuesday 27 th

QOTD- Lecture 11 Q.1. Briefly discuss what is meant by criteria, measures & dependability. Q.2 What methods are appropriate for evaluating questionnaire data? Q.3 Should the same analyses be used for nominal/category data and continuous data?

Questionnaire design QOTD –included in slides Methods in Work Analysis & Design –Dependability of measures Evaluating questionnaire data Factor analysis Internal consistency –Questionnaires examples & scales

Q.1 Briefly discuss what is meant by criteria, measures & dependability This is related to methods of data collection & evaluation A criterion is an evaluation standard that can be used as a surrogate or correlate of outcome measures…such as system effectiveness, human performance and attitudes. –Eg. For driving performance we previously considered stopping distance and time to lane change (or lane deviations)

Q.1 Briefly discuss what is meant by criteria, measures & dependability Important aspects of criteria include: –Relative strengths/weaknesses of the data collection methods –Balance between costs of methods & available resources (eg. Consider motion capture test bed). –Costs include money, time, personnel & expertise Relevance

Q.1 Briefly discuss what is meant by criteria, measures & dependability Relative strengths/weaknesses of the data collection methods (dependability) Today, aspects of criteria & methods of evaluation Dependability is related to the validity & reliability Brief intro to dependability and measures of validity & reliability – more in lab tomorrow Important aspects of criteria include: –Relevance, Linearity and Homogeneity

Q.1 Briefly discuss what is meant by criteria, measures & dependability Relevance –Evaluation criteria must contribute to overall system mission Eg. Speed, response time, errors, comfort, acceptance Linearity – usually assumed –However, for industrial performance vs. intelligence, initially, no relationship was shown Additional analysis showed up to IQ of 90, r=0.46; between 90 and 110 r=0.04. For IQ>110, r=-0.52 (piecewise linear; see graph) –(non linear) IQ initially contributes to performance, then no impact, then too high IQ appears related to boredom see also Salvendy & Carayon (1997) supplementary reading for more detail Performance IQ

Q.1 Briefly discuss what is meant by criteria, measures & dependability Homogeneity – performance vs. time of day –Note how performance changes with time there appears to be warm up & slow down for the same operator The lack of consistency in performance over time of day raises issues of how/when to best collect data

Methods – evaluating questionnaire data Q.2 Which statistical technique can be used to evaluate questionnaire data?

Methods – evaluating questionnaire data Q.2 Which statistical technique can be used to evaluate questionnaire data? –It will depend on the objective of the analysis...

Methods – evaluating questionnaire data Which statistical technique becomes especially appropriate when questionnaire responses are measured on a nominal scale? First, what is meant by nominal scale? –Quality categorized as high, medium, low (discrete data) –Machine breakdowns due to mechanical failure electrical failure or operator misuse –For analysis of discrete data you can use a Chi-Square analysis

Methods – evaluating questionnaire data Q.3 Which statistical technique becomes especially appropriate when questionnaire responses are measured on a nominal scale?

Methods – evaluating questionnaire data Q.3 Which statistical technique becomes especially appropriate when questionnaire responses are measured on a nominal scale? A Chi Squared analysis For example – you can test whether color choice is related to gender FemaleMaleTotal Green chose green 38.9%22.2%61.1% chose green 64.6%36.4%64.6% of those who chose green were female Blue chose blue 16.7%22.2%38.9% chose blue 42.9%57.1%57.1% of those who chose blue were male Is there a significant difference in color choice, depending on gender? A Chi square statistic (shown after analysis of the original data set in SAS) shows  2 =7.5, p=0.006 (p<0.05). Hence, we would conclude yes.

Methods – evaluating questionnaire data Q.3 How might a response scale be designed to maximize the chance the responses (on a questionnaire) will be on a ratio measurement scale? –Use continuous, numerical, and anchored scales and do pilot testing – the average of questions/items can give a continuous measure. –A factor analysis will likely be done before an ANOVA (test of differences). -Factor analysis tries to find a factor (can think of it as a new variable) that will provide the highest set of correlations with the original variables (squares of these variables) producing the largest eigenvalue.

Examples of Scales Eg. Range from Very important – Very unimportant –With Neither important or unimportant in the middle – could have ‘moderately’ important in between. Or …Strongly agree, agree, undecided, disagree, strongly disagree Very much, much, fair, a little, not at all Much worse than usual, worse than usual, about the same, better than usual, much better than usual Excellent, very good, good, fair, poor All, most, a good bit, most, some, little, none Always, very often, fairly often, sometimes, almost never, never

The Hackman & Oldham Job Satisfaction Survey (1975) How satisfied are you with this aspect of your job? __1.The amount of job security I have. __2. The amount of pay and fringe benefits I receive __3. The amount of personal growth and development I get in doing my job. __4. The people I talk to and work with on my job. __5. The degree of respect and fair treatment I receive from my boss. __6. The feeling of worthwhile accomplishment I get from doing my job. __7. The chance I get to know other people while on the job. 1 Extremely Dissatisfied 2 Dissatisfied 3 Slightly Dissatisfied 4 Neutral 5 Slightly Satisfied 6 Satisfied 7 Extremely Satisfied

Methods – evaluating questionnaire data -Check eigenvalues of each factor (or variable) before deciding how many variables to include/consider - Eigenvalues should be greater than 1 for each factor that you include -Check percent of variance explained by each factor by taking the eigenvalue divided by the # of items -For example, if the eigenvalue (from linear algebra) is 3.66 with 6 questions (max. 6 units in the eigenvalue) – -then 3.66/6 or 0.61 (61%) of total variance in the questionnaire (whole questionnaire) is explained by the first factor. -You would expect a sum of at least 50% of the variance to be explained by the factors you have chosen to represent important variables.

Methods – evaluating questionnaire data -Check each factor before deciding how many variables to include/consider - Eigenvalues should be greater than 1 for each factor that you include -Check percent of variance explained by taking eigenvalue divided by the # of items (eg. If eigenvalue is 3.66 with 6 questions (max. 6 units in the eigenvalue –eg. 3.66/6 or 0.61 (61%) of total variance in the questionnaire (whole questionnaire) is explained by the first factor. An illustration of a principal components factor analysis with varimax rotation – item coefficients are shown below…(look for coefficients > 0.4) The values shown below are after rotation - and are easier to interpret Factor 1Factor 2 Ques. 1Feel blue Ques. 2People stare at me Ques. 3People follow me Ques. 4Basically happy Ques. 5People want to hurt me Ques. 6Enjoy going to parties Note: an item/question that loads on two different factors (such as question 5) would likely be dropped from further consideration; these factor analyses tend to be more useful when n is large (eg. n>100 participants)

Methods – Dependability - evaluating questionnaire data (cont.) -Factor analysis tries to find a factor (you can think of it as a new variable) that will provide the highest set of correlations with the original variables (the squares of these variables) producing the largest eigenvalue.

Structured questionnaire –Anchor –Avoid ambiguity –Avoid leading questions –Measure component –Measure whole –Sensitivity of scale –Halo effect –Dependability

Internal Consistency – testing reliability of the measures Internal consistency is the extent to which tests or procedures assess the same construct. It is a measure of the precision between the observers or of the measuring instruments used in a study. Cronbach’s Alpha measures how well a set of items (or variables) measure a single latent construct. Cronbach’s Alpha can be used as a measure of internal consistency. Cronbach’s Alpha as a measure of Internal Consistency

General steps toward analysis using a questionnaire Step 1: Conduct the survey using the structured questionnaire Step 2: Analyze the collected data Step 3: Make recommendation; report, presentation of results

Examples of inferences that can be drawn from questionnaire data 1) Which generic features (across manufacturers and models) are liked most, disliked most, or having greater difficulties in usage? 2) Which manufacturer and which model of the manufacturer is preferred regarding to each feature by the survey customers? 3) Which generic features (within manufacturers) are liked most, disliked most, or having greater difficulties in usage?

Can a measure be valid if it is not reliable? A measure can not be valid if it is not reliable. –If we can not measure it consistently, it is then hard to imagine that it can be correct. –Shown quantitatively… In lab more on…Reliability of Predictors and Criterion

Reliability of Predictors and Criterion –R 0 = Observed correlation (Validity) between the predictor and criterion –R T = “True” correlation (Validity) between the predictor and criterion The “True” correlation is one previously reported – possibly in the literature. –R p = Reliability of predictor –R c = Reliability of criterion Criterion related validity

–Suppose there are four tests in a battery with reliabilities of R 1 =0.60, R 2 =0.70, R 3 =0.78 and R 4 =0.92 (eg. These could be internal consistency reliabilities); –and three criteria are utilized with the following reliabilities: r 1 =0.45, r 2 =0.60, and r 3 =0.75. In other words, these reliabilities (r) are believed to be the relationships between two (or more) measures intended to be the measuring same criterion –Then: (part of Lab exercise tomorrow) Impact of Reliability on Validity

Cronbach’s Alpha as a measure of Internal Consistency k is the number of items in the group. s 2 res is the variance of residual components, which can not be controlled. s 2 p is the variance component for person. * The alpha coefficient is interpreted as the ratio of true score variance to observed score variance.

Cronbach’s Alpha as a measure of Internal Consistency For example, suppose 13 people were asked to rate a pair of questions on a 7-point scale. The pair of questions look different but they are testing the same item. For example –How much do you like the weather today? –How do you feel about the weather today?

People Observed Score (for each question) Total (Yi.) p p235 8 p333 6 p443 7 p55510 p653 8 p76612 p853 8 p95510 p p p p Total ( Y.j ) (Y..) Layout of Data Sheet calculation as part of lab exercise tomorrow

Questionnaire Design Supplemental material in R.W. Bailey, Human Performance Engineering 3 rd Ed. pp (Appendix). –And Webpage as shown: