Management of Computer System Performance Chapter 12 IT Surveys and Data Collection Techniques

IT Surveys and Data Collection Techniques Agenda: Studies, Surveys and Alternatives Identifying Intangible Value How collection can be done When should data collection be done. Objective: Students should be able: to analyze and describe IT surveys into a job sheets. .

IT Surveys for benefits measurement Survey creation is a complex process. A sample survey is provided for reference purposes only. It is strongly suggested that a great deal of research be done to properly prepare the survey instrument. This will ensure that: The results received meet the requirements. That the survey does not skew the resulting data (leading questions, etc.)

Identifying Intangible Value The process of identifying soft or intangible costs or benefits is one of the most difficult in performing a cost or benefit analysis. It may also be the most important. The goal is to assign value. This requires identifying and quantifying something that is often ancillary to hard costs or benefits.

On-line banking On-line banking can be used as an example of this problem. Some hard benefits and costs are as follows: Tangible Costs Tangible Benefits On-line consumer systems Lower Labor costs (Fewer Tellers) Added system security Increased automatic transactions Testing Greater fee generation Additional IT labor Fewer/Smaller facilities (lower rent) Marketing & Advertising Lower Insurance costs

On-line banking Intangible benefits are initially intuitive or perceived. These could include: Intangible Costs Consumer Dissatisfaction with Intangible Benefits Consumer Satisfaction with System/GUI Not having to go to the bank/No Lines No personal contact No one available to ask questions Simple transactions/One stop shop Need a fast computer Easy access Keeping track of passwords Better lifestyle “fit”

Identifying Intangible Value The tangible values can be calculated either directly readily estimated on hard values. Intangible valuation requires additional data gathering. Measured data will yield the best results. Considering the alternatives that offer the greatest flexibility, surveys must be addressed as a valid means to acquire measurable data. Guessing isn’t acceptable. The corollary is that surveys are not cheap.

Data Collection Costs Task Costs – Manual Surveys Labor Costs Cover letter and questionnaire design & typing  Purchasing mailing list cost (if necessary)  Addressing mailing envelopes  Cover letter and survey printing costs  Envelope costs (both ways + more)  Postage costs (both ways + more)  Incentives Data entry and verification  Statistical analysis software  Distribution of the final report

Data Collection Costs Task Costs – Automated Surveys Labor Costs Planning Specifications Applications Hardware Questionnaire Design Notification System Participant identification Incentives Data entry and verification  Statistical analysis software  Distribution of the final report

The Survey Process-Methodology There is a sequential methodology that can be used to optimize the creation and use of a survey. Knowing what you want to achieve and planning how to get there are the best way to succeed. This approach does take time and costs money.

The Survey Process – Goals Step 1: Identify what you want to accomplish This step identifies the goal you want to achieve. In this case, a valuation of one or more intangible elements of a project. A goal is not the same as a result. A goal should be a neutral and not drive towards a predetermined result. Have a trusted third party check your work. Predeterminations sneak in sometimes.

The Survey Process - Planning Step 1: Planning Questionnaire Research Consider the advantages and disadvantages of using questionnaires. Prepare written objectives for the research. Have your objectives reviewed by others. Review the literature related to the objectives. Determine the feasibility of administering your questionnaire to the population of interest. Prepare a time-line.

The Survey Process - Design Step 2: Design Methodology Is it written, on-line web based, verbal or application driven? Consider the advantages and disadvantages of using questionnaires. Prepare written objectives for the research. Have your objectives reviewed by others. Review the literature related to the objectives. Determine the feasibility of administering your questionnaire to the population of interest. Prepare a time-line.

The Survey Process - Feasibility Step 3: Determine the Feasibility Is it written, on-line web based, verbal or application driven? Can a survey collect the needed information? Can it be done in the time provided? Can it be done with the available budget? Are the topics too amorphous?

The Survey Process - Instruments Step 4. Develop the Instruments Is it written, on-line web based, verbal or application driven? Target Audience? Conduct "think-aloud" with several people. Carefully select individuals for think-aloud. Consider asking about 10 individuals to write detailed responses on a draft of your questionnaire. Ask some respondents to respond to the questionnaire for an item analysis. For electronic surveys, consider Heuristics for neutral design and navigation. Consider use tracking or cookie based systems for response validation purposes.

The Survey Process - Sampling Step 5. Select a Sample Set Validate the questionnaire prior to full role out. Identify the accessible population. Avoid using samples of convenience. Simple random sampling is a desirable method of sampling. Systematic sampling is an acceptable method of sampling. Stratification may reduce sampling errors. Consider using random cluster sampling when every member of a population belongs to a group.

The Survey Process - Sampling Step 5. Select a Sample Set (cont.) Consider using multistage sampling to select respondents from large populations. Consider the importance of getting precise results when determining sample size. Remember that using a large sample does not compensate for a bias in sampling. Consider sampling non respondents to get information on the nature of a bias. The bias in the mean is the difference of the population means for respondents and non respondents multiplied by the population non-response rate.

The Survey Process – Pilot and Revise Step 6. Pilot Project Using the sample set previously identified, roll out the survey to this group. Validate the results using statistical and other means. Determine if the survey is collecting the correct data and that the information is being accurately presented. Step 7. Post Pilot Project Revisions Based on inaccuracies discovered as a result of the Pilot project, the survey instrument may need to be modified. Make the perceived modifications and repeat Step 6. Note: Steps 6 and 7 may need to occur several times. With each major revision, the project should be run again.

The Survey Process-Research Step 8. Conduct Research There are many aspects of this. Commitment of individuals to participate in the Survey. Advanced notification of the Survey Instructions to the participants. Survey scheduling, processing and collection. Cover letter and questionnaire Design & typing  Purchasing mailing list cost (if necessary)  Addressing mailing envelopes  Following up on non-respondents  Cover letter and survey printing costs  Envelope costs (both ways + more)  Postage costs (both ways + more)  Incentives  Data entry and verification  Statistical analysis software  Distribution of the final report

The Survey Process – Analysis Step 9. Analyze data Prepare a table of frequencies. Consider calculating percentages and arranging them in a table with the frequencies. Use the median as the average for ordinal data. Consider using the mean as the average for equal interval data. Use the median as the average for highly skewed, equal interval data. Use the range very sparingly as the measure of variability. If the median has been selected as the average, use the inter-quartile range as the measure of variability. If the mean has been selected as the average, use the standard deviation as the measure of variability.

The Survey Process – Report Step 10. Prepare Report The final report must contain several conclusions as to the valuation of the intangible elements being examined. Statistical Tables and Figures Averages and Variability Relationships Margins of Error

Class 8a Summary VFM looks at all internal aspects of IT department operations. HCR looks at all external aspects of an IT departments impact on the corporation, Both require IT management buy in and commitment to the process. The results of these types of analysis will identify numerous items that can be “optimized” Range from staffing changes to outsourcing.

Summary Use surveys appropriately. Match benefits with costs Follow a methodology in developing the survey. Remember: "Statistics are no substitute for judgment."  --  Henry Clay

Homework

References: Text Dr. Hossein Arsham http://www.ubmail.ubalt.edu/~harsham/stat-data/opre330Surveys.htm#rvsd Designing Surveys and Questionnaires http://www.statpac.com/surveys/index.htm#toc