1. Management of Computer System Performance
Chapter 12
IT Surveys and Data Collection Techniques

2. IT Surveys and Data Collection Techniques
Agenda:
Studies, Surveys and Alternatives
Identifying Intangible Value
How collection can be done
When should data collection be done.
Objective:
Students should be able: to analyze and describe IT surveys into a job sheets. .

3. IT Surveys for benefits measurement
Survey creation is a complex process. A sample survey is provided for reference purposes only.
It is strongly suggested that a great deal of research be done to properly prepare the survey instrument. This will ensure that:
The results received meet the requirements.
That the survey does not skew the resulting data (leading questions, etc.)

4. Identifying Intangible Value
The process of identifying soft or intangible costs or benefits is one of the most difficult in performing a cost or benefit analysis.
It may also be the most important.
The goal is to assign value.
This requires identifying and quantifying something that is often ancillary to hard costs or benefits.

5. On-line banking
On-line banking can be used as an example of this problem.
Some hard benefits and costs are as follows:
Tangible Costs
Tangible Benefits
On-line consumer systems
Lower Labor costs (Fewer Tellers)
Added system security
Increased automatic transactions
Testing
Greater fee generation
Additional IT labor
Fewer/Smaller facilities (lower rent)
Marketing & Advertising
Lower Insurance costs

6. On-line banking
Intangible benefits are initially intuitive or perceived.
These could include:
Intangible Costs Consumer Dissatisfaction with
Intangible Benefits
Consumer Satisfaction with
System/GUI
Not having to go to the bank/No Lines
No personal contact
No one available to ask questions
Simple transactions/One stop shop
Need a fast computer
Easy access
Keeping track of passwords
Better lifestyle “fit”

7. Identifying Intangible Value
The tangible values can be calculated either directly readily estimated on hard values.
Intangible valuation requires additional data gathering.
Measured data will yield the best results.
Considering the alternatives that offer the greatest flexibility, surveys must be addressed as a valid means to acquire measurable data.
Guessing isn’t acceptable.
The corollary is that surveys are not cheap.

8. Data Collection Costs Task Costs – Manual Surveys Labor Costs
Cover letter and questionnaire
design & typing 
Purchasing mailing list cost (if necessary) 
Addressing mailing envelopes 
Cover letter and survey printing costs 
Envelope costs (both ways + more) 
Postage costs (both ways + more) 
Incentives Data entry and verification 
Statistical analysis software 
Distribution of the final report

9. Data Collection Costs Task Costs – Automated Surveys Labor Costs
Planning
Specifications
Applications
Hardware
Questionnaire Design
Notification System
Participant identification
Incentives Data entry and verification 
Statistical analysis software 
Distribution of the final report

10. The Survey Process-Methodology
There is a sequential methodology that can be used to optimize the creation and use of a survey.
Knowing what you want to achieve and planning how to get there are the best way to succeed.
This approach does take time and costs money.

11. The Survey Process – Goals
Step 1: Identify what you want to accomplish
This step identifies the goal you want to achieve. In this case, a valuation of one or more intangible elements of a project.
A goal is not the same as a result.
A goal should be a neutral and not drive towards a predetermined result.
Have a trusted third party check your work. Predeterminations sneak in sometimes.

12. The Survey Process - Planning
Step 1: Planning Questionnaire Research
Consider the advantages and disadvantages of using questionnaires. Prepare written objectives for the research. Have your objectives reviewed by others. Review the literature related to the objectives. Determine the feasibility of administering your questionnaire to the population of interest. Prepare a time-line.

13. The Survey Process - Design
Step 2: Design Methodology
Is it written, on-line web based, verbal or application driven?
Consider the advantages and disadvantages of using questionnaires.
Prepare written objectives for the research.
Have your objectives reviewed by others.
Review the literature related to the objectives.
Determine the feasibility of administering your questionnaire to the population of interest.
Prepare a time-line.

14. The Survey Process - Feasibility
Step 3: Determine the Feasibility
Is it written, on-line web based, verbal or application driven?
Can a survey collect the needed information?
Can it be done in the time provided?
Can it be done with the available budget?
Are the topics too amorphous?

15. The Survey Process - Instruments
Step 4. Develop the Instruments
Is it written, on-line web based, verbal or application driven? Target Audience?
Conduct "think-aloud" with several people.
Carefully select individuals for think-aloud.
Consider asking about 10 individuals to write detailed responses on a draft of your questionnaire.
Ask some respondents to respond to the questionnaire for an item analysis.
For electronic surveys, consider Heuristics for neutral design and navigation.
Consider use tracking or cookie based systems for response validation purposes.

16. The Survey Process - Sampling
Step 5. Select a Sample Set
Validate the questionnaire prior to full role out.
Identify the accessible population.
Avoid using samples of convenience.
Simple random sampling is a desirable method of sampling.
Systematic sampling is an acceptable method of sampling.
Stratification may reduce sampling errors.
Consider using random cluster sampling when every member of a population belongs to a group.

17. The Survey Process - Sampling
Step 5. Select a Sample Set (cont.)
Consider using multistage sampling to select respondents from large populations.
Consider the importance of getting precise results when determining sample size.
Remember that using a large sample does not compensate for a bias in sampling.
Consider sampling non respondents to get information on the nature of a bias.
The bias in the mean is the difference of the population means for respondents and non respondents multiplied by the population non-response rate.

18. The Survey Process – Pilot and Revise
Step 6. Pilot Project
Using the sample set previously identified, roll out the survey to this group.
Validate the results using statistical and other means. Determine if the survey is collecting the correct data and that the information is being accurately presented.
Step 7. Post Pilot Project Revisions
Based on inaccuracies discovered as a result of the Pilot project, the survey instrument may need to be modified.
Make the perceived modifications and repeat Step 6.
Note: Steps 6 and 7 may need to occur several times. With each major revision, the project should be run again.

19. The Survey Process-Research
Step 8. Conduct Research
There are many aspects of this.
Commitment of individuals to participate in the Survey.
Advanced notification of the Survey
Instructions to the participants.
Survey scheduling, processing and collection.
Cover letter and questionnaire
Design & typing 
Purchasing mailing list cost (if necessary) 
Addressing mailing envelopes 
Following up on non-respondents 
Cover letter and survey printing costs 
Envelope costs (both ways + more) 
Postage costs (both ways + more) 
Incentives 
Data entry and verification 
Statistical analysis software 
Distribution of the final report

20. The Survey Process – Analysis
Step 9. Analyze data
Prepare a table of frequencies.
Consider calculating percentages and arranging them in a table with the frequencies.
Use the median as the average for ordinal data.
Consider using the mean as the average for equal interval data.
Use the median as the average for highly skewed, equal interval data.
Use the range very sparingly as the measure of variability.
If the median has been selected as the average, use the inter-quartile range as the measure of variability.
If the mean has been selected as the average, use the standard deviation as the measure of variability.

21. The Survey Process – Report
Step 10. Prepare Report
The final report must contain several conclusions as to the valuation of the intangible elements being examined.
Statistical Tables and Figures
Averages and Variability
Relationships
Margins of Error

22. Class 8a Summary
VFM looks at all internal aspects of IT department operations.
HCR looks at all external aspects of an IT departments impact on the corporation,
Both require IT management buy in and commitment to the process.
The results of these types of analysis will identify numerous items that can be “optimized”
Range from staffing changes to outsourcing.

23. Summary Use surveys appropriately.
Match benefits with costs
Follow a methodology in developing the survey.
Remember:
"Statistics are no substitute for judgment."  --  Henry Clay

24. Homework

25. References:
Text
Dr. Hossein Arsham
Designing Surveys and Questionnaires