1. An Independent Evaluation of the OK-FIRST Decision Support System Thomas E. James and Paula O. Long Institute for Public Affairs / Department of Political Science University of Oklahoma Mark Shafer, Oklahoma Climatological Survey University of Oklahoma

2. OK-FIRST Customized, county-level environmental information Target group: emergency managers Training (computer workshop, data interpretation workshop) 3 classes (June 1997, October 1997, March 1998) Refresher courses with Focus Groups

3. Independent Evaluator Unbiased and objective perspective A collaborator, NOT a monitor Combines strengths: –Institute of Public Affairs: methodological expertise –OK-FIRST staff: substantive context-specific knowledge Ensured collection of information useful to the program Continuous feedback to improve training methods or data collection techniques

4. EVALUATION “The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.” (Michael Patton 1997)

5. EVALUATION GOALS (1)Determine the extent to which OK-FIRST provided access and training to enhance participants abilities (2)Document the use of skills and impacts (3)Identify lessons learned

6. EVALUATION DESIGN Background Information –self-administered questionnaires –job experience –sources of information –familiarity with computers

7. EVALUATION DESIGN Workshops –pre-test / post-test –computer workshop: observations of performed tasks coupled with written identification –data interpretation workshop: identification, definitions, application of concepts (written)

8. EVALUATION DESIGN Self-Assessment –questionnaire administered at the beginning of the data interpretation workshop, covering computer skills –questionnaire at refresher workshops on computer skills and use of information –group discussions focusing on insights and experiences

9. COMPUTER TRAINING Statistically significant increase in skills –57% average pre-test score; 79% post-test Reduced range of post-test scores (those initially at bottom benefited most) After receiving training, almost all participants proficient enough to access data Despite significant increases in some areas, still room for improvement

10. DATA INTERPRETATION Significant increase in knowledge –44% average pre-test score; 62% post-test Reduced range in post-test scores Assessing pre-test and post-test knowledge (clusters) allows targeted training Feedback on performance from first class allowed OK-FIRST staff to adjust training, resulting in improved performance of other classes

11. WEB PAGES Follow-up questionnaire to determine utility 100% satisfied overall –98% very satisfied 91% very satisfied with content NIDS (WSR-88D) most frequently accessed –61% use daily 71% said very easy to navigate

12. BULLETIN BOARD Designed to share information and seek assistance Not highly used –37% did not use at all, rest infrequently Of those who used it, majority said posted information was useful Integration with web access may make it more useful

13. MEASURING OUTCOMES Long-term Outcomes –broad goals, which program itself cannot accomplish alone –e.g., saving lives or property Intermediate Outcomes –measurable aspects which facilitate accomplishment of long-term outcomes –e.g., application of data, knowledge, and skills Primarily measured from focus groups

14. UTILIZATION Overall –“I’m proactive now rather than just reactive” –“This program has become critical to our organization” Access / Application of Data –“OK-FIRST information … has kept disasters from happening” –“This allows the emergency manager to check his own specific area”

15. UTILIZATION Severe Weather –most frequent application –identify threats earlier –pinpoint storms more accurately –utilize scarce resources more effectively Floods –better management of road closures –fewer false alarms

16. UTILIZATION Fires –identifying approaching wind shifts –Fire Danger Model Hazardous Materials –trajectories / evacuation decisions Public Works (unanticipated outcomes) –assessing need for snow crews –appropriate conditions for road work –little league baseball

17. CONCLUSIONS The OK-FIRST team has been successful in meeting the key needs of program participants Able to enhance significantly the knowledge and skills of the project participants in a very short period of time Integration and coordination of training, access, and ongoing support vital to success

18. CONCLUSIONS “The project was able to change the behavior of local public safety officials and their approach to decision making.” Increased participants confidence, abilities, effectiveness, and range of application Empowered local officials to make decisions “The OK-FIRST organization and team should serve as a model for others.”