1. Thar’s Gold in Them Thar AAR/IPs

2. Welcome and Agenda Purpose and Value of After Action Reports
How to develop an Improvement Plan
Esther Corwin, MEP
Exercise Training Officer, Texas Division of Emergency Management
Contact Info is on handout

3. What is an AAR
A consolidation of information gathered during the testing and evaluation of a community’s emergency operations plan through an exercise
Provides feedback to executive leadership, participating entities and governing agencies in the achievement of the exercise objectives and overall capabilities of the community
Information gleaned from this process, identifies and guides future improvement actions

4. Why Write an AAR
Creates a multi-agency process to review an exercise or incident
General exercise identification information, but also includes evaluation information
Facilitates honest and critical appraisal of actions based on plans and policies

5. Where Do We Look OR Exercises to simulate a stressful realistic scene
Exercises to evaluate a new or seldom used equipment or skill
Incidents presenting unexpected challenges
Incidents having “less than expected” results OR
Incidents employing innovative actions
Incidents with unexpected positive results

6. Exercise Evaluation is Based On
Community policies, Emergency Operations Plan, agency SOPs and SOGs, MOUs, etc.
Exercise Evaluation Methodology
S.MA.R.T. objectives
Exercise Evaluation Guides
Observational Data
Hotwash
Participant Feedback

7. Incident Evaluation is Based On
Community policies, Emergency Operations Plan, agency SOPs and SOGs, MOUs, etc.
Incident Action Plans
Briefings, De-Briefings
Observational Data
Participant Feedback

8. Types of Observational Data
Descriptive reporting
Typically yields reliable data.
Inferential reporting
Includes terms like “adequate” or “timely”
Yields inconsistent data
Evaluative reporting
Includes terms like “efficient” or “successful”
Difficult to reliably to collect

9. Three Levels of Analysis

10. Three Levels of Analysis
Capability Being Evaluated
Critical Resource Logistics & Distribution
Onsite Incident Command
Level of Evaluation
Capability-Level Analysis
Resource Management – Identify, dispatch, mobilize, track, demobilize and pay for resources
Direct/Control incident management activities.
Activity-Level Analysis
Identify the resources based on incident need
Develop all necessary components of an IAP and obtain approvals
Task-Level Analysis
Inventory resources by type/ category that are available to support this incident
Establish incident objectives and priorities and operational periods

11. Exercise Evaluation Guides
Identify the activities, tasks, and performance measures to be observed
Developed and customized before the exercise
Evaluators complete the EEG by:
Logging times and actions accurately
Decision making processes and participants
Player roles and responsibilities
Coordination and cooperation
How actions were performed, resources involved
Documenting issues and recommendations

12. Data Analysis
The goal is to evaluate the ability of involved functions to perform target capabilities

13. Preliminary Analysis
Evaluators organize observations into key issues and a chronological narrative
At a minimum, should include:
Description of the assigned function or operation, analyzed by capability, activity, and task
Documented record of significant observed actions

14. Review Objectives Review exercise objectives
What was the intent of the objective?
What would demonstrate the successful performance of the objective?
If the objective was not met, what factors contributed to this result?
Review incident operational objectives
Review exercise objectives
What was the intent of the objective?
What would demonstrate the successful performance of the objective?
If the objective was not met, what factors contributed to this result?
Review incident operational objectives

15. 4 Steps of Data Analysis Identify issues Determine root cause
Compare performance data to standards
Identify differences
Determine consequences
Determine root cause
Discuss conditions leading to observed behavior
The “Why” staircase
Fool’s Gold

16. Why Staircase Each step should explain the step above …
Why did it happen?
Each step should explain the step above …
Why did that happen?
Why was that?
And why was that?
And why was that?
ROOT CAUSE
… down to the underlying root cause.

17. 4 Steps of Data Analysis Develop recommendations
Sustain and improve
Short and long term
What needs to be changed and how
Give references
Capture lessons learned
Innovative practice or a piece of knowledge gained from experience
Provides guidance for approaching a similar problem in the future
Save time, conserve money, and accelerate preparedness

18. Draft AAR
Provides feedback to participating entities and governing agencies regarding the achievement of objectives and overall capabilities
Records recommendations for improvement
Establish consensus and buy-in on next steps

19. The AAR Suggested AAR format includes: Executive Summary Overview
Exercise Goals and Objectives
Analysis of Capabilities Demonstrated
Conclusion

20. Observations and Recommendations
References
Analysis
For improvements, list consequences of action or inaction
Recommendations
Observations
References
Analysis
For improvements, list consequences of action or inaction
Recommendations
Observations
References
Analysis
For improvements, list consequences of action or inaction
Recommendations
Observations
References
Analysis
For improvements, list consequences of action or inaction
Recommendations

21. After Action Conference
Planning Team, evaluators, and stakeholders meet to review and refine draft AAR
Evaluation leads to a disciplined process for implementing improvement actions and continually strengthening preparedness

22. After Action Conference
Address identified issues
Develop specific improvement actions
Prioritizes action items
Assigns responsibility to track implementation

23. Developing Improvement Actions
These questions aid development:
What changes need to be made to plans and procedures to improve performance?
What changes need to be made to organizational structures?
What changes need to be made to leadership and management processes? ?

24. Developing Improvement Actions
More questions to aid development:
What training is needed?
What changes to, or additional equipment are needed?
What lessons can be learned that will direct how to approach a similar problem in the future?

25. Improvement Benchmarks
Must be clearly defined and attainable
Examples include:
Number of personnel trained in a task
Percentage of equipment that is up-to-date
Finalization of an interagency agreement within a given amount of time
Include concrete deadlines to track progress toward full implementation

26. Activity – Improvement Development
Older elementary school in a rural area just before dismissal on a spring day, parents are arriving to take students home
NWS issues tornado warning estimating less than 10 minutes before impact
Principal activates “bell” for sheltering, but students/parents confuse it with dismissal
Some parents enter school to take child and “run” from the storm
As a large group, given a scenario and issue, discuss possible recommendations and develop a simple improvement action.

27. Activity – Develop Improvement
Issue:
Student Safety
Basic Recommendation:
Develop something to assist the principle in communicating emergency information
Changes to plans and procedures
Changes organizational structures
Changes to leadership/ management processes
What training is needed
Changes or additional equipment
Lessons learned for the future
Jarrell Tornado May 27, 1997

28. Improvements make things better than before
Summary
Objectives and Evaluation Methodology are the foundation of Improvement Planning
Root Cause analysis helps find the key policy, procedure, training or equipment to which an improvement could be made
Applies to Exercises and actual Incidents
Improvements make things better than before

29. Any question can lead to a good idea. Synergy is the Energy of People.
Questions ???
Any question can lead to a good idea.
Synergy is the Energy of People.
Esther Corwin, MEP
Exercise Training Officer, TDEM