1. Build Your Own Forecaster
03/23/11
Build Your Own Forecaster
Immunization Project
Nathan Bunker
03/29/2011

2. Build Your Own Forecaster
03/23/11
Build Your Own Forecaster
Design model
Write software
Populate meta data
Integrate software
Identify test cases
Test forecaster
Make changes
Repeat steps 5 – 7

3. 03/23/11

4. Step 1: Design Model Break vaccines down into antigens
03/23/11
Step 1: Design Model
Break vaccines down into antigens
Break schedule down to dose states that indicate which vaccination dose is due next
Vaccinations given on schedule transition a patient through the states
Measles
MMR
Mumps
Rubella
Protected
Dose 3
Dose 2
Dose 1
Born

5. Initial Design for MMR in 2009
03/23/11
Initial Design for MMR in 2009
MMR
No doses given yet!

6. Initial Design for PCV-7 in 2009
03/23/11
Initial Design for PCV-7 in 2009
PCV-7

7. 03/23/11
MMR

8. Step 2: Write Software Create core forecasting software
03/23/11
Step 2: Write Software
Create core forecasting software
Should support these activities:
forecaster recommendation report
reminder/recall activities
vaccination rate assessment
Forecast should:
Evaluate vaccines given as invalid/valid
Set dates for next vaccine dose due
Identify missed vaccination opportunities

9. 03/23/11
Raw Forecast Output

10. Step 3: Populate Meta Data
03/23/11
Step 3: Populate Meta Data
Establish vaccination expert team
Train expert team on the model
Technical team must be involved
Expert team should consult:
Pink Book
ACIP
CDC (MMWR, NIP Info)
Immunization Action Coalition
Expect a Long Process!

11. 03/23/11
Now Put It All Together

12. Step 4: Integrate Software
03/23/11
Step 4: Integrate Software
Immunization Information System DB
Post
Processing
Core
Forecaster
Meta Data
Test System
Core
Forecaster
Meta Data

13. 03/23/11
Not Done Yet

14. Step 5: Identify Test Cases
03/23/11
Step 5: Identify Test Cases
Assemble test cases
Patient DOB
Vaccination history
Contraindications
Expected results
Considering rotavirus and PCV
Forecaster changes with patient age
Possible date types: relative and specific
Specific dates must "aged" periodically

15. Step 6: Test Forecaster Create testing system that can:
03/23/11
Step 6: Test Forecaster
Create testing system that can:
Create forecast recommendation
Provide forecast result explanation
Compare expected and actual results
Track pass-fail status
Have space for notes and comments
Support regression testing

16. 03/23/11
Forecast Test System

17. 03/23/11
Forecast Test System

18. Step 7: Make Changes Step 8: Lather, Rinse, Repeat
03/23/11
Step 7: Make Changes
Create new test cases as needed
Fix problems found in testing
Most problems fixed in meta data
Some problems fixed in forecaster
Deploy changes in testing system
Verify problems were fixed
Perform regression testing
Deploy changes to IIS
Step 8: Lather, Rinse, Repeat

19. Integrating Forecaster with EHR
03/23/11
Integrating Forecaster with EHR
There is no standard way to integrate so we explored several options:
Write integrated EHR application module
Create integrated EHR application module that supports a web services interface and create forecasting web service engine
Create an HL7 query interface to allow EHR to query for forecast
Create simple web link in EHR, display forecast results in web browser

20. 03/23/11
Lessons Learned

21. 03/23/11
Lessons Learned
Creating a forecasting algorithm itself is relatively simple
Creation and maintenance of meta data is a much bigger effort
The technical guidance from ACIP covers expected scenarios; the forecaster must cover all potential scenarios

22. 03/23/11
Lessons Learned
Consistent process should be developed and applied to deal with gaps in recommendations
Some recommendations were difficult to operationalize

23. Lessons Learned A robust testing system is critical:
03/23/11
Lessons Learned
A robust testing system is critical:
Ensures accuracy of forecaster
Retains shared expert knowledge
Facilitates collaboration
Helps answer questions from users
Helps you sleep at night!

24. 03/23/11
What We Need

25. 03/23/11
What We Need
National expert forecast guidance to translate clinical recommendations into implementable computer algorithms
National forecaster test case repository
More collaboration between other forecaster development projects
Common web service interface for getting forecasts from a forecaster

26. Thank You Expert Team Technical Team Leila Sahni, MPH Laura King, BSN
03/23/11
Thank You
Expert Team
Leila Sahni, MPH
Laura King, BSN
Rachel Cunningham, MPH
Julie Boom, MD
Technical Team
Nathan Bunker, BS

27. Contact Nathan Bunker Immunization Project Texas Children's Hospital
03/23/11
Contact
Nathan Bunker
Immunization Project
Texas Children's Hospital