1. 2008 PHIN Conference Atlanta, GA August 28 2008
Communication Successes and Challenges: Experience from a Nationwide Pilot Project on Tracking Vaccine Doses Administration Using CDC’s CRA System
2008 PHIN Conference
Atlanta, GA
August
Good morning. I am delighted to be with you today to make this presentation on ‘Communication successes and challenges: Experience from a nationwide pilot project on tracking vaccine doses administered using CDC’s CRA system.’
CRA stands for Countermeasure and Response Administration and briefly it is CDC’s application or system to track medical or non-medical countermeasures.
Sanjeeb Sapkota, MBBS, MPH
SRA International Inc. supporting NCPHI/ CDC
Countermeasure and Response Administration
Division of Emergency Preparedness and Response
National Center for Public Health Informatics

2. Co-authors Jeanne Tropper Sanjeeb Sapkota Charles Williams
Marshall Quin
M.Leigh Bennett-Conner
Let me introduce you to the co-authors.
Ms. Jeanne Tropper, who is seated…………………………
Jeanne is informatics science lead of CRA.
Charles Williams, Marshall Quinn and Leigh Bennett-Conner, all work for Division of Emergency Preparedness and Response in CDC where CRA is located.
Jeanne and colleagues from CRA team will be available to answer questions together with me at the end of the presentation.

3. If we are to succeed, we must communicate. Bob Johnson
The biggest problem with communication is the illusion that it has occurred.
George Bernard Shaw
I am sure that everyone in the room already know and agree that communication is indispensible for the success of any project. These are two words of wisdom on communication. PAUSE

4. Agenda The 2007 Nationwide Pilot Test on Vaccine Doses Administered
Strategic Communication
Study on association of webinar participation in successful participation of the pilot test
Implications for 2008 Doses Administered Exercise
So what I am going to talk today?
I will begin by describing briefly the national pilot test on vaccine doses administered that was conducted in 2007.
Then I will talk in detail how we planned and implemented the communication for the pilot test.
Then I will describe a little study we conducted to look at the association between participating in webinar and being successful for the pilot test.
I will conclude by highlighting how we applied the lessons learned during the 2007 pilot to the 2008 pandemic influenza exercise vaccine doses
Administered.

5. What is CRA?
Established (CDC) application for tracking critical pharmaceuticals and non-pharmaceuticals during a public health emergency or urgency
Predecessor was Pre-event Vaccination System (PVS); it tracked smallpox vaccination doses administered
CRA has provided services to multiple jurisdictions at all levels: city/county, state and federal
Let us learn little bit more about CRA?
As I mentioned, CRA is an established CDC application for tracking critical pharmaceutical and non-pharmaceutical countermeasures during a public health emergency or urgency. The pharmaceutical countermeasure include vaccines, antiviral, antibiotics. Non pharmaceutical countermeasure include isolation and quarantine.
The predecessor of CRA was PVS system, OR Pre-event vaccination system. This system was used to track smallpox vaccine doses administered in 2003 to military personnel.
CRA has provided services to jurisdictions at all levels: city health department, county health department, state health department and have been utilized in activities at federal levels.

6. Learning Objectives
Recognize the process involved in planning and implementing communication strategies for a nationwide pilot
Realize the association of participation in webinars and submitting requested information/ data
Recognize overall communication lessons learned from the pilot test
So the take home messages of the presentation will be.
By the end of this presentation you will be able to recognize the process involved in planning and implementing communication strategies for a nationwide pilot project.
You will be able to recognize if participating in webinar have any association in submitting the information and data that is requested and thus being successful in the pilot test.
You will recognize lessons in communications we learned from the pilot test.

7. 2007 Nationwide Pilot Test on Vaccine Doses Administered (November 1 – December 31, 2007)
So let us get started with the pilot test on vaccine doses administered that was conducted across the country during the November and December of last year.

8. Pilot Test: Purpose
To test CDC and states’ capability to monitor vaccine doses administered during pandemic influenza using seasonal influenza vaccine as proxy
The purpose of the pilot was to test whether or not CDC was able to monitor the utilization or administration of vaccine doses during a pandemic influenza event AND whether or not states were able to collect, and transmit aggregate data on vaccine doses administered during pandemic influenza event.
Now keep in mind that we do not yet have the vaccination for the pandemic. So what we did was we used regular seasonal influenza vaccine doses as proxy for pandemic influenza vaccine. 8

9. Phase I: Pre-Pilot Planning
Pilot Test: Timeline
Phase I: Pre-Pilot Planning
Apr-Oct 2007
Phase II: Pilot Test
Nov-Dec 2007
Phase III: Post-Pilot
Jan-Mar 2008
The pilot test timeline was divided into three phases:
Phase I, during which the planning and preparation took place, from April to October, 2007
Phase II, during which the actual implementation of pilot occurred between November and December of 2007.
Phase III, the post pilot period, during which feedback, lessons learned and results were shared among participants, stakeholders, partners.

10. Information Requested from States
Identify Point of Contact (POC)
Select an option to submit data
Identify at least two clinics for the pilot
Submit data from these two clinics within 48 hours of the conclusion of clinic/ clinic sessions
Now let me briefly tell you what we asked from the states during the pilot project.
We asked them to identify the point of contact
Then we asked them to select an option to submit or transmit data to CDC/ CRA
We asked them to identify at least two clinics from where the vaccine doses administered will be collected and send to CDC via CRA system
And more importantly, we asked states to submit vaccine doses administered within 48 hours of the conclusion of clinic/ or clinic sessions.
To be fully successful in the pilot each state has to meet this requirement: that is submit data from at least two clinics within 48 hours of clinic or clinic session.

11. Pilot Test: Three Options to Transmit Data to CDC
Pipe-delimited File
XML File
Option 1
Data exchange
CRA
HL 7
CRA
Option 2
Direct web entry
Let me, very quickly, give you an idea of what are the different option states could use to submit data to CDC using CRA system.
CLICK:
Option 1 is data exchange. States could use their existing Immunization information system or Immunization registry or any equivalent system to send data to CDC using one of the three content format: pipe-delimited, XML, HL7.
Option 2 is direct data entry into CRA application. This is the screenshot of CRA application. Data that have been aggregated before is put into CRA web-based application and once the data enterer hits submit it gets submitted to CDC into CRA system.
Option 3 or Individual level data entry is similar to option 2 in that same CRA application is utilized to enter and submit data. The difference though in option 3 individual level information could be entered into the system.
Option 3
Individual level data entry
CRA

12. State/ Jurisdiction Option Choices
Data Exchange
Chicago
American
Samoa
Web Entry aggregate
NY City
Marshall
Islands
Web Entry Detail
Guam DC
Mariana
Islands
LA county
Virgin
Islands
This slide show the different options that states notified us that they were going to use to transmit data to CDC via CRA system.
Now I have been saying states but its not only the states who participated.
In fact 62 different jurisdictions participated and they cover the entirety of United States. This include 50 states, eight US territories and islands (POINTER to ISLANDS and TERRITORIES) and four major metropolitan area that include (POINTER) Los Angeles county, Chicago (POINTER), New York City (POINTER) and District of Columbia.
So keep in mind that when I say states it represent all the 62 different jurisdictions across the United States.
Palau
Puerto Rico FS
Micronesia 12

13. The pilot was successful with 89% of states sending some data.
The pilot test was highly successful. Eighty nine percent of states send at least some data and over 50% of states become successful in meeting the requirements of pilot test that I mentioned earlier.
This is a picture from pilot clinic from the state of kansas. In the picture you can see Kansas’s mascot for immunization with who promotes the slogan ‘Bee Wise - Immunize’
Pilot Clinic, Kansas
Kansas immunization Mascot: ‘Bee wise – Immunize’

14. Strategic Communication for the Nationwide Pilot Test
Now that we have put this presentation in the perspective of pilot test, let us go to the topic of presentation: which is communication.
We are calling it strategic communication. Just not to confuse you, by strategic we mean systematic approach to communication.

15. Strategic Communication
Define objectives/goals
Develop communication plan
Indentify point person for each state
Indentify option choice
Ensure each state submit data to CDC
Preparation of communication-materials
Use various channels of communications
Primary channels: Webinar, Conference calls
Secondary channels: s, one-on-one telephone call
Kept track of number of webinars each state attended/ did not attend
PAUSE…I am going to spend some time on this slide.
So what are the different components of the strategic communication we undertook for the pilot test ?
CLICK:
First and foremost we defined the objectives and goals of the communication. Obviously the goal was to gather necessary information: such as point person in each state, option choice, clinic date and finally data within 48 hours of clinic session.
Then we developed communication plan that involve how to identify point person for each state, how to educate states on different option choice and facilitate them select one that best fit their existing infrastructure and resources, how to ensure that each of them submit data to CDC within the 48 hours.
CLICK
Strategic communication also involves preparation of communication-materials. Such as to develop content of webinars, frequently asked questions, modify and update website, preparing training materials both general and the option-specific, preparing checklists and guides.
Next step or next parallel step is to chose various channels of communication to reach out to to the identified point persons who would then share information within their respective state and jurisdictions.
We primarily communicated via webinars. And I assume everyone in the room know what webinar is. But I won’t be surprised if someone does not.
PAUSE, SLOW…Webinar is presentation via a website. So that if you are sitting in Alaska you will be able to view the presentation being conducted real time in Atlanta and participate in the presentation as if you were sitting in the same room in Atlanta.
Conference call was our another primary channel of communication we used. In the conference call many states could listen and participate the audio presentation and discussion.
We conducted seven webinars and one conference while preparing the states for the pilot test.
Beside these mass channel of communications, we also used phone and as channels for one-on-one communication.
Then we track the number state’s point person who attended or did not attend each webinar, conference calls

16. Study on Association of Participating in Webinar and being Successful in the Pilot Test
Now let me go to the next phase of the presentation in which I will describe a little study we did to see if there is any association between the state’s participation in webinar and being successful in the pilot test

17. Study Question
Are states who participate in webinars more likely to send the requested information/ data?
The question of this study was ‘Are states who participate in webinar more likely to send the requested information or data and being successful in the
pilot project ?

18. Study Rationale
The association of participation in webinars and submitting requested information/ data has not been adequately studied.
Why we did this study ?
We really wanted to know if participating in webinars has any influence on submitting the requested information or data. Also we found, with a little research, that these association have not been adequately studied. So this study would help throw light on the association.

19. Methodology
States were invited to participate in seven webinars and one conference call conducted during the pre-pilot phase (April – October 2007)
Number of webinars/conference calls that each state participated was compared with the number of states who submitted requested information/ data
This slide shows the methods of the study:
First the invitation was sent out to all the 62 point of contacts for each of the webinar and conference call held. A total of seven webinar and one conference call was held.
SLOW..We then looked at the number of states who participated in two or more webinars or conference call and examined whether or not they sent the requested information.

20. Webinar Topics and Attendance
Webinar/ Date
(in 2007)
Topics
No. of Participants
May 09
General Aggregate Reporting 99
May 30
(conf call)
Question Answer Session 80
June 06
Option 1 Aggregate Reporting
110
June 13
Option 2 & 3 Aggregate Reporting 65
June 27
Planning on Pilot Test
121
Aug 08
Status of Pilot Test 46
Oct 10
Update for Option 1 Users 38
Oct 17
Update for Option 2 & 3 Users 33
No. of States Represented 32 39 38 33 45 36 19 22
PAUSE
This slide shows the topics of the webinars during the pre-pilot period, PAUSE, the dates they were held, number of participants and ,PAUSE, the number of states that were represented.
Topics include general information on aggregate data reporting both general aggregate reporting and specific aggregate reporting, question answer sessions,
Planning on pilot test and status of the pilot.
You may notice that there was no webinar in July and in September.
KEEP THE SLIDE FOR 3 SECONDS AFTER SPEAKING

21. Results (n=62 Participants)
Submitted option choice by deadline
Did not submit by deadline
Participated in webinar*
41 (66%) 15
Did not participate in webinar
0 (0)
6 (10%) 56 6
Now let us look at the results.
The results are shown in this two by two table. (POINTER) The rows show the exposure, if you will, and that is whether or not state participated in two or more webinar. The column (POINTER) show whether or not state submitted required information, in this case the information requested was option choice by deadline.
(BE AWARE OF QUESTION ‘DID THOSE 15 WHO DID NOT SUBMIT DATA BY DEADLINE ULTIMATELY SEND INFO. THE ANSWER IS ‘ALL STATE SEND THAT INFO, BUT ONLY 66.1% SEND ON TIME)
PAUSE
The number represent the state or jurisdiction, and there were 62 state or jurisdiction in total.
NOW Let us look at this number 6 (POINTER). Of the six state or 10% of state who did not submit an option choice by the deadline, also did not participate in any webinars or conference calls. (SHOW THE POINTER)
In comparison, 41 states or 66 percent of states who did submit option choices by the deadline also participated in two of more of the webinars or conference calls.
The p value show that this association is statistically significant at 95% confidence interval.
What does this mean?
It means that there is a significant association that states who did participate in webinar were significantly likely to submit option choice on time than those who did not participate in webinar or conference call.
Also participation in webinar implies participating in two or more webinar. 21 41
* two or more webinars/conference calls
p <

22. Results (n=62 Participants) (Cont.)
Submitted
clinic date
Did not submit clinic date
Participated in webinar 53
Did not participate in Webinar 3 6
53/ 53 = 100% 53 9
PAUSE:
In this slide, rows again shows participation in webinar (POINTER) but the column show whether or not state submitted clinic date when the pilot was going
To take place.
PAUSE and CLICK:
Over 65% OR 6 out of 9 states who did not participate in any webinars or conference calls also did not submit clinic dates.
In comparison, 100% OR 53 out of 53 states who submitted clinic data also participated in one or more webinar or conference calls.
Again the p value shows that this is highly significant association at 95% confidence interval.
6/ 9 = 65% 56 6
p <

23. Results (n=62 Participants) (Cont.)
Submitted
some data
Did not submit data
Participated in webinar 55
Did not participate in webinar 1 6
55/ 55 =100% 55 7
PAUSE.
This slide shows how many states eventually did not OR did send at least some data.
PAUSE:
Here again, the row is participation in webinar (POINTER) but the column shows states who submitted some data.
PAUSE, CLICK:
During the event phase of the pilot (Nov 01 - Dec 31, 2007), 85% (6/7) of the
states who did not submit any data also did not participate in any webinars or conference
calls.
PAUSE and CLICK:
In comparison, 100% of states or 55 out of 55 states who submitted some data also
participated in one or more of webinars or conference calls.
P value shows that this is a significant association. 56 6
6/ 7 = 85%
p value: <

24. Study Conclusion
States who participated in webinars are more likely to send requested information and be successful in pilot participation.
So the conclusion of the study is:
States who participated in webinars are more likely to send requested information or data and be successful in pilot participation.
PAUSE
Now keep in mind that we do not say there is a causal effect between webinar participation and being successful. What this is showing is just a simple association.

25. Study Strengths and Limitations
National level data
Limitations
We did not systematically examine all reasons of state’s inability to submit information/ data (e.g. technology issues, limited staff, competing priorities)
We also did not examine all factors of state’s ability to submit information/ data
The strength of this study is that we had participation data from the entire country.
However there are several limitations of this study.
We did not systematically examine all reason why a state or a jurisdiction was unable to submit the information or data we requested. There could be a whole host of reasons, including issues related to technology, resource constraints in terms of staff or other, competing priorities and even lack of motivation.
Likewise we did not examine all factors why states were able to submit the information and data we requested. Attending to webinar could be one reason, but there could be other contributing factors associated which this study did not examine.

26. Overall Communication Lessons Learned
Regular communication with states is necessary
Factors for states not to respond could be many and should be examined individually and in detail
Do not give up; Open all channels of communication
Increase frequency of communication
Now let us look at the lessons learned from communication perspective during the planning and implementing pilot test.
It is very important to be in regular communications, particularly during the planning phase, even if there is no update to share.
PAUSE..Regular communication is reassuring to states that the project is viable and that we are serious in what we are saying and doing.
PAUSE..It also help them and us identify emerging challenges and address them in timely fashion. We were in touch with states via webinar or conference call at least once a month during the pre-pilot phase except for the month of July and September. This raised concern among some state that they were wondering if anything went wrong with the project. But once we resumed webinars states, were again excited about the project. PAUSE:
Second bullet on lesson learned. PAUSE:
There could an array of reasons for states not to respond to our request. It could be technical issues, operational, communication, time or resource constraint.
PAUSE: The reasons for states in the mid-west United states could be different than from those in the south-east, the reasons for the pacific island jurisdiction could be different than from continental US states. PAUSE, SLOW
All these need to be examined and teased out diligently on one-on-one basis if we are to attain 100% participation.
One of the other lessons we learned is that we should not give up easily until the information and data is obtained. If one communication channel does not work then another channel should be sought. If the designated point person does not respond your or phone then try contacting his or her officemate or even his or her supervisor and explain the situation.
And finally, as our study shows, increase the frequency of communication profoundly assist in obtaining required information.
Now how are we going to apply the lessons learned? PAUSE. We are going apply this in the follow up exercise this year.
CLICK NEXT SLIDE

27. Implications for 2008 Doses Administered Exercise (DAX), Oct 01 – Dec 31, 2008
Reconfirmed the Point of Contacts
Increased frequency of webinars/conference calls
Reach out at personal level: at Doses Administered & Safety Conference, Aug 21-22, 2008
Increasing one-on-one conference calls with individual states
Communicating with smaller sub-groups beside webinars
As mentioned we are going to hold a similar but much extensive follow up exercise this year from October thru December.
The lessons learned from the pilot test is already being implemented for the upcoming exercise in 2008.
We have completed reconfirming points of contacts for the exercise.
We have increased the frequency of webinar/ conference call. We have had more webinars at frequent intervals this year.
We have not missed opportunity to reach out to CRA point person at personal level and one such opportunity was during the doses administered/ safety conference that was held last week where most of the CRA point person from across the United States attended.
We have also increased one-on-one conference calls with individual jurisdictions whenever issue arises or even to answer simple questions. States have appreciated this and have found reassuring.
To better address issues of a particular region we are conducting conference call among a smaller group of states. For example we are conducting series of conference calls with US island jurisdictions.
With all these communication strategies in place we hope to have an outstanding participation during the 2008 DAX exercise.

28. Acknowledgement
CRA point persons (POC) in states, major metropolitan areas, US island & territories
Colleagues from:
Immunization Services Division, CDC
Division of State and Local Readiness, CDC
National Center of Public Health Informatics, CDC
Northrop Grumman Information Technology
BearingPoint
SRA International Inc.
I would like to acknowledge colleagues listed in this slide.
Firstly, the CRA point persons in states, major metropolitan areas, US island and territories.
Colleagues from CDC from immunization services division, division of state and local readiness and Informatics center.
Colleagues who are contractors and working for CDC from Northrop Grumman, BearingPoint and SRA International.

29. Further information CRA website: www. cdc
Further information CRA website: For questions/ comments: Phone: or Thank you for your attention!
This slide has my contact information and the website of CRA.

30. Thank you for your attention!
Questions ?
The findings and conclusions in this presentation are those of the author(s) and do not necessarily represent the official positions of the Centers for Disease Control and Prevention.