1. Michael L Washington, PhD Deputy Director, Preparedness Modeling Unit Industrial & Systems Engineer Centers for Disease Control and Prevention August 2009 Evaluating the Capacity, Efficiency, and Cost of a Mass Influenza/ Pneumococcal Vaccination Clinic Via Simulation DIMACS/MBI US - African BioMathematics Initiative: Workshop on Economic Epidemiology, 3 Aug 2009

2. Engineering Jokes Engineers aren’t boring people, we just get excited over boring things. Half Full or Half Empty? To the optimist, the glass is half full. To the pessimist, the glass is half empty. To the engineer, the glass is twice as big as it needs to be.

3. Agenda Measurements to evaluate model or actual clinic (including efficiency) Estimating the capacity of a mass vaccination clinic Issues with the optimized clinic

4. Measurements How do you evaluate a good and efficient clinic? First, let us evaluate a mock clinic and learn about a technique to improve the efficiency of a clinic.

5. Line Balance 8 hours or 480 minutes Arrival Rate = 1 person per minute 1 person per station FYI. Does not reach a steady state for 1 to 2 hours and queuing theory does not work.

6. Which Set-up is Best? And what measure to use?

7. Some Measures of Performance Number of people served # of people served or treated over time (Throughput) Average time people are waiting or in the clinic Average # of people in clinic or station (WIP) Resource utilization – people and equipment Cost Customer satisfaction Effort per person (i.e., $/person or CE ratio)

8. Measures MeasurementProsCons Number of people served Simple - count people Familiar No measure of efficiency No customer satisfaction measure ThroughputSimple - people/hr Familiar No measure of efficiency No customer satisfaction measure (a Honda plan - 36 cars/hr) Exit rateSimple – ID bottleneck Helps with things down the line No measure of efficiency Can be miss leading

9. Measures MeasureProCon Clients time in the clinic Measure of efficiency and customer satisfaction Measure variability If measured per stations, measure training Easy way to ID bottleneck Additional data required Difficult to collect per station No standards Not = throughput Ave # of people in clinic (or station) Measures of efficiency If measured per station, can identify bottleneck Can easily measure for the entire clinic May have to be done electronically Difficult to measure per station manually.

10. Measures MeasureProCon Resource utilization Measures efficiency ID over- or understaffing Can be calculated post- event if the right data are collected Does not provide information about one main goal CostMeasure of resources needed to accomplish an objective Compare across clinics It takes time to consider all costs Some consider it unimportant in an emergency (which I think is a mistake)

11. Measures MeasureProCon Customer satisfac- tion Being able to measure how the people you serve feel about your service is always a great idea. Surveying clients is trick, and clients can be very fickle. $/personProbably the best Measure the effort involved in treating one person Easy to get cost if documentation is good May be difficult to collect cost, especially indirect cost No standard Don’t worry about $ in emergencies (mistake)

13. Results (100 Runs) Mean (min, max) Measurement123 Number of people served 454 (411, 510) 454 (409, 494) 192 (175, 213) Throughput (client/hr) 57 (51, 64) 57 (51, 62) 24 (22, 27) Exit rate112.5

14. Results (100 Runs) Mean (min, max) Measurement123 Clients time in the clinic (min) 19 (7, 57) 19 (9, 45) 145 (122, 172) Average # of people in the clinic (people) 26 (0, 96) 28 (2, 81) 287 (219, 342) Resource utilization (%) 48 (40, 56) 80 (71, 88) 100 (99, 100)

15. Results (100 Runs) Mean (min, max) Measurement123 Direct Cost ($28/hr/staff) $1,120$672$224 Indirect Cost ($18/hr/client) $155,372 ($52,478, $523,967) $160,795 ($64,059, $396,121) $502,961 ($387,009, 659,256) Total Cost$156,492 ($53,598, $525,089) $161,467 ($64,722, $396,792) $503,186 ($387,233, $659,480)

16. Results (100 Runs) Mean Measurement (easiest) 123 Direct Cost/client 2.471.481.17 Indirect Cost/client 3433542,626 Total Cost/client 3453562,627

17. Agenda Measurements to evaluate model or actual clinic (including efficiency) Estimating the capacity of a mass vaccination clinic Issues with the optimized clinic

18. Simulation Model Evaluate one clinic design The main measure is the # of people vaccinated We will examine other measures in evaluating the clinic Try to improve the clinic design based upon the main measure

19. Optimizing a Clinic History –Anticipated vaccination 15,000 in 17 hours –Only 8,300 showed up and were vaccinated –Could they have vaccinated 15,000 with current design and staff Simulation (very difficult with queuing) –Arrival rate was not consistent –Violate a big rule in queuing (service rate > arrival rate) –Looked to optimize staff placement

20. Clients “Medicaid” – Assumed to be retired, > 65 years old “Special” –A sub-population of “Medicaid” –Usually the physically challenged “Cash” – Normal work force

22. Staff Special Copy Special Flu Form Special Flu Vac Pnu Form Yellow = RN White = Support staff Contractor/Volunteer not included (EMT, security, runners, etc) Pnu Vac Medicaid Copy Medicaid Flu Form Reg Flu Vac Cashier

23. Inputs Service Time at Each Station Station# of Staff a Type of StaffDistribution b (SD) (minutes) Sourcen Special Flu Copy1APBeta0.40 (0.813)EO- Special Flu Registration2APGamma2.63 (0.930)TS102 Special Flu Vaccination2RNLognormal1.13 (0.590)TS158 Pnu Registration2AP Medicare ClientLognormal1.01 (0.416)TS c Special ClientGamma2.18 (3.140)TS c Pnu Vaccination4RN Medicare ClientLognormal1.18 (0.561)TS c Special ClientLognormal1.13 (0.590)TS c Medicare Copy4APBeta0.40 (0.813)EO- Medicare Registration18APLognormal1.01 (0.416)TS402 Medicare/Cash Flu Vaccination20RNLognormal1.18 (0.561)TS895 Cashier3APBeta0.43 (0.084)EO-

24. Facility Costs Printing Vaccine Copiers T-Shirts Signs/Banners Food Advertising Tables/Chair Rental Rent Supplies Medical Billing Sharps Removal Refrigerator Rental

25. Some Cost Salary –Nurses –Support Staff Usage – Every client used Vaccine, printing, copies, supplies, etc…

26. Inputs SuppliesTypeDistributionMinimum ($)Maximum ($) Influenza Vaccine a Uniform9.7119.94 Pneumococcal Vaccine a Uniform14.2424.19 Copying/Printing Cost b 0.10 General Supplies b Varies form $0.20 to $0.25 per station StaffTypeDistributionSalary (SD) - $/hr Registered Nurses c Lognormal23.1 (4.5) Administrative Personnel c Lognormal13.3 (3) ClientsTypeDistributionSalary (SD) - $/hr Special d Lognormal21.5 (79.5) Medicare d Lognormal21.5 (79.5) Cash c Lognormal18.1 (15) Why worry about Client salary?

27. Optimizing a Clinic Maximize: # of people vaccinated ST: Support staff <= 30 RNs <= 26 Staffing per station >= 1 Only RNs can vaccinate Only support staff provide support function

28. Tool Arena 11.0 Discrete-event computer simulation OptQuest 11.0 –Neural network –Scatter search (use infeasible values) –Tabu search (cannot use previous values)

30. Original Model Optimized Model Arrival Intensity (%)804080140 Max Client Vaccinated13,13813,03914,81715,096 Special Flu Vaccination2111 Pnu Vaccination4222 Medicare Registration1815 Medicare/ Cash Flu Vaccination2023 Cashier3444 Others* 9 Total Staff5654 * Special Flu Copy (1), Special Flu Registration (2), Pnu Registration(2), Medicare Copy (4) Arrival intensity is the % increase in arrival rate above the original arrival rate.

31. Number Vaccinated

32. Time in Clinic Medicare follows path similar to Special. Throughput rate Original – 770 clients/hr Optimal – 880 clients/hr

33. Time in Clinic Maximum Throughput Original – 770 clients/hr Optimal – 880 clients/hr

34. Cost

35. Other Cost Amount Contract staff$2,882 Copiers600 T-Shirts1,043 Signs/Banners350 Food997 Ads1,150 Tables/chairs391 Rent2,000 Medical billing5,284 Sharp removal375 Refrigerator Rental80 Total$15,152

36. CE Ratio (Opt Model)

37. Agenda Measurements to evaluate model or actual clinic (including efficiency) Estimating the capacity of a mass vaccination clinic Issues with the optimized clinic

38. Issues with Optimization Targeted group with: –Little processing times –Few stations to visit –Larger numbers Alternative objective functions and constraints could have limited this disparity at the expense of efficiency

39. Issues with Optimization Obj function (instead of max # vacc) –Max revenue – focus on one group of clients (who pays the most) –Min cost – vaccinate no one –Max profit – we are the government –Max societal benefit minus cost – programming dependent (societal perspective) –Min time in clinic (client’s perspective)

40. Issues with Optimization Constraints Limit the optimization to where no one spends more than a specific amount of time in the clinic; however, this also decreases efficiency if you want to maximize the # vacc

41. Issues with Optimization Result (from the optimized model) –Elderly suffer: small number and slow Still good to separate the elderly from others –High resource utilization if want 15,000 More staff are needed and capacity (safety) issue –Planners did a good job in designing the clinic

42. Modeling Resources We must do this –Do we have enough –How much training is need –Where are they and how to allocate them Clinics and hospitals Logistics, distribution, transportation Appropriate supplies What is the right design or process

43. Thank You

44. Opportunities PE Fellowship EIS ORISE Other fellowship programs –Presidential Management Intern –Informatics –Many more

45. PE Fellowship Across the CDC, 2 Year program Deadline Feb 1 Ph.D. (econ, decision sciences, HSR, IE, OR, or related fields) Accept non-US Citizens Pay GS12 (2009), step 1 in ATL $70,399 US ($83,714 after graduation) Have a publication list