1. 1 Performance Management Presentation Maintain Safe Working Environment Radiation Safety Team Leader: Nancy Newman Team Members: Kelly Austin, George Redmond ORS National Institutes of Health Date: January 21, 2004

2. 2

3. 3 Customer Perspective

4. 4 Customer Perspective (cont.)

5. 5 Customer Scorecard Methodology (Radiation Safety) Incorporated ORS Customer Scorecard into monthly newsletter Mailed to 950 Authorized Users Issued reminder in following month’s newsletter Response rate 24%

6. 6 Scatter Diagram (Radiation Safety) FY03 Customer Importance and Satisfaction Ratings Note: The Importance rating scale ranges from 1 - 10 where “1” represents Unimportant and “10” represents Important. The Satisfaction rating scale ranges from 1 - 10 where “1” represents Unsatisfactory and “10” represents Outstanding.

7. 7 Customer Perspective (cont.)

8. 8 How do you prefer to obtain information about the radiation safety program? 1 = Most preferred method 5 = Least preferred method N = 225

9. 9 Relationship Among Performance Objectives Enhancing communication with our customers would Maintain compliance with regulations Increase the use of on-line services

10. 10 Internal Business Process Perspective

11. 11 Internal Business Process Perspective

12. 12

13. 13 Internal Business Process Perspective

14. 14

15. 15

16. 16 Internal Business Process Perspective

17. 17 IB8: Improve Scheduling of Radioactive Waste Service

18. 18 Financial Perspective

19. 19 F1: Compare this year’s unit cost to last year’s

20. 20 F4: Minimize Unit Cost for Acquisition/ Distribution/Disposal of Radionuclides

21. 21 Financial Perspective Cost per AU increased minimally RAMOS not cost effective and now obsolete

22. 22 Financial Perspective What actions are planned? Effective October 1, the RAMOS system was discontinued. Cost of radioactive materials now incorporated into radiation safety fee for service Continue streamlining

23. 23 Financial Perspective (cont.)

24. 24 Conclusions

25. 25 Conclusions from PMP Our customers prefer to communicate via e-mail Need to increase positive face-to-face interactions Decrease number of security violations by providing incentive for compliance Having people complete the RSAU is worthwhile even if they don’t become Authorized Users Achieved reduction of unnecessary radwaste pickup trips Will advertise on-line scheduling option

26. 26 Table of Contents PM Template ……………………………….………………………………. Customer Perspective……………………….………………………………. Internal Business Process Perspective……………………………………… Learning and Growth Perspective…………………………………………… Financial Perspective………………………………………………………… Conclusions and Recommendations…………………………………………..

27. 27 Appendices Template29-31 Customer Perspective32 C1: Increase Customer Satisfaction33 C1: Score on Customer Scorecard34-58 C2: Enhance communication with customers59 C2a: Results of survey question60 C2d: Number of people trained61-63 Internal Business Process Perspective64 IB6: Maintain NIH in compliance with NRC regulations65 IB6: Number of security violations66 IB7: Evaluate long-term benefit of RSAU class67 IB7: Percent of attendees who become participating users 68-69 IB8: Improve effectiveness of radioactive waste scheduling70 IB8a: Decrease unnecessary pick-up trips71 IB8b: Improve ease of scheduling pick-ups Learning and Growth Perspective LG1: Maintain effective staffing level in RSB73 LG1: Compare this year’s turnover rate with last year’s74 LG2: Increase number of awards and dollars per award 75 LG2a: Number of awards LG2b: Dollars per award LG3: Increase sponsored training for RSB76 LG3a: Number of training hours per person LG3b: Training budget for past 5 fiscal years77 LG3c: Cost of training for past 5 fiscal years77

28. 28 Appendices (cont.) Financial Perspective78 F1: Minimize unit cost for technical assistance in radiation safety F1: Compare this year’s unit cost to last year’s79 F4: Minimize unit cost for acquisition/distribution/disposal of radionuclides F4: Compare this year’s unit cost to last year’s80-81 Process Maps85 Provide technical assistance in radiation safety86 Authorized User training and approval87-88 Acquire/manage/distribute radioactive material89-90 Conclusions91-93

29. 29

30. 30

31. 31

32. 32 Customer Perspective

33. 33 Customer Perspective (cont.)

34. 34 Results from the FY03 ORS Customer Scorecard Service Group: Maintain Safe Work Environment Product/Service: Radiation Safety 26 November 2003 Summary prepared by the Office of Quality Management (OQM) and SAIC

35. 35 Methodology Radiation Safety discussed with OQM their proposed customer assessment methodology Distribute survey similar to FY02 with a few new questions Determined method of survey distribution Hard copy survey mailed with monthly newsletter to all on the distribution list Surveys returned to OQM via NIH mail system Preserve customers’ anonymity Ensure the integrity of the results Survey data were entered into a database and analyzed

36. 36 Survey Distribution FY03 Administration Number of Surveys Distributed950 Number of respondents225 Response Rate24% FY02 Administration Number of Surveys Distributed574 Number of respondents 90 Response Rate16%

37. 37 Satisfaction Ratings: Comparison by Fiscal Year N = 225 N = 90 Unsatisfactory Outstanding * FY03 ratings significantly lower (p<.05) than FY02 ratings.

38. 38 Importance Ratings: Comparison by Fiscal Year N = 225 N = 90 Very Unimportant Very Important

39. 39 Scatter Diagram FY03 Customer Importance and Satisfaction Ratings Note: The Importance rating scale ranges from 1 - 10 where “1” represents Unimportant and “10” represents Important. The Satisfaction rating scale ranges from 1 - 10 where “1” represents Unsatisfactory and “10” represents Outstanding.

40. 40 Scatter Diagram FY03 Customer Importance and Satisfaction Ratings: A Closer Look Note: A smaller portion of the chart is shown so that the individual data points can be labeled.

41. 41 FY03 Satisfaction Ratings: Radiation Safety Comparison to ORS Overall N = 225 N = 5,698 UnsatisfactoryOutstanding

42. 42 FY03 Importance Ratings: Radiation Safety Comparison to ORS Overall N = 225 N = 5,698 Very Unimportant Very Important

43. 43 How easy is it to obtain information from web site? Scale ranges from 1 = Very Difficult to 10 = Very Easy N = 225

44. 44 Reviewing Comments Realize comments are qualitative data and are not meant to be counted and tallied Comments provide a different type of information from your customers regarding their satisfaction Comments are NOT representative of the perceptions of all your customers Review them but don’t over react to an individual comment Comments are a great source for ideas on how to improve

45. 45 What was done particularly well? Waste pick up staff/timeliness fantastic at GRC. Thomas Johnson is very helpful and conscientious! I have been very pleased with the exposure monitoring program and the department specific refresher training. Response to spill. HP's respond very quickly to any questions. On-line orientation and RSB class sign-up is a great idea and works well. Follow-up to questions asked. Contractors are friendly and helpful overall. I have always had a productive, helpful relationship with the RSB. Services have been accessible and reliable. Radiation survey. This office is the best one at the NIH. It takes complex area and helps investigators. It's been this way for 20 years. Do not change it. Delivery of radioactive products. Radio isotopes delivery. Notification of problems with surveys and upkeep of the radiation work area - one on one contact with the HP. Prompt completion of requests. If RSB personnel doesn't have answer, they forward quickly to another who can help.

46. 46 What was done particularly well? (cont.) Quality of isotopes purchased. Notification of inability to deliver because of lack of some form or information has improved. Pre renovations information on how to handle all details, Ms Cathy McC____ was very helpful. RSB Service and personnel are excellent (fast, efficient, competent, and very helpful). Surveys. Quick direct contact to get lab AU approved, room posted, quick waste pick up, drop offs. Phone and email correspondence is usually conducted in a timely fashion. Ordering isotope is very quick and economical. Checking out hot jobs prior to renovations/moving out. Good communication during our HP transitions. The radiation safety branch has always been great to work with. I ship and receive radioactive materials and both ways they do a wonderful job. :) Always providing good advice. Monitoring how we are doing is well appreciated. The competence of the staff at all levels is exceptional. Rapid turn around on all requests. Whenever I call my questions are answered - and phone is always answered by a real person! Fantastic! Everything. They do a good job. Especially the RSC, Ms. Coronado, our Health Physicist.

47. 47 What was done particularly well? (cont.) Response to questions, quick shipment of order and a clear (non-radioactive) container. Safety monitoring. Lab refresher course of HP. Reliability Monthly Status Reports. Both the willingness to help solve a problem and the sincerity with which it is handled. Ordering, delivery goes smoothly. Disposal of rad waste. Response to a spill. Waste pickup. All aspects of the RSB I have dealt with have done a great job. Handling changes from one AU to next. My positive impressions of RSB are based on interactions with the health physicists, who have been uniformly competent and helpful. Neena Patel is exceedingly so. Surveys. Delivery of radioactive materials. I have routinely gotten good information or good response from Radiation safety. This is a really well-run group. Responsiveness, helpful attitude. Everything.

48. 48 What was done particularly well? (cont.) Training, survey. Everything! On line services have made a radical improvement in the last 2 - 3 years. Response time with any inquiry was very short. I especially appreciated George Redmond's helpful and generous manner as health physicist. Radioactive waste pick up. Entering new people into the system inventory. Tracking and communication. Providing solutions to laboratory problems. Delivery of radio isotopes is timely and efficient. 1. The web training/refresher is useful. 2. Only important notifications are sent out to all users. Radiation Safety is very helpful and competent. I have no complaints. I find monthly memos are a good reminder for me to check on our inventory of isotopes and to make sure the survey was carried out. Survey contractors are very good and helpful. Customer service.

49. 49 What was done particularly well? (cont.) Guidance through tough situations. Radioactive waste pickup. All deliveries are timely. In addition, the Authorized Users Course that I took was comprehensive and very well done. Any concerns or questions were answered in a timely manner. Timeliness. Keeping track of the survey reports, phone contacts. HP's are very responsive/competent. For the amount of material received/used at NIH, size of campus, etc., I think the Rad Branch is pretty well managed! Waste disposal. Please note that I haven't used RSB services recently since I haven't ordered products recently. Timely delivery and repackaging of materials. Response to "hot trash" problems. Problems with surveys. Answering questions. Health physicist and radiation shipping/receiving personnel very knowledgeable and helpful. Isotopes are always ordered and delivered in a timely manner. I use very little isotopes any more.

50. 50 What was done particularly well? (cont.) RSO is good. Waste pick up. Training services. I don't currently use radioactivity, but will start soon, so will have more info next year. Radiation safety provides a valuable and essential service for the NIH. Everything. RSO is good. Audits are done extremely well and problems are fixed very efficiently. Pick ups are timely and efficient. Record keeping/all round excellent service! Responsiveness when I have questions or problems. Interactions with RSB personnel. Everything was very good. Note. I always make a copy of the NIH 88 form because they used to get lost often. We often got calls that our radioactivity had arrived but they needed the NIH 88 form again. This has not happened for a very long time, although we are not using as much radioactivity as we had in the past. Quick service for waste pick-up.

51. 51 What needs to be improved? Communication about missed delivery but Israel Putnam goes out of his way to inform and assist when involved!!! He is fabulous! When I search the NIH home page (Radiation Safety), your web site should be the first hit. Ordering. Need to add 88 form calculations to web site. Isotope delivery seems to take varying amounts of time making it difficult to rely on fresh isotope being available for planned experiments. The delivery of short T 1/2 isotopes for in-human use. Delivery of Radio Isotopes. Don't change it. Paperwork tracking. Every year I receive a threat that my AU status will be taken away because I failed to file 2-3 monthly reports. Every time it has turned out that we sent the reports in on time, but Radiation Safety lost the reports. Radiation Safety must lead the way in rolling back the security regs governing radioactive substance usage. Lock-downs never were and still are not appropriate! Refreshing course for users - didn't have any for years. (Could be computer on-line one.) Very little. Delivery time.

52. 52 What needs to be improved? (cont.) Encourage HPs to obtain more accurate and detailed knowledge of protocols to which they are assigned, thereby increasing the quality and effectiveness of communication with users, particularly regarding problem solving related to procedure and documentation. Monthly lab survey should be simplified, for example, to room didn't get to use isotope, should be exempt from surveying. Ordering system. Nothing. I am happy as is. Convenience can be improved and more than likely will be improved with maturity of the website. Enforcement of RS rules for dress in Labs. No complaints. On the Monthly AU Status Report page 1, is there a way to highlight an item that is new or of extreme importance to distinguish that item? Nothing really. The radioactive disposal forms should be available electronically and should be filed through email. Would be nice to have an advance notification about the delivery of radioactive materials to the lab, to make sure that somebody is present to receive it.

53. 53 What needs to be improved? (cont.) Nothing really. The handling of problems in a efficient manner. Nothing. Everything at a good level. Record-keeping regarding transfer of radioactive materials to new authorized users. On-line information. Do not contract out this function! It should be classed as inherently governmental or core to the NIH. This branch is an asset to the NIH. Recent statements by their staff allude to the A-76 outsourcing initiative - This should not occur to radiation safety. An on-line form 88 would be good. Don't A-76 them. In most cases, suspension penalties aren't needed. Better to review procedures and work with labs and investigators. Nothing. Haven't had complaints. Better coordination between registering a user and actually allowing a person to use materials. I have not experienced anything seriously lacking in RSB performance.

54. 54 What needs to be improved? (cont.) Cost decreases for services. Authorized user status reports are not up to date with faxed disposal records. Could all training be done on the web? Haven't used it in a long time. No need to hire outside contractors to survey so frequently! It's a waste of money. 1. If anything, I think that the folks who take care of Radiation waste pick-up could be a little more "focused" - come in, remove proper waste, leave. They do a good job, but have made mistakes. 2. Overall, I think record keeping ("materials in possession") is good, but could be improved. Fortunately, when I do find errors, they are corrected quickly. 3. Employees answering 6-5774 could be a little less abrupt, friendly, servicing. Announcing delivery of radioactive materials by E-mail before they arrive at the door. AU portal extremely slow. Don't see anything right now. I get praise for my records by the audit team but I feel very nervous when my health physicist comes around. I never felt like that with other physicists or anyone else for that matter (from RSB). In other words, I get conflicting opinions about my performance, one is great and the other is hardly satisfactory. Consistent policies for when labs are relocated. I-88 forms, ordering in AMBIS (RAMOS). I know this has changed and there will be on-line authorized user I-88 filing, but please hurry. Our purchasing experience has been terrible.

55. 55 Other Comments I have not used isotopes for 12 years. I am Scientist Emeritus since 1995 and do not use Radiation Safety Services. Not applicable, no service provided in the past 10 years. Have not used services lately, but appreciate the web site.

56. 56 What would you like to see added to the Radiation Safety Branch Web-site? Information on each isotope as a quick reference for shielding, half-life, etc. A quick link to "forms" - e.g., left-hand, lower home page. Radioactive decoy calculator needs to have some PET isotopes added. RSC 88 Forms. Need to make it clearer how to transfer radioactivity from one licensed investigator to another. Nothing. How are we charged for Radiation Safety Services? I have no specific requirements. The current web site is quite user friendly and efficient. Don't use. Looks complete to me…pretty well organized. Basic knowledge about how to convert different units of radioactivity. Or some of the things I learned in Authorized User Training but use rarely: kind of an additional guide about radioactive material. The status of an isotope order: received/not received, processing, to be delivered AM/PM, etc. I do not use it that often, so I can't really say.

57. 57 What would you like to see added to the Radiation Safety Branch Web-site? (cont.) Information of status and delivery of orders of radioactive materials (in AU portal). Downloads of forms. "Hot" buttons for most popular info FAQs. Full package information or "inquiries screen". A way to track orders in their system. Hard to find things - needs a site map. No system jams (cleaner programming), broader search power. No complaints - no suggestions. More thorough integration of links to related sites. A monthly contamination survey that could be filled out on line and emailed to RSB. Information given/taught during Authorized Users Course. Fine for our needs. Course dates posted on or near home page. Don't use it much. Better search engine. Log in windows for people to check their radiation user status and survey report status for their laboratory. Seems pretty complete to me (well designed, user friendly).

58. 58 N/A. Have not used radioactive materials since 2000. Perhaps a section/index on "new information" at the site so you can periodically check it out for updates. Password access repaired. Updates of the list of health physicists. Assignments change but no update info on the website. This web site is not recognized by my server. I don't have time to browse the internet. I will read something if directly sent there by an e- mail memo. 1. Enter NIH IC, then make appointment for waste pick up and get result of monthly surveys. 2. Any information about new rules, news….. 3. Notifications. Simplified instructions on use of a single isotope, e.g., 35s, from beginning to end, including ordering, delivery, use, disposal, waste pickup and paperwork. File on individual users to track schedule accessible by individual only, of course. I would like to see surveys I've missed or the nucleotide list on line. Information on each isotope as a quick reference for shielding, half-life, etc. What would you like to see added to the Radiation Safety Branch Web-site? (cont.)

59. 59 C2: Enhance communication with customers

60. 60 How do you prefer to obtain information about the radiation safety program? 1 = Most preferred method 5 = Least preferred method N = 225

61. 61

62. 62

63. 63

64. 64 Internal Business Process Perspective

65. 65 IB6: Maintain NIH in compliance with NRC regulations

66. 66

67. 67 Internal Business Process Perspective

68. 68

69. 69

70. 70 IB8: Improve effectiveness of radioactive waste pick-up scheduling

71. 71 Internal Business Process Perspective

72. 72 IB8: Improve Scheduling of Radioactive Waste Service

73. 73 LG1: Maintain effective staffing levels

74. 74 LG1: Maintain Effective Staffing Level

75. 75 Learning and Growth Perspective

76. 76 Learning and Growth Perspective

77. 77 LG3: Increase Sponsored Training

78. 78 Financial Perspective

79. 79 Financial Perspective (cont.)

80. 80 F1: Compare this year’s unit cost to last year’s

81. 81 F4: Minimize Unit Cost for Acquisition/ Distribution/Disposal of Radionuclides

82. 82 F4: Minimize Unit Cost for Acquisition/ Distribution/Disposal of Radionuclides

83. 83 Financial Perspective Cost per AU increased minimally RAMOS not cost effective and now obsolete

84. 84 Financial Perspective What actions are planned? Effective October 1, the RAMOS system was discontinued. Cost of radioactive materials now incorporated into radiation safety fee for service Continue streamlining

85. 85 Process Maps

86. 86

87. 87

88. 88

89. 89

90. 90

91. 91 Conclusions

92. 92 Conclusions from PMP Our customers are highly satisfied with the services we provide Our customers prefer to communicate via e-mail Need to increase positive face-to-face interactions Will try to decrease number of security violations by providing incentive for compliance Having people complete the RSAU is worthwhile even if they don’t become “participating” Authorized Users

93. 93 Conclusions (cont.) Achieved 11% reduction in unnecessary radioactive waste pick-up trips Will advertise on-line scheduling option for radioactive waste pick-ups Training budget is tracking well with training costs