Anne Sholander, MT(ASCP), International QA/QC Coordinator, SMILE DAIDS clinical trials outside the US: Furthering the quality of laboratory operations The NIAID SMILE resource (November 2006) Anne Sholander, MT(ASCP), International QA/QC Coordinator, SMILE

Who are the partners? DCLOT PPD Non-Network SMILE Sites Networks

How does DAIDS manage clinical laboratory oversight for HIV trials? There are currently nearly 170 laboratories outside the US performing DAIDS funded HIV clinical trials.

Patient Safety Monitoring in International Laboratories SMILE Patient Safety Monitoring in International Laboratories Our Mission: To Provide Quality Assurance Support for DAIDS-Funded HIV Clinical Trial Laboratories (Non-US) To Maintain a Web-Based Repository of Records and Resources

“This contract will provide a critically needed comprehensive, uniform program to evaluate the capability of laboratories in developing countries, advise and train when deficiencies are identified, and ensure the ongoing quality of study test results and ultimately the outcome of clinical research performed in these countries.” --original contract language --outlined the goals of the project --we remain true to these original goals

Clinical Laboratory Research Resources We aim to achieve Safety of trial subjects. Safety of those performing tests. Quality of the laboratory data. Increased appreciation of labs for good laboratory practices that translates to better local service. What are the goals of clinical laboratory research resources? What do we hope to achieve with laboratory oversight?

Who is SMILE? Contract resource between NIH/NIAID-DAIDS & Johns Hopkins University Staff of 11 with a wide range of laboratory experience Combined total of 185 years of experience Wide variety of backgrounds and areas of expertise we come from clinical labs, research labs, public health labs and from DAIDS funded HIV lab (Zimbabwe). Depth of experience in variety of lab disciplines. SMILE staff is experienced in: Microbiology/TB , Malaria Chemistry/Hematology/Immunology Validation of new methods, instrument troubleshooting Manual differentials/urinalysis Scientific writing and editing We are especially fortunate to have a PI with a strong background in Bioinformatics. This has lead to development of extensive computer and web based tools used by SMILE in this project.

How does SMILE work to achieve the goals of our contract? Primarily from our office in Baltimore. Extensively through email Regular teleconferences with our site labs, networks and DAIDS. Each lab is assigned a primary and a back-up coordinator. regular (sometimes daily) communication with the labs --develop a personal relationship with the labs.

A year in the life of a SMILE supported lab… The best way to show you the tools that we use at SMILE is to walk through a typical year in the life of a SMILE supported lab. How our tools work to achieve these goals.

Laboratory Audits Independent assessments are performed annually by PPD or DAIDS Based on the standards of Good Clinical Laboratory Practice (GCLP) The “year” for the lab can be said to begin with the audit. Independent of the SMILE contract—avoids any conflict of interest. Audits performed by DAIDS contractor, PPD or by DAIDS. PPD (or DAIDS) performs the audits and SMILE assists with remediation of audit findings. Audits based on GCLP

Good Clinical Laboratory Practice (GCLP) Organization and Personnel Testing Facilities Operation Verification of Performance Specifications Quality Management Equipment, Physical Facilities Personnel Safety What is GCLP? US has a set of accepted guidelines and standards for clinical labs, there was a need for a unifying set of standards for international labs. DAIDS AND contractor PPD developed GCLP guidelines. GCLP are a set of standards based on FDA, CLIA, CAP, ISO, SANAs guidelines to ensure the quality of study data and allow for accurate reconstruction of experiments. Some of the sections of the audit include: Audit gives a comprehensive picture of the lab—and covers all aspects of Good Clinical Laboratory Practice. :

Audits and Action Plans Audit report sent to SMILE Action Plan is prepared Networks designate critical items Action Plan and audit sent to lab What do we do with the audit information? Audit reports sent to SMILE for review and preparation of an Action Plan. Action plan contains the audit finding or observation, and SMILE prepares a suggested action for remediation of the finding. Affiliated networks review Action Plan and designate items critical to their studies. This process also works for non-network affiliated labs. Once the Action Plan is finalized it is sent to the lab along with the Audit.

The Action Plans are “Keyed” to the audit shell The Action Plans are “Keyed” to the audit shell. Lab can cross-reference original audit report. SMILE suggests an action for remediation and provides resources/training. and we track the history of that action item. The AP is color coded. Can see at a glance where the lab stands. Generates feedback and interaction --between the lab and SMILE to resolve items. SMILE is able to provide guidance and training based on our SMILE resources and experience of SMILE staff to address specific problems in the lab. Used by networks and DAIDS as a tool to assess lab readiness for opening of new studies. Also used as a tool to assess the readiness of a brand new lab to open it’s first study. Goal is to close the Action Plan before the next audit.

SMILE Site Assistance and Training Email Phone Web conference Adobe Connect At conferences On site The Action Plan often leads to opportunities to help our labs by teaching or training. Many times this takes place in email or telephone conversations. Our computer expertise and support has also allowed us to use web tools and create computer based training programs. We utilize the Adobe Connect software which allows us to connect with our sites via the web and show them any presentations or walk them through any trainings that we have. We have also presented trainings at conferences. TB Safety training was given at the last ACTG meeting and will be presented in Thailand in January. We also are occasionally asked to visit a lab to provide specific training or assistance. Accomplish a specific goal or need—focused visit. Our goal is to empower the labs through training. By doing this the knowledge and the experience stays with the lab even after the DAIDS study is finished.

Another type of lab assistance or training has been the SMILE TB lab observational visits. SMILE has travelled extensively to assess and assist our labs with their TB testing capabilities. Through these SMILE visits and the support of DAIDS and the cross-network TB working group, the quality, capacity, and safety of TB laboratories in developing countries has improved dramatically. (We have already been to 17 network TB labs, 4 National Labs one of which is going to be added to network oversight and will have a PPD visit, 3 more will be in process during the presentation, 1 is scheduled for January.) (This picture was taken in Uganda.)

Audits occur only once per year. How do we monitor laboratory performance continuously throughout the year? Back to the year in the life of our SMILE lab….and how we are managing laboratory oversight. The lab has had an audit and is working on their action plan with SMILE. But the audit only occurs once per year. How do we know throughout the year that the test results released from the lab are accurate?

Proficiency Testing Proficiency Testing (PT) is a means of evaluating a laboratory's performance through analysis of unknown samples provided by an external source.

What is proficiency testing? Also called PT or EQA (External Quality Assurance) Blinded samples sent on a regular schedule Run as patient samples Results are compared to peers The proficiency testing process is independent of SMILE—we cannot influence the outcome. It is similar to audit where SMILE uses tool provided by an independent group to learn information about the lab’s quality. SMILE provides assistance with ordering EQA. Based on our experience with different providers and the requirements of the different network protocols, we are able to customize the PT order to the needs of the lab and the study.

Before we order EQA we need more information Protocol analytes Instrumentation in use Back-up plans Specimen flow In order to be able to create a customized PT order for a laboratory, we need detailed information about the testing that will take place. We first need to know which analytes will be needed for the protocol—both for safety testing and for study monitoring and endpoint data. We also need to know which instruments and methods the lab will be using. We need to know what the lab plans to do if their primary method of testing fails—what is their back-up plan. It is also helpful to understand the specimen flow in the lab. All of this information is captured on a Protocol Analyte List (or PAL) spreadsheet. Protocol Analyte List (PAL)

This form was originally developed jointly by the networks/DAIDS and SMILE and it has become a critical tool for SMILE. Specific to a particular protocol. It shows the details of analytes are needed for a protocol —and how and where they will be tested. It also shows how we intend to cover those analytes with EQA—who the provider is and how frequently the panels will be delivered. It is a way to communicate what the specific needs of a protocol are.

How does SMILE use proficiency testing? Monthly review and monitoring of EQA results Evaluation Review Tracking and summary Reporting Using the PAL, we have ordered EQA and the lab has begun testing the samples and submitting the results. What does SMILE do with this information? PT results are obtained from the EQA provider directly (usually). PT provider has evaluated results. SMILE provides the lab with a review of their survey results. Review process is a required element of GCLP. Summarize the information and notify of investigations needed. SMILE tracks PT performance of each lab on an EQA Summary Spreadsheet. SMILE sends a monthly email report to the lab and all affiliated networks (copying the DCLOT POC)detailing the performance on that month’s surveys and any pending investigations.

EQA Summary Schedule On this first page of the spreadsheet is the year’s schedule for EQA shipments. Lab knows a year in advance when to expect their survey shipments This page also summarizes the laboratory’s status at a quick glace. By color coding we can quickly see that there is an outstanding investigation required for Gram stain.

The second page of the EQA spreadsheet gives detailed information about each analyte tested in the lab. The EQA history for 3 years is maintained on this sheet. Color coding also gives a quick picture of how well the lab is performing. So, for example, in 2008 this lab had difficulty with their platelet counts. This result would require an investigation—any result less than 100% successful (for a protocol analyte) requires a documented investigation. In the case of this platelet count—the problem appears to have been corrected on the next survey.

Investigations Required for EQA failures SMILE provides assistance, tools and resources Root cause and prevention of future failure Documentation of corrective action is end product SMILE assists with investigations of EQA failure and provides labs with tools and resources for corrective action. The process of troubleshooting the cause of EQA failure is another example of how important our extensive experience in the laboratory is. The process often involves a lot of back and forth email and research and discussion of possible causes. Our goal is to teach the labs how to investigate problems and prevent them in the future by the use of good laboratory practices.

SMILE Investigation Form The SMILE investigation form is designed to aid in the process of troubleshooting. It identifies the analyte, and then asks a series of questions designed to lead the lab through a thorough investigation.

Root cause analysis The goal of an investigation is root cause analysis and prevention of further failures. We often help labs discover problems by prompting them to probe deeper into EQA failures. We push them to find the cause of a failure and fix it, so that the integrity of patient testing is ensured.

Where do we store all of this documentation? We are generating a lot of documentation What do we do with all of this documentation—where is it kept? One of our missions was to create a document repository. We file all of each lab’s documentation on our website. Psmile.org—this is our home page

Document Repository EQA surveys, reviews and investigations on www.psmile.org Available to site labs (privately by log-in) Available to DAIDS and the Networks Also on www.psmile.org: Audits and Action Plans Site visit reports Protocols and Protocol Analyte Lists (PAL) Currently there are approximately 18,000 site lab documents stored on psmile and we upload an average of about 400 new documents each month.

Each site lab is updated at least monthly with their most current EQA summary, and new PT surveys and reviews. Each lab can see only their own information—DAIDS and the affiliated networks also have access to this information. Any investigations submitted during the month are also stored here. We also store the PPD Audits, action plans and PALs on psmile. However, Psmile is much more than just a document storage website.

Resources on pSMILE.org One of the things we are proudest of is our collection of resources posted on psmile. SMILE resources are open to all—no log in is required. This area of psmile is searchable by keyword. For example, if a lab is looking for information on Water Quality, they can simply type this in the search box. There are currently ~1400 documents and weblinks posted in the resources section. Many of these resources were written or developed by SMILE staff. We have devoted a considerable amount of our efforts to improving and perfecting our web resources. Audit remediation, Training and Competency, Instrument validation Laboratory move planning GCLP compliance and Safety Specific instrument/kit information The success and evolution of our webpage is a credit to our PI Dr. Miller and his experience in the application of computer technology to the laboratory. We are constantly encouraged to enhance and improve our use of computer and web-based resources.

Monthly reports sent DAIDS Let’s recap the year… Audit Action Plan EQA Monthly reports sent DAIDS 1. The lab has had an audit 2. SMILE provides resources and works with the lab individually on their audit remediation by working to resolve action plan items. 3. SMILE reviews and evaluates EQA on a monthly basis. SMILE is tracking the laboratory’s EQA and Action Plan progress and uploading all updates to www.psmile.org 4. Monthly reports are sent to DAIDS that summarize all 170 SMILE labs

Summarizing the data EQA Exceptions Delinquent Investigations Tracking Log Nearly 170 labs-- how do we summarize the month for DAIDS ? How do we keep DAIDS informed of what is going on with the labs, without overburdening them with the detailed information? While we maintain all the detailed information, and it is available at any time—we also summarize the information on a monthly basis and send it to DAIDS. The EQA Exceptions list shows labs that have two out of three unsuccessful rounds of EQA. Alerts DAIDS and the networks that there may be a problem and testing may need to be moved to back-up. Delinquent Investigations is a list of labs that are >60 days delinquent in submitting investigation. Tracking log gives a brief summary of the month’s activities for each lab.

All of these activities create an ongoing dialogue between: SMILE and the labs SMILE and the networks SMILE and PPD SMILE and DAIDS SMILE and IQA and VQA All of these activities create an ongoing dialogue between: This dialogue leads to constant, continuous improvement for our labs, for our processes, for our resources. Although our focus at SMILE is on patient safety testing, we become more than just a resource for safety testing. We also become a key contact and facilitator. We triage the questions to the appropriate organization. We provide support to practically every test that appear in protocols, except for CD4 testing, where the IQA is responsible to monitor performance and for HIV virological tests where the VQA is responsible.

Additional benefits of SMILE Experience Discounted costs and improved services Consistency Flexibility Collaboration Experience—we have extensive experience in all aspects of laboratory oversight. We utilize every member of the staff and draw on his or her strengths to give the best possible service to our labs. Discounts--Because of the number of labs we assist, we have an advantage in dealing with proficiency providers. We are the third largest user of CAP proficiency testing products, which gives us a tremendous advantage. We have points of contact and a relationship with the PT providers—we are able to work with them to solve problems unique to our labs. Ability to negotiate and manage premium shipping where needed. Consistency across the SMILE staff so that all labs are treated in a uniform manner. Extensive internal SOPs, constantly refining the process and identifying areas for improvement. Flexibility—we are not locked into the use of one PT provider, one shipper, etc. We have the flexibility to look at what will work best for the labs and customize their coverage accordingly. Use of SMILE creates a sense of collaboration—between all affiliated networks, the labs and DAIDS. A unifying force that ensures that the requirements for each network and DAIDS are met and maintained in the lab.

In conclusion… How does SMILE work to achieve the goals of Clinical Laboratory Research Resources? To sum up—let’s look back at the original goals outlined for Clinical Laboratory Research Resources. There were 4 goals outlined for laboratory oversight. How does SMILE help achieve these goals?

Safety of trial subjects We have an obligation and responsibility to protect the safety of the patients in our clinical trials By ensuring that laboratory tests are accurate we can ensure the safety of trial subjects.

2. Safety of those performing tests Good Clinical Laboratory Practices place emphasis on laboratory safety. Audit and Action Plan focus on safety of laboratory staff.

3. Quality of the laboratory data. Quality and integrity of study data. Why conduct research if the data is not reliable? Laboratory oversight ensures that study data is valid and reproducible. Documentation is created that allows reconstruction of study experiments and shows corrective action for any problems that occur during the study.

4. Increased appreciation of labs for good laboratory practices that translates to better local service. When we improve the quality in the labs, it creates a ripple effect. The culture of emphasis on quality spreads—to other labs, to clinics, to hospitals. Education and job training for laboratorians improves. Infrastructure and resources are improved. All of this translates into improved quality of healthcare in developing countries.

Some of the things we are proud of… Laboratories taking the initiative to improve their quality measures on their own Laboratories striving for (and achieving) CAP, ISO and SANAs accreditation Laboratories publishing/receiving awards Recognition from local authorities of improved laboratory quality Improved safety and increased testing capability and quality of TB diagnostic laboratories Some of the things we are most proud of are the side-effects, or by products of our involvement with labs. We strive to improve the lab as a whole—not to narrowly focus on only those things that directly affect our studies. The SMILE approach to lab oversight reflects a shift in the paradigm, away from the old model of placing Westerners on the ground to manage the labs. We aim for lab empowerment and improvement of local staff and infrastructure. We are very careful to train, mentor, assist and encourage—without doing the work FOR the labs. We want the quality of these labs to persist long after the DAIDS study has closed.

DAIDS clinical trials outside the U. S DAIDS clinical trials outside the U.S. -furthering the quality of laboratory operations The NIAID SMILE resource Dr. Robert E. Miller, M.D.—Principal Investigator Kurt L. Michael, M. Ed., MT (ASCP)—SMILE Project Manager