1. Risk-Based Monitoring Quantitative Metrics Toolkit
04 November 2016

2. Contents Purpose of this Document 3 Objectives 4 Process 5
Core Metrics
Recommendations for Historical Control Calculations
Recommendations for Historical Control Periods
Guidance for Potential Surrogates
Establishing Target Ranges
Expected Observations and Potential Alternative Observations
Final Considerations
RBM Metrics Report 1Q2016 (Bi-annual)
Trial Inventory (RBM Uptake, trials planned or in progress)
How to read the metrics (Stacked charts)
Actual metrics report from 1Q2016 3 4 5
6-7 8 9 10
11-12
13-14 15
16-26 17 18
19-26

3. Purpose of this Document
As the TransCelerate RBM initiative developed the methodology and framework for voluntary RBM implementation, there was a recognized need to quantitatively evaluate progress and impact.
The team developed 8 core metrics which could be used to evaluate the impact of the TransCelerate RBM methodology on clinical development programs across three broad categories; Quality, Efficiency and Cycle Time
A process was developed to support sponsors in determining historical controls, setting target values, measuring the metrics, and assign a dashboard rating. All TransCelerate RBM metrics have been reported anonymously to a neutral third party for aggregation.
Overtime, it was recognized that companies needed the flexibility of defining the metrics differently due to internal systems, procedures, metrics, etc. Guidance was provided to companies in an effort to move toward some consistency in measurements for some metrics in accordance with regulatory agency requests. Companies may determine the extent to which they follow the recommendations.
Additional guidance is also provided to assist with historical control calculations and periods, potential surrogates and setting target ranges.
All information, recommendations or guidance contained herein is voluntary for sponsors to utilize.

4. Quantitative Metric Collection Objectives
Determine the impact and effectiveness of the proposed RBM methodology on managing quality and risks associated with the conduct of clinical trials.
Determine if the RBM methodology works from the standpoint of operational impact on an organization, clinical sites and investigators.
Keep in Mind:
Benefits realized must be accompanied by either an improvement or maintenance of current criteria in data quality and subject safety.

5. RBM Quantitative Metric Analysis Process
Determine Historical Control
Set Target Values
Measure Metrics
Assign Dashboard Rating
Recommendation Generate historical control at one of four levels:
1.) IP level
2.) therapy area level
3.) cumulative
4.) split study
This will remain constant through assessment
Recommendation Determine target values on three levels:
1.) Improvement at X%
2.) Worsening at Y%
3.) Negligible change (About the same) between X% and Y%
Measure the metrics that are feasible to be quantified
Compare quarterly metric to target values and assign dashboard rating of better, about the same or worse

6. Core RBM Quantitative Metrics (1 of 2)
Indicator
Metric
Optional Guidance: For consistency amongst sponsors
Quality
Number and classification of major/critical audit findings per site audit
Better + 25%
Worse -25%
About the Same > -25% & < +25%
Number of unreported, confirmed SAEs as discovered through any method
Better + 10%
Worse -10%
About the Same > -10% & < +10%
Significant Protocol Deviation rate per treated subject (total # of deviations/ total # of subjects for the protocol)
Recommend normalize to per treated subject or patient.
Do not include bio or stat programmed reports. Include only Significant PDs identified by central or site monitoring or data cleaning activities.
Efficiency
Average Monitoring (all types) cost per site
Average interval between on-site monitoring visits per site
Recommend measuring using Start Date to Start Date

7. Core RBM Quantitative Metrics (2 of 2)
Indicator
Metric
Optional Guidance: For consistency amongst sponsors
Cycle Time
Median number of days from visit to eCRF data entry
Primary recommendation is to use first data entered.
Better + 10%
Worse -10%
About the Same > -10% & < +10%
Median number of days from query open to close
Recommend focus on site activity, use the response by site for consistency, exclude any auto generated queries
Median days from significant/major issue open to close
Recommend focus on significant or major findings if able to. Companies should define what they feel are significant.

8. Recommendations for Historical Control Calculation
Metric
Determination – historical controls and metrics
Average number of major/critical audit findings per site audit
Calculate the number of findings and divide by the number of site audits per quarter
Potential sources to include: QA audit reports
Percentage of unreported, confirmed SAEs as compared to total SAEs as discovered through any method
Calculate the number of findings and divide by the total number of SAEs
Potential sources to include: Monitoring reports
Significant Protocol Deviation rate per treated subject (total # of deviations/ total # of subjects for the protocol)
Calculate the number of findings and divide by the total number treated subjects.
The definition of “Significant” to be defined by each sponsor
Potential sources to include: EDC platform
Average Monitoring (all types) cost per site
Compile all costs associated with monitoring the trial and divide by the number of sites
Potential sources to include: CTMS, Finance
Average interval between on-site monitoring visits per site
Determine the interval between on-site monitoring visits for all sites and divide by the number of sites.
If a site has not had a second visit to perform analysis in the quarter, omit that site from analysis
Potential sources to include: CTMS
Median number of days from patient visit to eCRF data entry
Calculate median
Median number of days from query open to close
Median number of days from significant/major issue open to close
The terms “issue”, “open” and “close” to be defined by each sponsor
Potential sources to include: Issue Tracking System

9. Recommendations for Historical Control Periods
Metric
Historical Control Period and Rationale
Average number of major/critical audit findings per site audit
Due to lower frequencies, consider at least a 2 year sample
Percentage of unreported, confirmed SAEs as compared to total SAEs as discovered through any method
Consider at least a 1 year sample
Significant Protocol Deviation rate per treated subject (total # of deviations/ total # of subjects for the protocol)
Average Monitoring (all types) cost per site
Due to fluctuations in costs and time value of money, consider at most a 1 year sample
Average interval between on-site monitoring visits per site
Median number of days from patient visit to eCRF data entry
Median number of days from query open to close
Median number of days from significant/major issue open to close

10. Guidance for Potential Surrogates
Metric
Potential Surrogates
Average number of major/critical audit findings per site audit
Percentage of unreported, confirmed SAEs as compared to total SAEs as discovered through any method
If a field does not exist on monitoring report, consider using data entry of SAEs vs. on-site monitoring visit date.
E.g. Calculate the number of SAEs with start date that were data entered >/= 2 days after an on-site monitoring visit and divide by the total number of SAEs
Significant Protocol Deviation rate per treated subject (total # of deviations/ total # of subjects for the protocol)
Consider defining “Significant” as those protocol deviations impacting primary or secondary endpoints.
Average Monitoring (all types) cost per site
If direct costs cannot be obtained, consider collaborating with finance to estimate
Also, consider determining average cost of visit and utilize decreased visit frequency to estimate
Average interval between on-site monitoring visits per site
Determine the interval between on-site monitoring visits for all sites and divide by the number of sites. If a site has not had a second visit to perform analysis in the quarter, omit that site from analysis
Median number of days from patient visit to eCRF data entry
Median number of days from query open to close
Median number of days from significant/major issue open to close

11. Establishment of Target Ranges (1 of 2)
Using historical controls as a guide, set ranges that will indicate “better”, “about the same” or “worse" since last quarter
If unable to calculate a historical control for any reason, the quantitative target ranges will have to be created using best judgment qualitatively
Example: Median Number of Days from Query Open to Close – historical control = 10 days
In this example, the metric, when compared to the target ranges, will inform the dashboard ranking
</= to 8 days would be better (e.g. could be top 10% or -2 standard deviations)
8-12 days would be about the same (e.g. could be mid – 80% or +/- 1 standard deviation)
>/= 12 days would be worse (e.g. could be bottom 10% or +2 standard deviations)
Historical Control = 10 days
Better
About the Same
Worse
8 days
12 days

12. Establishment of Target Ranges (2 of 2)
Example: Average number of major/critical audit findings per site audit – historical control = 2 findings
In this example, the metric, when compared to the target ranges, will inform the dashboard ranking
</= 1 finding would be better (e.g. could be top 10% or -2 standard deviations)
1-2 findings would be about the same (e.g. could be mid – 80% or +/- 1 standard deviation)
>/= 2 findings would be worse (e.g. could be bottom 10% or +2 standard deviations)
Note the range is tighter for this particular metric
If no audits have been performed, the metric would be 0 and per this target range the dashboard ranking would be better
Historical Control = 2 findings
Better
About the Same
Worse
1 finding
2 findings

13. These Metric Narratives illustrate the expectations of RBM’s impact as well as alternative observations
Metric
Expected Observations
Potential Alternative Observations
Average number of major/critical audit findings per site audit
Given RBM, it would be expected that the average number of major/critical findings per site audit will decrease
Early during implementation, findings may rise due to it being a new procedure not necessarily a focus on critical data and processes with expectation that they would decrease over time
Percentage of unreported, confirmed SAEs as compared to total SAEs as discovered through any method
Given RBM, it would be expected that number of unreported, confirmed SAEs will decrease
Early during implementation, findings may rise due to shift in focus from SDV to SDR with expectation that they would decrease over time
Significant Protocol Deviation rate per treated subject (total # of deviations/ total # of subjects for the protocol)
Given RBM, it would be expected that number of Significant Protocol Deviations will decrease
New process implementation for Protocol Deviation review may reflect unexpected increases

14. These Metric Narratives illustrate the expectations of RBM’s impact as well as alternative observations
Metric
Expected Observations
Potential Alternative Observations
Average Monitoring (all types) cost per site
Given RBM, it would be expected that average monitoring costs will decrease
Costs may remain flat until second quarter of analysis or later
Average interval between on-site monitoring visits per site
Given RBM, it would be expected that interval between on-site monitoring visits will increase
Average interval between on-site monitoring visits may remain flat until second quarter of analysis or later
Median number of days from patient visit to eCRF data entry
Given RBM, there are no expectations for the median number of days from patient visit to eCRF data entry, however, a decrease would be beneficial
This metric measures an unintended consequence of RBM, namely, the site’s delay in performing a crucial function that empowers central monitoring due to the potential decrease in on-site visits
Median number of days from query open to close
Given RBM, there are no expectations for the median number of days from query open to close, however, a decrease would be beneficial
This metric measures an unintended consequence of RBM, namely, the site’s delay in performing a crucial function due to the potential decrease in on-site visits
Median number of days from significant/major issue open to close
Given RBM, it would be expected that the median number of days from issue open to close will decrease
Early during implementation, findings may rise if issues management process is new to the organization with expectation that they would decrease over time

15. Final Considerations
Once the historical controls for your metrics are determined, they should remain static for the duration of analysis
As RBM becomes more pervasive in a company (“business as usual”), controls will switch from “non-RBM trials vs RBM trials” to your normal baseline for control, (e.g., previous calendar year)

16. Progression of RBM Uptake
Progression of RBM Uptake* at Member Companies Blinded Inventory of RBM Trials (planned and ongoing)
Voluntary adoption information is reported to a blinded third party for aggregation

17. RBM Metrics Context for Collection and Analysis
Collection process and limitations across 81 trials eligible to report data across 7 Companies for periods ending Q42015, Q12016 (out of inventory of 162 trials planned or in progress)
Analysis
Trial data with similar level of maturity (“RBM + x months”) is aggregated into stacked charts. In example below, the first quarter of metric data was grouped together into stacked charts(red circles), the second quarter of metric data was grouped together for analysis (green circles) and so on, regardless of actual calendar quarter.
Metric #1 Example
Trial 1
RBM + 3 Months
RBM + 6 Months
RBM + 9 Months
RBM + 12 Months
RBM + 15 Months
Trial 2
RBM + 3 Months
RBM + 6 Months
RBM + 9 Months
RBM + 12 Months
Trial 3
RBM + 3 Months
RBM + 6 Months
Note that not all metrics are reported for all trials across all time periods
Trial 4
RBM + 3 Months

18. How to read the metrics (stacked charts)
What do the X and Y axis’ represent?
The Y-axis (stacks and the numbers inside the stacks) represent the total number of trials reported by member companies (via voluntary reporting) that have utilized RBM methods during this report period.
The X-axis shows how long the trials have been running using RBM methods (e.g., “RBM + 3 months” = trial using RBM methodology for 3 months)
What do the colors mean?
Dark Blue indicates this metric for this reporting period for these trials is Better compared to the control
Light Blue indicates this metric for this reporting period for these trials is About The Same compared to the control
Yellow indicates this metric for this reporting period for these trials is Worse compared to the control
Refer to slides 6-7 for optional guidance defining “Better”, “About the Same” and “Worse”

19. Quality: Audit findings per audited site
Average number of major/critical audit findings per audited site
not all metrics are reported for all trials across all time periods

20. Quality: SAE reporting
Percentage unreported, confirmed SAEs as compared to total SAEs
not all metrics are reported for all trials across all time periods

21. Quality: Significant Protocol Deviations
Significant Protocol Deviation rate per treated subject
not all metrics are reported for all trials across all time periods

22. Efficiency: Overall Monitoring Cost
Average Monitoring (all types) cost per site
not all metrics are reported for all trials across all time periods

23. Efficiency: On-site visit interval
Average interval between on-site monitoring visits per site
not all metrics are reported for all trials across all time periods

24. Cycle Time: eCRF Entry
Median number of days from patient visit to eCRF data entry
not all metrics are reported for all trials across all time periods

25. Cycle Time: Query Open to Close
Median number of days from query open to close
not all metrics are reported for all trials across all time periods

26. Cycle Time: Issue Open to Close
Median number of days from issue open to close
not all metrics are reported for all trials across all time periods