1. Collaboration and Consensus – Standards Around Data, Definitions and Operational Workflow
Matt Wilson, RN
Director CEC-SS-ACL DCRI

2. Objectives: Standards Best Practices in adjudication? Data-Definitions
Operational workflow
Guiding principles
Start to Finish
eCRF
Charter
Triggers
Quality
Metrics that matter

3. The CEC Mission
Provide HIGH quality adjudicated endpoint data with scientific rigor, efficiency and innovation by coordinating and conducting Systematic, Consistent, Unbiased, Blinded, and Independent Clinical Events Adjudication.
Efficient
Accurate
Always Improving

4. Are we saying the same things?
What should we adjudicate?
Primary, Secondary, Events of Interest, Causality, Eligibility, etc.
How should we identify (trigger) events for adjudication?
Outcome pages, Lab data, Medical records, Free text, Hospitalizations, AE’s, SAE’s, etc.
What should we collect?
eCRF (patient profile report), DC summary, H&P, ER report, Lab slips, Image reports, EKG’s, Death Certificate, Autopsy report, Consult notes, MARS, etc.
Who are our partners?
Sponsor, CRO, ARO, Ops, Data, Stats, Reviewers, Sites, FDA, IRB, EC, SC, DSMB etc.
What is quality?
Key Risk Indicators, Currency, Traceability, Major Discordance, Missing data, Missing events, etc.
Who is monitoring the eCRF?
Site Management, CEC, Stats, Sponsor, Safety, Technology, etc.

5. Key data elements, GCP, CFR, HIPPA, GDPR, IRB
Do we have standards?
Key data elements, GCP, CFR, HIPPA, GDPR, IRB
Quality-KRI’s
Operational workflow
Data-Definitions

6. Guiding Principles - Defining Quality
Were all suspected events triggered as planned?
Were all events processed per charter procedures?
Were all adjudications data based accurately?
Were all QC procedures performed/documented with identified issues addressed?
Did any major SOP/WI/HIPAA/GCP/IRB violations occur?

7. Start to Finish! Protocol Charter eCRF design Triggers Edit Checks/DVC
+Efficacy vs Safety?
+Endpoint driven?
Charter
+Definitions?
+SD table
eCRF design
+QB/EVENT Pages
Triggers
+Clinical Trigger spec
Edit Checks/DVC
+Stats/DM partners

8. Definitions 2014 FDA Hicks KA et al: Circulation 2014 –
2012 Key Data Elements and Definitions for PAD –
Fourth Universal Definition MI –
TAVR VARC-2 Consensus document –

9. Source Documentation When is it too Much? When is it not enough?
“Fine balance between getting enough information and getting ALL the information to review a case”

10. Which way do we go? Source documentation vs. eCRF data
Source Documents
eCRF data
Comprehensive
Reduce site burden
Data Security (PHI considerations)
Efficiency (cycle times, cost)
Translations
Consistency (same questions)

11. Big Picture – what, when, where…
Recommended Event-Specific eCRF data/Source Documents
Death and Hospitalization Events
eCRF
Clinical Event Narrative
NT-pro-BNP results (if performed)
Procedure reports; PCI/CABG (if performed)
Right heart catheterization results (if performed)
Cardiac Markers with ULN (if performed)
Source Documents (tracked by CEC)
Discharge Summary (if hospitalized)
Autopsy Report (if performed)
Event ECG’s pre, during, post (if performed)
Neurology Consult notes (if performed)
Urgent Heart Failure Visits
Exam report
Treatment plan
Labs to confirm evidence of new or worsening HF
ER or clinic visit note
“We are quickly approaching an era of limited source data and a dependency on CRF data…the necessity of a good clinical narrative…is REALLY important…
conventions that the sponsor and/or FDA agree upon re: adjudication from CRF data only will be important”

12. Designing the Data Collection Process

13. eCRF Best Practice: Quarterback Page
Captures Key Components/ Critical Variables
Yes/No response required for each event type
Dynamically creates event page to collect further details when event is answered ‘Yes’
Through telephone or clinic visit
Recommend collecting no less frequently than every 3 months
Important in later clinical phases

14. eCRF Best Practice: Event/Endpoint Detail Page
Support the Endpoint Definitions
Measurable and Objective
Strategic “Net” to identify events
Complements source documents
Reduces source document collection
Pertinent Information
Consistent same questions
Chronology of care
Trigger on this page
Death, Stroke, Myocardial Infarction, Bleeding, Congestive Heart Failure, Acute Pancreatitis, Neoplasms….

15. eCRF Best Practice: CEC Relevant Pages
Hospitalization Page
Strategic monitoring strategy
Lab Value Pages
Cardiac Markers, Troponin I/T, CK & CK-MBs
Labs of Interest: HGB/HCT
Transfusions
Procedural Pages
Cardiac Catheterization (CATH)
Percutaneous Coronary Intervention (PCI)
SAE/AE Pages

16. No matter how good (or bad) your CRF is, if it’s flawed — you won’t get intended results.

17. Design and Implement an effective CRF (data capture and edit checks)
Routine QC checks (involve Stats, DM, CEC, Site Management)
Cross checks
Lab data suspicious of event, no event reported
Duplicate events (e.g. >2 in 1 week)
Reconcile SAE/AE (coded terms) to CEC triggered events
Negatively adjudicated events (follow up for potential SAE/AE)
Data that you cannot algorithmically assess: Free text review; Hosp, SAE/AE pages, SAE labs)
2 Large ACS trials, DCRI methodology applied (possible Missed MI’s)
41,965 lab records, 2,247 possible triggers, 29 triggered & confirmed
10,269 hosp records, 370 possible triggers, 49 triggered & confirmed
Source document review (Medical records submitted for adjudication, review for SAEs/additional endpoints)

18. Operational Workflow –
CEC project staffing (Clinical team)? CEC PI, Members, PL, CTC, CTA, CDS
Who trains who? CEC PI, CTC
What are we training our CEC teams on? Protocol, Charter, Conventions, etc.
How do we adjudicate? Phase 1, Phase 2, Single Clinician Review, Third Reviewer
What qualifies a reviewer to adjudicate? TA/Clinical experience, CEC experience
What % of cases reach committee? >30% retrain, assess for issues
What if CEC is receiving more triggers than expected? Check your net, patient demographic
Are the sites submitting the right SD in a timely manner? 2 weeks
Are the sites responding to queries in a timely manner? 2 days
Query escalation plan in place? Swat Team approach with CRAs, CEC, Sites, Sponsor
Are there calls between site management and the CEC coordinators to discuss what’s needed on the more difficult cases?
Who is responsible for cleaning the CRF? Site management

19. Metrics that matter Currency % Query (10-15%) % Complete (80-90%)
Cycle Times
Query to Clean (10-30 days)
Clean to Complete (5-10 days)
Discordance (Major disagreements)
Site vs CEC (10-20%)
CEC vs CEC (10-20%)

20. Adjudication cycle times – how are we doing?
Event date to triggered*
10 days
Triggered to assigned to reviewers
15 days
Assigned to reviewers to case complete
5 days
Adjudication cycle times – how are we doing?
30 days total
Set targets:
Query to Adjudication <30 days
x-days, escalate
Adjudication to Complete <10 days
x-days, escalate
* The event date to triggered can be dependent on visit windows

21. Adjudication Completeness “Currency” – Gold 80%, Platinum 90%, Executive Platinum 95%
Event Types
# Events Triggered
# at Query
# at Phase 1 Physician Review
# Ready for Committee Review
# at QC
# at Re-Review
# at No Action Needed
# at Complete + # at No Action Needed
# at Complete
Bleed
219 23 15 3 4
178
174
Death 36 8 2 1 25 MI 6 18 16
Stent Thrombosis 14
Stroke 12 9
Totals
308 39 20 5
244
238
Troubleshoot problems: queries, translations, reviewer delays. Know where to push!

22. Do I have enough to adjudicate the case?
Recommend a joint Ops/Reviewer strategy
1.) Ops tracking
Quantify what is in the cases
Assist with query process
Active/End of trial metrics “what cases were missing x?”
2.) Reviewer assessment
Answers the key question – “did I have enough?”

23. Insufficient Source Documentation question and follow-up:
Entered by reviewers
“reviewer form”
Entered by querying team
“tracking page”

24. CEC Key Risk Indicators (KRI’s)
time between event occurred and event triggered
time between "in query" to “ready for adjudication"
time between Phase 1 and a) final outcome or b)phase 2 [overall]
time between Phase 1 and a) final outcome or b)phase 2 [by adjudicator]
time between Phase 1 disagreement and Phase 2
ratio of # agreement/#disagreement by adjudicator
# documents received/# documents expected by site & event type
# agreement events [sites and CEC]/#total events [sites and CEC] and event type

25. Design/Implement plan early on, include key stakeholders (FDA, Sponsor, EC)
Methods used require validation steps
Documentation is key (project plan, trigger spec, charter)
Involve SMEs (CEC, Stats, DM, SM)
Develop and Implement plan early on (startup/FPI) to avoid the following:
Missed events
Difficulty collecting data retrospectively
Limit free text fields and train sites on effectively capturing clinical context in eCRF narrative
Integrate your CEC and Safety teams for “Cross Talk”
Real time reconciliation of reporting (CEC/Safety)