Instruction in The Protection of Human Research Subjects A web based model Paul Braunschweiger Ph.D. Department of Radiation Oncology Director, Office of Research Education University of Miami Co-Founder, CITI Program

Talking Points  PHS HSR Training mandate  Why the web? –Pros and cons –How to get started  CITI Program as an example of a web based platform –CITI Program Overview –How is the CITI course site is used  DL Software Demo and Admin utilities  Evaluation of the web based program

June 2000 DHHS Mandates Training in Human Subjects Protection N=6559 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% TOTALLY UNNECESSARY ABSOLUTELY VITAL On a scale of , how necessary is the PHS requirement that all key personnel engaged in human subjects research must participate in a human subjects protection educational program? Mean

Strongly disagree strongly agree 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% % of Responders (729) In my opinion, the June 2000 Federal education mandate HAS raised the level of awareness of investigators and staff at my institution to the vulnerabilities of human subjects in research. N=764, 22 months June 2000 DHHS Mandates Training in Human Subjects Protection Mean

CITI – Startup Organizations  University of Miami –Paul Braunschweiger,Ph.D., Ken Goodman, Ph.D.  Fred Hutchinson Cancer Research Center –Karen Hansen, BA  Albany Medical Center –Jeff Cooper, MD  Children's Hospital-Boston –Susan Kornetsky, MPH  Dartmouth College –Liz Bankert, MA  University of Kentucky –Ada Sue Selwitz, MA, Norma Epley, MS  The National Comprehensive Cancer Network –Diane Paul, MS  University of Nebraska –Bruce Gordon, MD, Ernie Prentice, Ph.D.  Group Health Cooperative –Barbara Young, Ph.D., Janelle Erickson, MPH  University of Washington –Helen McGough, MS

Web based instruction  Basic instruction  Refresher courses. –Economy of time for investigators –Economy of time for trainers. –Economy of time for administrators. –Cost efficient for institutions hrs/session >$10,000/session20 2 hrs/session >$10,000/session $50,000 for a 4hr tutorial – Thousands of learners.$50,000 for a 4hr tutorial – Thousands of learners. CITI is $1,000 / institution for unlimited usage.CITI is $1,000 / institution for unlimited usage. NCI tutorial is “Free”NCI tutorial is “Free”  Flexible –Rather easy to update and modify –CITI reviews and updates semiannually  Learners like using the web for instruction. –Some like it less

% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% % of Responders AudioVideoClassroomPaper No Change Assuming an equal time commitment, would you prefer any of the following to this web-based program? N=1132

Web based instruction What do you need to get started from scratch.  Content (html) and content providers –Content reviewers  Distance Learning Software and support  Servers and IT support.  Administrative support –Help Desk

Useful Features of most DLS Packages  User ID and Password protected.  HTML, slideshow +/- audio, video presentations  Linear presentation of course content  Hyperlinks to other information sources on the web  Assessment of achievement = quiz –Feedback –Minimum level of understanding  Records and stores data to document compliance.  Administrator access to the database to track usage and compliance.  Certificate of achievement

Is there a downside to web based instruction?  Learning styles vary. –Some people would rather listen than read. –Little or no opportunity for real time question and answer period.  Web Navigation can be frustrating for some users. –browsers, ISPs, internet, bandwidth, software issues.  “Gaming the software” – Cheating –Hard to prevent but, not impossible.  Development costs might be prohibitive for small organizations. –Multi-institutional agreements

A web based model The Collaborative Institutional Training Initiative (CITI)

What is the CITI Program?  CITI is a volunteer organization whose goal is to develop and distribute high quality, peer reviewed educational resources designed to raise awareness to human subjects protection issues for all members of the research team.  CITI maintains an educational website where participating organizations can design and roll out a customized curriculum for their investigators, staff and administrators.

CITI – Organization Participating Institutions and Organizations (~580) CITI Developer Group (~40) CITI Editorial Board (15) CITI Administration – Office of Research Education, University of Miami

Semi-annual Developer Meetings Seattle Florida

CITI Institutional Members (578) Member Category No.% VA15727% Academia19333% Healthcare Organizations 5510% Hospitals / Med Centers 468% Research Institutes 387% D of Energy 203% Military / DoD 203% Industry132% Independent IRBs 92% International152% Others122%

International Participants

% of AAHRPP accredited institutions utilize the CITI program as a component of their HSR educational  Cedars-Sinai Medical Center  Chesapeake Research Review Inc.  Children's Hospital Boston  Copernicus Group IRB  Dana-Farber/Harvard Cancer Center  East Tennessee State University  Fox Chase Cancer Center  Hunter Holmes McGuire Veterans Affairs Medical Center  Iowa Health - Des Moines  Massachusetts General Hospital  Brigham & Women's Hospital  Michigan State University  New England Institutional Review Board  Newton-Wellesley Hospital  Saint Francis Hospital and Medical Center  The University of Iowa  The University of New Mexico Health Sciences Center  The University of South Dakota  University of Arkansas for Medical Sciences  University of Louisville  University of Maryland, Baltimore  University of Minnesota  University of Pittsburgh  Vanderbilt University  Washington University in St. Louis  Western Institutional Review Board

CITI program and the VA  Financial support  Software  Software development  Software and IT support. Special Thanks to Drs. Joan Porter and Mike Fallon

CITI - The Content  CITI Basic Course –12 Biomedical Modules –11 Social / Behavioral Research Modules – 5 General Interest Modules –Quizzes  CITI Refresher Course –Case studies and scenarios –Biomed Refresher I, –Biomed Refresher II –SBR Refresher I –SBR Refresher II (6-2006)  CITI International Research Course –Chinese, Spanish, –Arabic, Portuguese, French  Good Clinical Practice (GCP) Spring 2006  HIPAA Summer 2006  RCR – Summer 2006  Lab Animal Welfare - Summer

CITI – Program Review  The CITI Content is extensively reviewed and critiqued. –CITI Developers –CITI Editorial Board –External peer reviewers –Professional editorial services (LERA Group, PTC of NY) –Our Participants via user satisfaction surveys Quiz questionsQuiz questions Recommendations for new course contentRecommendations for new course content RCRRCR

CITI Program The Participating Institution:  Registers with CITI –Institutional user fee, unlimited access  Establishes Learner Groups –Biomedical Research PI’s, IRB Members, Administrators, Lab Researchers, StudentsPI’s, IRB Members, Administrators, Lab Researchers, Students –Social & Behavioral Research PI’s, IRB Members, Administrators, Interviewers, StudentsPI’s, IRB Members, Administrators, Interviewers, Students  Designs a specific curriculum for each Learner Group –Choose “required” and “optional” modules –Set passing score –Set recertification requirement timeline.  Provides additional organization specific instructional materials. –The Institutional Page

How do member organizations use the CITI Program?  Investigator certification.  IRB member training.  IRB coordinator training  Refresher course, re-certification, continuing education.  Web based alternative to seminars or lectures.  Undergraduate / graduate bioethics or research methods courses.

Coordinator “Survey says”: N=248, 64%

CITI “Help Desk”  Resolve learner problems –Registrations –Technical Issues  Institution set up support –Learner Groups –Program Curriculum –Institutional page setup –Adding institutional content  Coordinator Support –Changes in contact information –Implement Modifications. –Report Utilities

The Target Audience

The Clinical Investigator The Anatomy Lecture of Dr. Nicolaes Tulp, Rembrandt 1632

The Social & Behavioral Scientist “The Psychologist” by Charles Bragg

The Basic Science Investigator Florescence In Situ Hybridization

Students “The Student” by Rembrandt

The IRB Frans Hals, Regents of the St Elizabeth Hospital of Haarlem

The IRB Coordinator The Tribunal of the Brabant Mint in Antwerp 1594, by Marten de Vos 1523 – 1603 J ustice triumphs over deceit and violence

Chair of the IRB Hans HOLBEIN, the Younger, Henry VIII and the Barber Surgeons c. 1543

The Institutional Official

0.0%5.0%10.0%15.0%20.0%25.0%30.0% % of The Survey Responders Student PI Clinical Investigator Other Research Assistant Study coordinator or Study nurse Interviewer IRB Member or Coordinator Lab Researcher Administrator Data manager Social Worker Role in Human Subjects Research n=1130 CITI Learner Characteristics

Registers Chooses Learner Group User Survey Institutional Trainer Completes the Required modules Multiple sessions Prints / Saves Completion Report Completes any Optional modules CME / CEU Log Off Refresher Notification Database Archive Username & Password Software must be user friendly The Learner

Software must be user friendly Institutional Coordinator / Administrator  Has access to the database –Download aggregate training data. –Retrieve individual learner’s record. –Review user comments.  Receives individual completion report by .

CITI Course Site Usage

CITI Program Growth PARTICIPATING ORGANIZATIONS 1-Aug Apr Aug-0213-Aug-0315-Aug-04 1-Aug-05 CITI PARTICIPANTS 2000 TO jJan-06

Recent Monthly Usage January 2005 March May July September November January 2006 NEW LEARNERS NEW LEARNERS Basic course = 88% Refresher course = 12% Registrants Completion Reports 87% Completion Rate

International Access Data from January 2006

CITI Cumulative Registrations Aug-00 Oct-00 Dec-00 Feb-01 Apr-01 Jun-01 Aug-01 Oct-01 Dec-01 Feb-02 Apr-02 Jun-02 Aug-02 Oct-02 Dec-02 Feb-03 Apr-03 Jun-03 Aug-03 Oct-03 Dec-03 Feb-04 Apr-04 Jun-04 Aug-04 Oct-04 Dec-04 Feb-05 Apr-05 Jun-05 Aug-05 Thousands Registrations nov-05 Jang Version 6 Version 5 Version 4 Version 3 Ver 1 & 2

% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% 50.0% Biomedical Social Behavioral Both Which "Basic" Course did you complete? % of the Responders

CITI Program Course Site Administrator tools

Program Evaluation What is the effectiveness of web based instruction. IRB Coordinator Perspective

Utility: Coordinator “Survey says”:

Acceptance by learners Coordinator “Survey says”:

Impact: Coordinator “Survey says”: 70%

Value: Coordinator “Survey says”:

Value: Coordinator “Survey says”:

Program Evaluation What is the effectiveness of web based instruction. Learner’s Perspective

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% % of Responders Easy Difficult Rate the ease of the registration procedure. Utility

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% % of Responders Easy Difficult Rate the ease of navigating the course. Utility

0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 16.0% % of Responders Hrs spent completing the course Approximately how many hours did you need to complete the Course? Usage

0% 5% 10% 15% 20% 25% 30% 35% % of Responders Strongly Agree Strongly Disagree Having completed the course, I now have a clearer understanding of human research subjects protection. Mean = 8.5 N= 1176 Impact

Impact 0% 5% 10% 15% 20% 25% % Responders Strongly agree Strongly disagree Having completed the CITI course, I am now more capable of advising a colleague on issues regarding human subjects protection. Mean = 7.7 N=1170

Since completing the CITI Course, I may be more willing to join an IRB or a Data Safety Monitoring Board to take a more active role in protecting human subjects. 0% 2% 4% 6% 8% 10% 12% 14% 16% % of Responders Strongly Agree Strongly Disagree 34% N=1146 Mean = 6.1 Impact

Average Response 1 = Strongly disagree; 10 = strongly agree Both Biomed and SBR Biomed Only SBR Only Students Clinical researcher PI IRB Types Males Females Overall The time spent doing the course was well justified. ** ** * * value

Rating 1-10 Both Biomed and SBR Biomed Only SBR Only Students Clinical researcher PI IRB Types Males Females Overall * * ** On a scale of 1 to 10, how would you rate the CITI Course, overall? value

Program Evaluation Learner Perspectives at 22 months

REFRESHER COURSE SURVEY no 0% 20% 40% 60% 80% 100% % OF THE RESPONDERS yes If you did complete the CITI "Basic Course", do you think the information in the Basic Course had a positive impact on how you have been conducting your human subjects research? N=549, AVERAGE TIME AFTER COPLETING BASIC COURSE = 22 MONTHS

REFRESHER COURSE SURVEY 0% 5% 10% 15% 20% 25% 30% % OF THE RESPNDERS The CITI Basic Courses, provided me with a knowledge base to protect human subjects from harms associated with being in a research project? STRONGLY DISAGREE STRONGLY AGREE N=549, AVERAGE TIME AFTER COMPLETING BASIC COURSE = 22 MONTHS

REFRESHER COURSE 0% 5% 10% 15% 20% 25% 30% % OF RESPONDERS The CITI Basic Course, helped me to a clearer understanding of the elements of consent. STRONGLY DISAGREE STRONGLY AGREE N=549, AVERAGE TIME AFTER COPLETING BASIC COURSE = 22 MONTHS

Refresher Course Survey 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% % of Responders (729) I am now reading more literature or going to more seminars on topics related to human subjects protection STRONGLY DISAGREE STRONGLY AGREE 40%

Refresher Course Survey 0% 10% 20% 30% 40% 50% 60% 70% 80% YESNO I have taken a more active role in protecting human research subjects by joining an IRB, a DSMB or by pursuing IRB professional certification. % of Responders N=766

REFRESHER COURSE SURVEY Strongly disagree strongly agree 0% 5% 10% 15% 20% 25% 30% 35% % OF RESPONDERS The CITI Refresher Course is a worthwhile component of my institution’s program to protect human research subjects. N=549

In Summary What have we learned about web based instruction from the CITI program experience.  The www as a tool –Web instruction can provide rich content in a time and cost effective manner. –A web program can play a central role in an organization’s HSR education program. –A web program can be utilized for both basic and refresher instruction. –A web platform is overwhelmingly preferred by investigators and training coordinators. –But, alas, the web is not for everyone.. and alternatives are advisable.

In Summary What have we learned from the CITI program experience.  Organization –A dynamic committed organization can keep the instruction fresh and current. Frequent updates.Frequent updates. Learner satisfaction surveys are very important to obtain feedback and guidance to improve the program.Learner satisfaction surveys are very important to obtain feedback and guidance to improve the program. –Help desk

In Summary What have we learned from the CITI program experience.  Utility –World wide training resource can be delivered in a cost effective and value added manner. –Customization capabilities are a key. –Institutional page to provide institution specific materials is VERY well received and a extensively used by some participating institutions.

In Summary What have we learned about web based instruction from the CITI program experience.  Evaluations –Learners indicate the course helped them understand human subjects protection issues and that the time required to complete the course is time well spent. –Learners indicate that the instruction has provided the confidence to take a more active roll in the protection of human subjects. –Institutional coordinators exhibit a high degree of satisfaction with the web based program.

Instruction in The Protection of Human Research Subjects Paul Braunschweiger Ph.D.