Intermediate methods in observational epidemiology 2008 Quality Assurance and Quality Control

Threats to Causal Inference in Epidemiologic Studies Confounding Experimental Design Adjustment/Control ThreatSolution Bias Quality Assurance Quality Control

QA: Activities to assure quality of data that take place prior to data collection (through protocol and manuals of operation) QC: Efforts during the study to monitor the quality of data at identified points during the collection and processing of data Definitions of Quality Assurance and Quality Control

STEPS IN QUALITY ASSURANCE (1) Specify hypothesi(e)s (2) Specify general design -- develop protocol (3) Select or prepare data collection instruments, and develop procedures for data collection/ processing -- develop operation manuals (4) Train staff -- certify staff (5) Using certified staff, pre-test and pilot study instruments and procedures. In the pilot study, assess alternative strategies for data collection- eg, telephone vs. in-person interviews (6) Modify (2) and (3) and retrain staff on the basis of results of (5)

(1) Specify hypothesi(e)s (2) Specify general design -- develop protocol (3) Select or prepare data collection instruments, and develop procedures for data collection/ processing -- develop operation manuals (4) Train staff -- certify staff (5) Using certified staff, pre-test and pilot study instruments and procedures. In the pilot study, assess alternative strategies for data collection- eg, telephone vs. in-person interviews (6) Modify (2) and (3) and retrain staff on the basis of results of (5) Based on a “grab” sample STEPS IN QUALITY ASSURANCE

(1) Specify hypothesi(e)s (2) Specify general design -- develop protocol (3) Select or prepare data collection instruments, and develop procedures for data collection/ processing -- develop operation manuals (4) Train staff -- certify staff (5) Using certified staff, pre-test and pilot study instruments and procedures. In the pilot study, assess alternative strategies for data collection- eg, telephone vs. in-person interviews (6) Modify (2) and (3) and retrain staff on the basis of results of (5) Based on a sample as similar as possible to the study population

STEPS IN QUALITY ASSURANCE (1) Specify hypothesi(e)s (2) Specify general design -- develop protocol (3) Select or prepare data collection instruments, and develop procedures for data collection/ processing -- develop operation manuals (4) Train staff -- certify staff (5) Using certified staff, pre-test and pilot study instruments and procedures. In the pilot study, assess alternative strategies for data collection- eg, telephone vs. in-person interviews (6) Modify (2) and (3) and retrain staff on the basis of results of (5)

QUALITY CONTROL PROCEDURES: TYPES 1. Observation monitoring “Over the shoulder” observation of staff by experienced supervisor(s) to identify problems in the implementation of the protocol. Example: - Taping of interviews

-Random repeat (phantom) measurements based on either internal or external pools (biologic samples) to examine:. Intra-observer. Inter-observer Advantages. Better overall quality of data. Measurement of reliability variability

Phantom sample based on an internal pool Internal phantom sample STUDY BASE BLOOD SAMPLES OF 7 PARTICIPANTS Aliquot 2: measurement in study lab Aliquot 1: measurement in gold standard lab

Aliquot 2: measurement in study lab Phantom sample based on an external pool Phantom sample from the gold standard lab STUDY BASE BLOOD SAMPLES OF 7 PARTICIPANTS

- Monitoring of individual technicians for deviations from expected values Example: monitoring of digit preference for blood pressure (expected: 10% for each digit)

Digit Preference in Systolic Blood Pressure (SBP) Measurements

Quality Control Indices Validity (Accuracy) Precision (Repeatability, Reliability)

Validity : Usually estimated by calculating sensitivity and specificity. The study (observed) measurement (“test”) is comparedwith a more accurate method (“gold standard”). When clearcut gold standard not available: “inter-method reliability” Problem: Limited to 2 x 2 tables

...Thus, traditional reliability indices (e.g., kappa, correlation coefficient) can be also used to estimate validity of continuous variables or variables with more than 2 categories Gold Standard results Study results

Reliability: Sources of Variability Measurement Error –Instrument/Technique/Lab –Observer/Technician Intra-observer Inter-observer Intra-individual (physiologic)

Blood collected from an individual (1 st measurement) To examine within-technician variability? Aliquot 1.2: Lab determination done by same technician Aliquot 1.2: measurement done by same technician in a masked fashion To measure within-individual variability? Blood collected from the individual (replicate measurement) Repeat blood collection in same individual X time later To examine between-lab variability? Send Aliquot 1.3 to a different lab Aliquot 1.3: Lab determination done at a different lab Time Design of a study to evaluate sources of variability (Based on Chambless et al, Am J Epidemiol 1992;136: ) For other sources of variability, use phantom samples Phantom sample Aliquot 1.2 Aliquot 1.3 Aliquot 1.1: Study lab determination Aliquot 1.4 To examine between-technician variability? Aliquot 1.3: Lab determination done by a different technician at study lab Aliquot 1.2: measurement done by a different technician in a masked fashion at study lab

Indices of Reliability (also used for validity) % differences between repeat measurements (expected if no bias: ½ positive and ½ negative) % observed agreement Kappa Correlation coefficient Coefficient of variation Bland-Altman plot

Indices of Reliability (also used for validity) % differences between repeat measurements (expected if no bias: ½ positive and ½ negative) % observed agreement Kappa Correlation coefficient Coefficient of variation Bland-Altman plot

Agreement Between First and Second Readings to Identify Atherosclerotic Plaque in the Left Carotid Bifurcation by B-Mode Ultrasound in the ARIC Study (Li et al, Ultrasound Med Biol 1996;22:791-9) Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading Percent Observed Agreeement: [ ] ÷ 986 = 88% Shortcomings Chance agreement is not taken into account If most observations are in one of the concordance cell(s), % Observed Agreement overestimates agreement

Agreement Between First and Second Readings to Identify Atherosclerotic Plaque in the Left Carotid Bifurcation by B-Mode Ultrasound in the ARIC Study (Li et al, Ultrasound Med Biol 1996;22:791-9) Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading Percent Observed Agreeement: [ ] ÷ 986 = 88% Shortcomings Chance agreement is not taken into account If most observations are in one of the concordance cell(s), % Observed Agreement overestimates agreement

Indices of Reliability (also used for validity) % differences between repeat measurements (expected if no bias: ½ positive and ½ negative) % observed agreement Kappa Correlation coefficient Coefficient of variation Bland-Altman plot

Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading The most popular measure of agreement: Kappa Statistics P O Observed agreement proportion P E Expected (chance) agreement proportion

Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading P O = [ ] ÷ 986 = 0.88 Kappa Statistics

Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading P O = [ ] ÷ 986 = 0.88 Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total: Kappa Statistics

Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading P O = [ ] ÷ 986 = 0.88 Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total: Kappa Statistics

Total Normal Plaque TotalNormalPlaqueSecond Reading First Reading P O = [ ] ÷ 986 = 0.88 Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total: Kappa Statistics Shortcomings Kappa is a function of the prevalence of the condition Can be calculated only for categorical variables (2 or more) Maximum agreement not due to chance Agreement not due to chance PE = [(209 x 192) + (777 x 794)] ÷ = 0.68 Thus, kappa values obtained from different populations may not be comparable

Interpretation of Kappa values (Altman & Bland, Statistician 1983;32:307-17) VERY GOOD GOOD MODERATE FAIR POOR

Indices of Reliability (also used for validity) % differences between repeat measurements (expected if no bias: ½ positive and ½ negative) % observed agreement and % observed positive agreement Kappa Coefficient of variation Bland-Altman plot

Coefficient of variation (CV) General definition: Standard Deviation (SD) as a percentage of the mean value

Calculation of the Coefficient of Variability X i1 and X i2 = values of repeat measurements on same lab sample X i = mean of these measurements For each pair of values: The mean overall CV over all pairs is the average of all pair-wise CVs and For each pair of repeat measurements:

Example of Calculation of the Coefficient of Variation - I Phantoms 1 2 Replicates (e.g., 2 different observers, 2 measurements done by same observer, 2 different labs, etc.) PAIR No k......

Pair (Split samples) No. 1: Measurement of total cholesterol Measurement No. 1 (X 11 )= 154 mg/dL Measurement No. 2 (X 12 )= 148 mg/dL V 1 = ( ) 2 + ( ) 2 = 18 mg/dL Phantoms 1 2 Replicates PAIR No. 1 Do the calculations for each pair of replicate samples Mean= [ ] / 2= 151 mg/dL Example of Calculation of the Coefficient of Variation - I Repeat the calculation for all pairs of measurements and calculate average to obtain overall CV

AnalyteIntra-Class Correlation Coefficient* Coefficient of variation (%)** Total serum cholesterol HDL HDL Reliability in the ARIC study (Am J Epi 1992;136:1069) *Best: as high as possible **Best: as low as possible

Indices of Reliability (also used for validity) % differences between repeat measurements (expected if no bias: ½ positive and ½ negative) % observed agreement and % observed positive agreement Kappa Coefficient of variation Bland-Altman plot