Group 5: Historical control data Follow-up of 2005 IWGT where the use of historical control data in interpretation of in vitro results was identified as an important topic, but was not discussed in depth because of lack of time. A short session took place in Basel to further discussed this topic. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

Assigned Participants Kerry Dearfield Makoto Hayashi (Chair) David Jacobson-Kram: not able to attend Peter Kasper David Lovell (Co-chair) Hans-Joerg Martus Veronique Thybaud (Rapporteur) 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

Objectives The group mainly discussed negative control data. Historical control data: - Until now used for acceptance criteria. - More and more used also for interpretation of results. Up to now there is no clear guidance. Objectives of the group are: - To provide generic advice for historical control data that could be applied to all assays, not to describe assay specific recommendations. - To make consensus on the following questions: What are the historical control data? How to make the historical control data? How to use the historical control data? 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

Quite helpful, formed the basis of initial discussion Statements still acceptable, but needed to be widened to include full range of assays including in vitro ones. An example of use for in vivo assays, i.e. data distribution and the use of quality control charts to explore changes over in negative control data in a large set of in vivo MNT data generated in a same laboratory.

What are the historical control data? -1- It is the complete set of data from previous experiments. It includes both individual values and the means of the negative control groups of that experiment. Individual values: For in vivo assays: animals For in vitro assays: cultures, plates All experiments must have a concurrent negative control and all concurrent negative controls should be added to derive at a representative negative historical control. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to build up the historical control data? -1- When you are building up historical control data you put in all data that have been accumulated over the past. Ideally data should be accumulated in order to visualize the distribution for individuals and the group means, and to calculate the key parameters to be used for data analysis. Graphical control charts may be useful for exploring the data. Data should only be excluded from the data set when there is a clear and valid reason to reject it. Reporting only a range (i.e., minimum, maximum) of data from the historical studies is not adequate. The data set should be managed so that more recent data carry more weight that older data. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to build up the historical control data? -2- The choice of the specific statistics such as mean, median etc. for the end-points to be assessed in historical control data should be determined by the assay community, based upon their experience on the most appropriate parameters. Historical control data set should ideally be as big as possible but probably comprise at least 10 separate experiments, but preferably 20. However, this may depend upon the use of historical data (e.g., acceptance criteria, distribution, internal quality control), on the variability within an assay, and also on the practicability (number of assays per year, etc). 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to build up the historical control data? -3- The experimental protocol must have remained fixed throughout the period covered by the historical data and the current experiment, unless it can be demonstrated that the change does not impact the values. - The method of scoring must be unchanged during the relevant period. - The experimental units must be comparable throughout the period. - There must exist no known systematic differences between the various control groups, current and historical, that would produce systematic differences in response, e.g., the data must have been gathered by the same investigators within the same laboratory. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to build up the historical control data? -4- Outliers: May need more work. Criteria for exclusion: - Should not be excluded just because unusual. - Should only be excluded if scientifically justified, as could be part of assay variability. If decisions are only based on ranges, outliers have disproportionate effect, that is one reason why the use of ranges is not recommended, and confidence interval type approaches are preferred. Values should be consistent with data reported in literature. Acceptable ranges should be defined for each assay by expert group, e.g., as already done for MLA. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use the historical control data? -1- Two distinct approaches to addressing negative historical control data: - To assess whether the concurrent control data is consistent with previous negative control data collected by that laboratory. Each assay will have its own specific acceptance criteria. - To assess within a laboratory whether the techniques used are still in control based upon a series of negative controls results. This can be assessed by formal quality control. The historical control is very important for the technical establishment and familiarization of the assay at the laboratory. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use the historical control data? -2- Concurrent negative controls are essential: - As part of the acceptance criteria for the evaluation of the technical acceptance of the study. - For appropriate statistical treatment of the data. - To accept a test (by comparing concurrent negative control values to the historical negative control). - To help assess the biological relevance of experimental data (e.g., statistically significant increases detected, but values comparable to historical control data). Nevertheless, it can be considered as a two step process: -First compare the experimental data with the concurrent negative control, -then consider if meaningful as compared to historical data distribution. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use the historical control data? -3- To design an assay (1): Assay/studies should be designed (number of duplicates, number of cells scored, etc.): - To detect a biologically important increase over the background. - To detect a predetermined effect size/ difference from background (test specific) of the observed endpoint at an a-level of 0.05 and a power of 80% was considered to be an easily achievable minimum standard. - Based on the a-level (type I error) for recognizing a predetermined increment over the spontaneous level with an acceptable statistical power (1-type II error). In some assays doubling is used but this may not mean the same thing biologically for all assays and depends on the background. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use the historical control data? -4- To design an assay (2): It should be generally required that a clear description of the design of the experiment including the power of the test, the statistical methods used to analyze the data and the level of significance actually achieved is given in every data report. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use historical control data? -5- How do you reject a control group, and therefore the experiment, from an experiment because it is unacceptable: - The group advice against the use of range for this acceptance limit criteria (see earlier). - These could be based upon either the observed spread of data or on calculated from the distribution values using (say) 2 or 3 standard deviations or upper 99% percentile from the mean of that which was actually observed. - The choice of criteria has to be determined for the individual assay based upon the specific characteristics of the assay. Each assay determines its own acceptance/limit criteria which is consistent with historical data generated by that scientific community. For example, could be useful to define performance standard and acceptable range or interval confidence for all assays as done before for MLA, or 5% for CA. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use of historical control data? -6- If a negative control group in an experiment is outside the acceptance criteria the experiment is generally no longer valid. How far outside the acceptance criteria is still valid should be managed for each assay and should be consistent with values reported in scientific community. However some use may be made with the information of the experiment if for instance the treated group show a clear increase. But care is needed in case such an increase is also a result of artifact such as problem with the culture in an in vitro study that could have impacted all cultures and the whole experiment. In case of negative results you would have to repeat the experiment. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

How to use of historical control data? -7- Historical control data may have a role in defining the size of effect that is considered biologically important. Such considerations may have a role in the future in putting dose-response relationship seen in experimental studies into perspective in the context of the identification of practical or pragmatic threshold. Considerable care would be needed in the choice of data to use in such assessment and more work is probably needed to assess the potential of this approach. 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data

Thank you 5th IWGT, Basel August 17-19, 2009 Group 5: Historical control data