Performance of Growth Models for Salmonella and Other Pathogens Thomas P. Oscar, Agricultural Research Service, USDA, Room 2111, Center for Food Science and Technology, University of Maryland Eastern Shore, Princess Anne, MD 21853; ; (fax); INTRODUCTION The prediction bias (B f ) and accuracy (A f ) factors of Ross 1 are the most widely used measures of model performance. However, B f does not detect some forms of prediction bias, B f and A f are mean values that are subject to bias by outliers and prediction cases involving no growth are excluded from calculation of B f and A f resulting in an overestimation of model performance. Thus, the objective of this study was to develop a method for evaluating model performance that overcomes the limitations of B f and A f. MATERIALS AND METHODS Published response surface models for lag time ( ) and maximum specific growth rate (  max ) of Salmonella Typhimurium in broth 2 or on sterilized, cooked chicken breast burgers 3,4 were evaluated for the ability to predict the data used to develop them (verification) and to predict data not used in model development but that were inside (interpolation) or outside (extrapolation) the response surface. Data for performance evaluation were collected with the same strain, previous growth conditions and modeling methods so as not to confound the comparison of observed and predicted values. Performance evaluation for interpolation and extrapolation. Independent data for performance evaluation of interpolation were collected with the same strain, growth media and modeling methods but different combinations of the independent variables that were within the response surface of the model. Independent data for performance evaluation of extrapolation were collected in the same manner except that the growth media used to measure growth kinetics was different and thus, the response surface models were evaluated for the ability to extrapolate to a different growth medium. Published data for other pathogens were also used to develop the performance evaluation method. Acceptable prediction zone method. Plots of B f for individual prediction cases versus predicted and  max were evaluated for acceptable prediction bias and accuracy using an acceptable prediction zone from a B f of 0.7 (fail-safe) to a B f of 1.15 (fail-dangerous). The acceptable prediction zone was wider in the fail-safe direction because greater prediction error can be tolerated in this direction when using models to predict food safety. The proportion of B f inside the acceptable prediction zone (pB f ) was calculated and used as a new measure of model performance. RESULTS AND DISCUSSION There is currently no consensus as to what mean values of B f and A f constitute a model that provides acceptable predictions of pathogen growth in broth or on food. However, for growth rate a mean B f in the range of 0.7 to 1.15 has been proposed as being acceptable 5. In the current study, all mean B f were in this range except for extrapolation of broth Model 1 to cooked chicken thigh burgers, which had a mean B f of 1.17 (Table 1). In general, mean A f increases by 0.1 to 0.15 per independent variable in the model 5. Thus, models with two independent variables, such as Models 3 to 6 in the present study, would be expected to have mean A f of 1.2 to 1.3 and models with three independent variables, such as Models 1 and 2 in this study, would be expected to have mean A f of 1.3 to All of the models evaluated in the current study had mean A f that fell below or in these expected ranges (Table 1). A limitation of mean B f as a performance factor is its inability to detect some forms of prediction bias such as under prediction in one region of the response surface and over prediction in another region of the response surface 5. For example, in the current study, a mean B f of 1.01 (Table 1), where one indicates no average bias, was obtained for extrapolation of broth Model 1 to cooked chicken breast burgers when upon graphical analysis of B f for individual prediction cases it was discovered that this model provided overly fail-dangerous predictions at short (< 4 h) and slightly fail-safe but not overly fail-safe predictions at longer (Fig. 1A). As indicated by Ross 1 it is important to confirm mean B f by using a graphical method to check for systematic prediction bias. problem, were obtained for models with acceptable mean B f and expected mean A f (Table 1). For example, a pB f of 0.5, a mean B f of 1.14 and a mean A f of 1.29 were obtained for interpolation of Model 5, which had two variables and an expected mean A f of < 1.3 and an acceptable B f of 0.7 to A second limitation of mean B f and mean A f is that they are biased for sets of data containing prediction cases where the model predicts growth but no growth is observed (i.e., observed =  and observed  max = 0) or where the model predicts no growth but growth is observed (i.e., predicted =  and predicted  max = 0) because B f and A f are ratios of observed and predicted values that cannot be calculated for these types of prediction cases. In contrast, such prediction cases by default fall outside the acceptable prediction zone and are included in the calculation of pB f. Thus, pB f is a more reliable indicator of model performance than mean B f and mean A f in situations involving no growth prediction cases (e.g. E. coli O157:H7 models in Table 2, which had 25 no growth prediction cases). A limitation of pB f is that it is unable to distinguish between models with global and regional (e.g., Model 1 for extrapolation in Fig. 1A) performance problems. However, use of pB f and a B f plot with an acceptable prediction zone was found to provide a reliable and complete evaluation of model performance. In particular, this combination was effective at identifying specific regions in the response surface where predictions were overly fail-safe or overly fail-dangerous. Together pB f and the B f plot form the acceptable prediction zone method for evaluating the performance of predictive models, a method that overcomes the limitations of B f and A f. REFERENCES 1 Ross, T J. Appl. Bacteriol. 81: Oscar, T. P J. Food Prot. 62: Oscar, T. P J. Food Prot. 62: Oscar, T. P J. Food Prot. 62: Ross, T. et al Int. J. Food Microbiol. 62: ACKNOWLEDGMENTS The author appreciates the excellent assistance of J. Ludwig and P. Shannon of ARS that made this research possible. In the present study, B f plots of individual prediction cases were used to confirm B f and in the process, B f plots were examined for overly fail-dangerous and overly fail-safe predictions using an acceptable prediction zone from a B f of 0.7 to a B f of The acceptable prediction zone was wider in the fail-safe direction because more tolerance can be allowed for predictions that error in this direction 5. In contrast to other methods for evaluating systematic prediction bias (e.g., normal distribution of residuals around zero and the runs test), a defined amount of systematic prediction bias is acceptable in the method developed here. In other words, as long as the systematic bias resides mostly within the acceptable prediction zone it is acceptable as was the case for extrapolation of broth Model 2 to cooked chicken breast and thigh burgers (Fig. 1B). A new performance factor (pB f ) that quantified the proportion of individual B f in the acceptable prediction zone was developed and used to evaluate model performance. Models that provided predictions with expected accuracy (i.e., mean A f < 1.3 for a two variable model and mean A f < 1.45 for a three variable model), acceptable bias (i.e., mean B f between 0.7 and 1.15) and B f plots without large systematic bias had pB f in the range of 0.7 to 1.0. Overall, pB f was a more sensitive and reliable indicator of model performance than mean B f and mean A f because low pB f (< 0.7), which indicated a performance