1. 1 Graduate Statistics Student, 2 Undergraduate Computer Science Student, 3 Professor and Director of Statistical Consulting Collaboratory 4 Chief Technology Officer, Integrien Corporation Introduction Conventional CUSUM Procedure Data Qi Zhang 1, Carlos J. Rendon 2, Daniel R. Jeske 3 Veronica Montes de Oca 1, and Mazda Marvasti 4 Alive TM is the major software product of Integrien Corporation that monitors, visually presents and reports the health of a business information technology system. The Statistical Consulting Collaboratory at the University of California, Riverside was contacted to develop a nonparametric statistical change-point detection procedure that would be applied to most types of univariate data. Our work extended the conventional CUSUM procedure to a nonparametric timeslot stationary context and is being implemented into the next release of Alive TM. CUSUM Screening of Historical Data CUSUM with Resetting Performance Evaluation For each simulated sample path, compute. H is the 100(1-  )th percentile of the EDF of these values, where  is the nominal false alarm level. Special Thanks To: The Staff of Integrien Corporation, Pengyue James Lin (CTO, College of Humanities, Arts and Social Sciences at UCR), Dr. Huaying Karen Xu (Associate Director of Statistical Consulting Collaboratory at UCR), Prof. Keh-Shin Lii (Dept of Statistics at UCR), Graduate Students of the Spring 2006 offering of STAT 293. Let X n denote the measurement of a univariate process at the n th time point and assume that with µ and σ 2 known. If X n shifts upward or downward more than K units from the mean, we say that there is a serious change. The CUSUM statistics are expressed as where K is generally called the reference value. If or are above some predetermined threshold H, we conclude that there is a change in the mean. The threshold H is determined to control the average run length (ARL) between false alarms, and is usually obtained from Monte Carlo Simulations. 1 2 3 4 5 6 target Level of change that is “serious.” For non-Gaussian measurements, use the 100  th and 100(1-  ) th percentile, and, for each timeslot instead of  + K and  – K. The generalized CUSUM becomes where  n = timeslot associated with the current hour  {1, 2, …168} Data from a real client was available. Data within each hour timeslot were assumed to be i.i.d. Empirical distributions for each timeslot are estimated from a rolling window of 12 weeks of historical data. Nonparametric CUSUM Procedure Implementation Flow Chart Reset CUSUM statistics after each alarm to eliminate the effect of previous alarm. Alarm end is determined via slope test. Real example from Integrien data H H Reset point Week1…121314…2021 Cycle 1 Predict Cycle 2 Predict … Cycle 7 Predict Cycle 8 Predict Cycle 9Predict Metric Number of Alarms Average detection Time (min) False Positives per cycle False Negatives per cycle Computation Time per Cycle (min) Live12 62.17 1.22017.11 Active1 8.00 0.00017.56 Resp. Time 10 352.00 1.1107.770 Oracle4 87.50 0.22012.38 Study Based on Simulated Data Monte Carlo Simulations for H 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 10 11 12 13 168 Timeslot Attribute Value Denotes median of distribution Assume the data windows causing alarms by the CUSUM procedure are anomalous. A slope test is used to find the start and end point of the data window. Start Point When the CUSUM statistic alerts, begin a backward sequence of fitted lines using windows of v points. Predicted start point is the rightmost point of the first window for which the hypothesis is not rejected on the basis of a t-test. End Point At the time the CUSUM statistic alerts, begin a forward sequence of fitted lines using windows containing the previous v points. Predicted end point is the time at which the CUSUM is the largest value within the first window for which the hypothesis is not rejected on the basis of a t-test. Inject an event that shifts the timeslot distributions by 100X% during the second half of the week. Report the average number of samples between the starting point of an injected event and the point at which the CUSUM signals. Average is based on 1,000 sample path simulations for each cycle. Study Based on Real Client Data Real-time Processing Off-line Processing Construct Timeslot Distributions Determine H for Screening Run CUSUM on Historical Data Screen Alerts from Historical Data Historical Data Construct Screened Timeslot Distributions Determine H for New Data CUSUM on New Data Monitor for Alerts Generalized CUSUM for Live Sessions A Nonparametric CUSUM Algorithm for Timeslot Sequences with Applications to Network Surveillance Variability in the Mean and Std. Dev. of the # of Live Sessions on a Network Server 12 weeks of historical data and 9 new monitoring weeks (cycles). True alarms were determined by subject matter expert. Conclusion: The procedure performs well with respect to 0 false negatives per cycle indicating alarms will be adequately detected. Conclusion: If the shift is small, the average number of samples until detection will be large. If the shift is large, the average number of samples until detection will small, therefore an alarm will be signaled immediately. - S + - S - - S + - S - H Signal an alarm here First backward-window before the alarm where the slope is not positive First forward-window after the alarm where the slope is no longer positive Predicted Start Time Predicted End Time t Illustrative Timeslot Distributions for # of Live Sessions on a Network Server