Bug Isolation via Remote Program Sampling Ben LiblitAlex Aiken Alice ZhengMike Jordan.

Slides:



Advertisements
Similar presentations
Welcome to Econ 420 Applied Regression Analysis
Advertisements

Linear Regression.
Building a Better Backtrace: Techniques for Postmortem Program Analysis Ben Liblit & Alex Aiken.
Building a Better Backtrace: Techniques for Postmortem Program Analysis Ben Liblit & Alex Aiken.
1 Bug Isolation via Remote Program Sampling Ben LiblitAlex Aiken Alice X. ZhengMichael Jordan Presented By : Arpita Gandhi.
Statistical Debugging Ben Liblit, University of Wisconsin–Madison.
Statistical Debugging Ben Liblit, University of Wisconsin–Madison.
Bug Isolation via Remote Program Sampling Ben Liblit, Alex Aiken, Alice X.Zheng, Michael I.Jordan Presented by: Xia Cheng.
Models with Discrete Dependent Variables
(Quickly) Testing the Tester via Path Coverage Alex Groce Oregon State University (formerly NASA/JPL Laboratory for Reliable Software)
Bug Isolation in the Presence of Multiple Errors Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken, and Michael I. Jordan UC Berkeley and Stanford University.
The Future of Correct Software George Necula. 2 Software Correctness is Important ► Where there is software, there are bugs ► It is estimated that software.
1 BA 275 Quantitative Business Methods Residual Analysis Multiple Linear Regression Adjusted R-squared Prediction Dummy Variables Agenda.
Lecture 19 Simple linear regression (Review, 18.5, 18.8)
Chapters 10 and 11: Using Regression to Predict Math 1680.
By Jayelle Hegewald, Michele Houtappels and Melinda Gray 2013.
Pop Quiz How does fix response time and fix quality impact Customer Satisfaction? What is a Risk Exposure calculation? What’s a Scatter Diagram and why.
Stochastic Population Modelling QSCI/ Fish 454. Stochastic vs. deterministic So far, all models we’ve explored have been “deterministic” – Their behavior.
Gaussian process modelling
PATTERN RECOGNITION AND MACHINE LEARNING
Scalable Statistical Bug Isolation Ben Liblit, Mayur Naik, Alice Zheng, Alex Aiken, and Michael Jordan, 2005 University of Wisconsin, Stanford University,
Scalable Statistical Bug Isolation Ben Liblit, Mayur Naik, Alice Zheng, Alex Aiken, and Michael Jordan University of Wisconsin, Stanford University, and.
Testing CSE 140 University of Washington 1. Testing Programming to analyze data is powerful It’s useless if the results are not correct Correctness is.
Bug Localization with Machine Learning Techniques Wujie Zheng
School of Electrical Engineering and Computer Science University of Central Florida Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated.
Logistic Regression Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata September 1, 2014.
Scalable Statistical Bug Isolation Authors: B. Liblit, M. Naik, A.X. Zheng, A. Aiken, M. I. Jordan Presented by S. Li.
Testing and Debugging Version 1.0. All kinds of things can go wrong when you are developing a program. The compiler discovers syntax errors in your code.
BY Lecturer: Aisha Dawood. The hiring problem:  You are using an employment agency to hire a new office assistant.  The agency sends you one candidate.
Correlation Analysis. A measure of association between two or more numerical variables. For examples height & weight relationship price and demand relationship.
Various topics Petter Mostad Overview Epidemiology Study types / data types Econometrics Time series data More about sampling –Estimation.
COMP 170 L2 L17: Random Variables and Expectation Page 1.
Testing and Debugging Session 9 LBSC 790 / INFM 718B Building the Human-Computer Interface.
Linear Trend Lines = b 0 + b 1 X t Where is the dependent variable being forecasted X t is the independent variable being used to explain Y. In Linear.
Copyright © Cengage Learning. All rights reserved. 13 Linear Correlation and Regression Analysis.
1 Summarizing Performance Data Confidence Intervals Important Easy to Difficult Warning: some mathematical content.
Design - programming Cmpe 450 Fall Dynamic Analysis Software quality Design carefully from the start Simple and clean Fewer errors Finding errors.
Bug Isolation via Remote Sampling. Lemonade from Lemons Bugs manifest themselves every where in deployed systems. Each manifestation gives us the chance.
28. Multiple regression The Practice of Statistics in the Life Sciences Second Edition.
Classification Ensemble Methods 1
Testing CSE 160 University of Washington 1. Testing Programming to analyze data is powerful It’s useless (or worse!) if the results are not correct Correctness.
Unit 2: Research & Statistics n Psychology deals with many experiments and studies n WHO? Every experimenter must decide on a SAMPLE, which is a group.
Week 12 - Wednesday.  What did we talk about last time?  Hunters and prey.
Logistic Regression & Elastic Net
Week 12 - Monday.  What did we talk about last time?  Defining classes  Class practice  Lab 11.
Stats Term Test 4 Solutions. c) d) An alternative solution is to use the probability mass function and.
CSC317 1 Randomized algorithms Hiring problem We always want the best hire for a job! Using employment agency to send one candidate at a time Each day,
Statistical Debugging CS Motivation Bugs will escape in-house testing and analysis tools –Dynamic analysis (i.e. testing) is unsound –Static analysis.
26134 Business Statistics Week 4 Tutorial Simple Linear Regression Key concepts in this tutorial are listed below 1. Detecting.
LECTURE 07: CLASSIFICATION PT. 3 February 15, 2016 SDS 293 Machine Learning.
Communications Range Analysis Simulation Set Up –Single Biological Threat placed in Soldier Field –Communication range varied from meters –Sensor.
Cooperative Bug Isolation CS Outline Something different today... Look at monitoring deployed code –Collecting information from actual user runs.
Giansalvo EXIN Cirrincione unit #4 Single-layer networks They directly compute linear discriminant functions using the TS without need of determining.
Automated Adaptive Bug Isolation using Dyninst Piramanayagam Arumuga Nainar, Prof. Ben Liblit University of Wisconsin-Madison.
Week 7: General linear models Overview Questions from last week What are general linear models? Discussion of the 3 articles.
1 LING 696B: Maximum-Entropy and Random Fields. 2 Review: two worlds Statistical model and OT seem to ask different questions about learning UG: what.
Stats 242.3(02) Statistical Theory and Methodology.
Bug Isolation via Remote Program Sampling Ben LiblitAlex Aiken Alice X. ZhengMichael I. Jordan UC Berkeley.
Regression and Correlation
YAHMD - Yet Another Heap Memory Debugger
10701 / Machine Learning.
Machine Learning Basics
Statistical Debugging
Testing UW CSE 160 Spring 2018.
Sampling User Executions for Bug Isolation
Public Deployment of Cooperative Bug Isolation
Methods of Economic Investigation Lecture 12
Finding and Managing Bugs CSE 403 Lecture 23
Statistical Debugging
Chap. 7 Regularization for Deep Learning (7.8~7.12 )
Presentation transcript:

Bug Isolation via Remote Program Sampling Ben LiblitAlex Aiken Alice ZhengMike Jordan

Motivation: Users Matter Imperfect world with imperfect software –Ship with known bugs –Users find new bugs –Bug fixing is a matter of triage Important bugs happen often, to many users Can users help us find and fix bugs? –Learn a little bit from each of many runs

Users as Debuggers Must not disturb individual users  Sparse sampling: spread costs wide and thin Aggregated data may be huge  Client-side reduction/summarization Will never have complete information  Make wild guesses about bad behavior  Look for broad trends across many runs

Sampling the Bernoulli Way Identify the points of interest Decide to examine or ignore each site… –Randomly –Independently –Dynamically Global countdown to next sample –Geometric distribution with some mean –Simulates many tosses of a biased coin

Countdown Predicts the Future “Fast path” when no sample is imminent –Common case –(Nearly) instrumentation free “Slow path” only when taking a sample Choose at top of each acyclic region –Is countdown < max path weight of region ? –Like Arnold & Ryder, but statistically fair

Sharing the Cost of Assertions What to sample: assert() statements Look for assertions which sometimes fail on bad runs, but always succeed on good runs Overhead in assertion-dense CCured code –Unconditional: 55% average, 181% max – 1 / 100 sampling: 17% average, 46% max – 1 / 1000 sampling: 10% average, 26% max

Isolating a Deterministic Bug What to sample: –Function return values –Client-side reduction Triple of counters per call site: 0 Look for values seen on some bad runs, but never on any good run Hunt for crashing bug in ccrypt-1.2

Winnowing Down the Culprits 1710 counters –3 × 570 call sites 1569 are zero on all runs –141 remain 139 are nonzero on some successful run Not much left! file_exists() > 0 xreadline() == 0

Isolating a Non-Deterministic Bug At each direct scalar assignment x = … For each same-typed in-scope variable y Guess some predicates on x and y x y Count how often each predicate holds –Client-side reduction into counter triples

Statistical Debugging Regularized logistic regression –S-shaped cousin to linear regression –Predict crash/non-crash as function of counters –Penalty factor forces most coefficients to zero –Large coefficient  highly predictive of crash Hunt for intermittent crash in bc-1.06 –30,150 candidates in 8910 lines of code –2729 training runs with random input

Top-Ranked Predictors void more_arrays () { … /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++) arrays[indx] = NULL; … } #1: indx > scale #2: indx > use_math #1: indx > scale #2: indx > use_math #1: indx > scale #2: indx > use_math #3: indx > opterr #4: indx > next_func #5: indx > i_base #1: indx > scale #2: indx > use_math #3: indx > opterr #4: indx > next_func #5: indx > i_base

Bug Found: Buffer Overrun void more_arrays () { … /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++) arrays[indx] = NULL; … }

Conclusions Implicit bug triage –Learn the most, most quickly, about the bugs that happen most often Variability is a benefit rather than a problem There is strength in numbers many users +statistical modeling =find bugs while you sleep!

Linear Regression Match a line to the data points Outcome can be anywhere along y axis But our outcomes are always 0/1

Logistic Regression Prediction asymptotically approaches 0 and 1 –0: predict no crash –1: predict crash

Training the Model Maximize LL using stochastic gradient ascent Problem: model is wildly under-constrained –Far more counters than runs –Will get perfectly predictive model just using noise

Regularized Logistic Regression Add penalty factor for nonzero terms Force most coefficients to zero Retain only features which “pay their way” by significantly improving prediction accuracy

Deployment Scenarios Incidence rate of bad behavior: 1 / 100 Sampling density: 1 / 1000 Confidence of seeing one example: 90% Required runs: 230,258 Microsoft Office XP –First-year licensees: 60,000,000 –Assumed usage rate: twice per week –Time required: nineteen minutes

Deployment Scenarios Incidence rate of bad behavior: 1 / 1000 Sampling density: 1 / 1000 Confidence of seeing one example: 99% Required runs: 4,605,168 Microsoft Office XP –First-year licensees: 60,000,000 –Assumed usage rate: twice per week –Time required: less than seven hours