Probabilistic Calling Context Michael D. Bond Kathryn S. McKinley University of Texas at Austin

Why Context Sensitivity? at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 Static program location not enough

at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 at com.mckoi.db.jdbc.MConnection.executeQuery():348 at com.mckoi.db.jdbc.MStatement.executeQuery():110 at com.mckoi.db.jdbc.MStatement.executeQuery():127 at Test.main():48 Static program location not enough Why Context Sensitivity?

at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 at com.mckoi.db.jdbc.MConnection.executeQuery():348 at com.mckoi.db.jdbc.MStatement.executeQuery():110 at com.mckoi.db.jdbc.MStatement.executeQuery():127 at Test.main():48 Motivated by  Complex programs  Small methods  Virtual dispatch Static program location not enough Why Context Sensitivity?

at com.mckoi.db.jdbcserver.JDBCInterface.execQuery():213 at com.mckoi.db.jdbc.MConnection.executeQuery():348 at com.mckoi.db.jdbc.MStatement.executeQuery():110 at com.mckoi.db.jdbc.MStatement.executeQuery():127 at Test.main():48 Motivated by  Complex programs  Small methods  Virtual dispatch Static program location not enough C/Fortran method Java/C# method return call return Why Context Sensitivity?

Context Is Nontrivial API calls ProgramCall sitesDistinct contexts antlr4,184128,627 bloat3,306600,947 chart2,335202,603 eclipse9,611226,020 fop2,22537,710 hsqldb94716,050 jython1,830628,048 luindex654102,556 lusearch pmd1,890847,108 xalan1,53017,905

Example: Residual Testing class EditorWindow { close() {... } Does behavior occur at production time that did not occur at testing time? class SimpleWindow { close() {... }

Example: Residual Testing class EditorWindow { close() {... } Does behavior occur at production time that did not occur at testing time? class SimpleWindow { close() {... } inputHandler() {... case CLICK_EXIT: w.checkUnsaved(); w.close();... } autoUpdate() {... for all windows w w.close();... }

class EditorWindow { close() {... } class SimpleWindow { close() {... } Example: Residual Testing Does behavior occur at production time that did not occur at testing time? inputHandler() {... case CLICK_EXIT: w.checkUnsaved(); w.close();... } autoUpdate() {... for all windows w w.close();... } Bug!

class EditorWindow { close() {... } class SimpleWindow { close() {... } Example: Residual Testing Does behavior occur at production time that did not occur at testing time? inputHandler() {... case CLICK_EXIT: w.checkUnsaved(); w.close();... } autoUpdate() {... for all windows w w.close();... } Bug! New behavior indicates bugs Context sensitivity helps find new behavior

Two-Phase Dynamic Analyses TrainingProduction Behavior observed New or anomalous behavior detected

Two-Phase Dynamic Analyses Residual testing Anomaly-based intrusion detection Anomaly-based bug detection TrainingProduction Behavior observed New or anomalous behavior detected What behavior occurs at production time that did not occur at testing time? [Vaswani et al. ’07] What new behavior occurs during a buggy program run? [Hangal & Lam ’02] Does a program exhibit anomalous behavior? [Inoue ’05]

Probabilistic Calling Context TrainingProduction Behavior observed New or anomalous behavior detected Adds context sensitivity to dynamic analyses Maintains value representing context  Unique with high probability  New value  new context  walk stack High accuracy: <0.1% false negatives Low overhead: 3% overhead, 0-8% for clients

Outline Introduction Previous approaches Maintaining the PCC value Evaluation  Accuracy  Performance

Previous Approaches Tracking context [Ammons et al. ’97] [Spivey ‘04]  Maintain CCT position at each call/return Walking the stack [Nethercote & Seward ‘07] Path profiling [Ball & Larus ’96] [Melski & Reps ’99]  Call graphs large  path explosion  Virtual dispatch complicates instrumentation

Previous Approaches Tracking context [Ammons et al. ’97] [Spivey ‘04]  Maintain CCT position at each call/return Walking the stack [Nethercote & Seward ‘07] Path profiling [Ball & Larus ’96] [Melski & Reps ’99]  Call graphs large  path explosion  Virtual dispatch complicates instrumentation Sampling [Zhuang et al. ’06]  Sacrifices coverage for low overhead

Outline Introduction Previous approaches Maintaining the PCC value Evaluation  Accuracy  Performance

PCC Function f ( V, cs )  V is PCC value  cs is call site ID

PCC Function f ( V, cs ) V ← f ( V, cs 1 ) V ← V saved V ← f ( V, cs 2 ) cs 2 cs 1  V is PCC value  cs is call site ID

PCC Function f ( V, cs ) ≡ 3V + cs (mod 2 32 )  V is PCC value  cs is call site ID

PCC Function f ( V, cs ) ≡ 3V + cs (mod 2 32 ) Motivated by MPI datatype hashing [Langou et al. ’05] [Gropp ’00] Cheap to compute Desirable properties:  Non-commutative  Composition efficient to compute

Differentiating Similar Contexts C A B V ← 3V + cs 1 V ← 3V + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2 …  A()  B()  … …  B()  A()  …

Differentiating Similar Contexts Non-commutative f ( f (V, cs 1 ), cs 2 ) ≠ f ( f (V, cs 2 ), cs 1 ) C A B V ← 3V + cs 1 V ← 3V + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

Efficiency at Inlined Calls C A B V ← 3V + cs 1 V ← 3V + cs 2

Efficiency at Inlined Calls C A B V ← 3 ( 3V + cs 1 ) + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

Efficiency at Inlined Calls C A B V ← 9V + 3cs 1 + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

Efficiency at Inlined Calls Composition efficient to compute C A B V ← 9V + 3cs 1 + cs 2 C A B V ← 3V + cs 1 V ← 3V + cs 2

Outline Introduction Previous approaches Maintaining the PCC value Evaluation  Methodology  Evaluating potential clients  Accuracy  Performance

Methodology Implementation in Jikes RVM  Available on Jikes RVM Research Archive Deterministic calling context profiling  Maintains CCT node at each call & return Benchmarks: DaCapo, SPEC JBB2000, SPEC JVM98 Platform: 3.6 GHz Pentium 4 w/Linux

TrainingProduction Behavior observed New or anomalous behavior detected Record values New value  new context  walk stack How Clients Use PCC

TrainingProduction Behavior observed New or anomalous behavior detected Evaluating Potential Clients Record values Check values (no new values) Global hash table

TrainingProduction Behavior observed New or anomalous behavior detected Evaluating Potential Clients Record values Check values (no new values) Global hash table Memory overhead: proportional to contexts

Evaluating Potential Clients Residual testing Check PCC value at Java API calls (calls to java.* ) Check PCC value at system calls (Network, I/O, OS) Anomaly-based intrusion detection Upper bound Check PCC value at all calls

Ideal Accuracy PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context

Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000, (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context

Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000, (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context API calls

Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000, (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context All calls

Ideal Accuracy Expected conflicts (false negatives) Distinct contexts32-bit values64-bit values 100,0001 (0.0%)0 (0.0%) 1,000, (0.0%)0 (0.0%) 10,000,00011,632 (0.1%)0 (0.0%) 100,000,0001,155,170 (1.2%)0 (0.0%) 1,000,000,000107,882,641 (10.8%)0 (0.0%) 10,000,000,0006,123,623,065 (61.2%)3 (0.0%) PCC maps context to value  New PCC value  new context  Familiar PCC value  probably familiar context Near-perfect accuracy

PCC’s Accuracy System callsJava API calls ProgramDynamicDistinctConf.DynamicDistinctConf. antlr211,4901,567024,422,013128,6273 bloat121001,159,281,573600,94740 chart ,891,525202,6034 eclipse14, ,507,343226,0205 fop181709,918,27537,7100 hsqldb12 081,161,54116,0500 jython5,9294, ,845,772628,04848 luindex2, ,733,214102,5560 lusearch ,511, pmd1, ,017,118847,10879 xalan137, ,105,838,67017,9050

PCC’s Accuracy System callsJava API calls ProgramDynamicDistinctConf.DynamicDistinctConf. antlr211,4901,567024,422,013128,6273 bloat121001,159,281,573600,94740 chart ,891,525202,6034 eclipse14, ,507,343226,0205 fop181709,918,27537,7100 hsqldb12 081,161,54116,0500 jython5,9294, ,845,772628,04848 luindex2, ,733,214102,5560 lusearch ,511, pmd1, ,017,118847,10879 xalan137, ,105,838,67017,9050

PCC’s Accuracy All calls ProgramDynamicDistinctConf. antlr490,363,2111,006, bloat6,276,446,0591,980, chart908,459,469845,43291 eclipse1,266,810,5044,815,9012,652 fop44,200,446174,9552 hsqldb877,680,667110,7951 jython5,326,949,1583,859,5451,738 luindex740,053,104374,20112 lusearch1,439,034,3366,0390 pmd2,726,876,9578,043,0967,653 xalan10,083,858,546163,2056

PCC’s Execution Time Overhead 3%

PCC’s Execution Time Overhead 3%

Summary PCC maintains calling context value  New value indicates new behavior Low overhead  Maintaining PCC value adds 3%  Checking PCC value 0-8%  Memory overhead proportional to contexts High accuracy  Less than 0.1% false negative rate PCC adds context sensitivity to clients that detect anomalous behavior

Summary PCC maintains calling context value  New value indicates new behavior Low overhead  Maintaining PCC value adds 3%  Checking PCC value 0-8%  Memory overhead proportional to contexts High accuracy  Less than 0.1% false negative rate PCC adds context sensitivity to clients that detect anomalous behavior Thank you!

Extra slides

Do paths capture enough behavior? C/Fortran method Java/C# method return call return Context Sensitivity Mostly Unused