The Vision of Self-Aware Performance Models Johannes Grohmann, Simon Eismann, Samuel Kounev International Conference on Software Architecture Seattle, 02.05.2018

Motivation What’s the appropriate model granularity? Performance Model Model database as black-box, as its never a bottleneck. What’s the appropriate model granularity? Use mean value analysis, as I do not contain any forks. Performance Model What’s an appropriate solver for this model? My prediction accuracy is currently 97%. How accurate is this performance model? How to adapt this model if the system evolves? Recalibrate service demand of component X. Capacity planning & system design analysis

Overview

Query-based Model Tailoring Required model granularity changes depending on requested metric and performed adaptations Idea: dynamically adapt (tailor) model to fit scenario Provide multiple component descriptions Dynamically select appropriate modeling granularity Model unrelated system parts as black boxes Benefit: Improves simulation time Scales better with system size Maintains accuracy

Overview

Query-tailored Model Solution Brosig et al. [1] showed: Significant time-to-result and accuracy differences between different simulation- based solvers Time-to-result and accuracy depend on model properties Idea: Predict accuracy based on information loss Predict time-to-result based on historic information Select best suited solver based on these predictions [1] Brosig, Fabian, et al. "Quantitative evaluation of model-driven performance analysis and simulation of component-based architectures." IEEE Transactions on Software Engineering 41.2 (2015): 157-175.

Overview

Model Validation Self-reflective parameter analysis Compare model variable descriptions with monitoring data Model confidence is aggregation of variable confidences Detects inaccuracies proactively Cannot detect structural inaccuracies Historic prediction accuracy analysis Compare performance predictions with monitoring data Includes structural inaccuracies Hard to pinpoint source of inaccuracy Can only validate accuracy for previously deployed system states Combination allows for holistic model validation

Overview

Inaccurate parameterization Structural inaccuracies Model Recalibration Inaccurate parameterization Relearn parameter with additional monitoring data Choose different learning approach (meta-learning) Structural inaccuracies Re-extraction of model from monitoring data Add black-box causing e.g., additional response time or utilization

Conclusion Self-aware performance models as a solution to common modeling problems Four concrete examples for self-awareness in performance models: Model dynamically adapts to each request Query-based Model Tailoring Model solving process adapts to input model Query-tailored Model Solution Model learns about its prediction accuracy Model Validation Model repairs itself in case of system evolution Model Recalibration

Thank you for your attention! Slides are available at https://descartes.tools/ Johannes Grohmann, Simon Eismann, Samuel Kounev International Conference on Software Architecture Seattle, 02.05.2018