Introduction to IRRIIS testing platform IRRIIS MIT Conference ROME 8 February 2007 Claudio Balducelli

IRRIIS Summary Design a testing environment for MIT Modelling and running attack and fault behaviours Testing strategies for MIT components Proposed test-bed configuration Conclusions

IRRIIS Target Infrastructures Models Vulnerabilities of the Target Infrastructures Fault/attack Scenarios Generation Models of faults & attacks Use domain knowledge Consider vulnerabilities Design a testing environment for MIT

IRRIIS Meaning of attacks and faults Attacks: A disturbance of the LCCI generated by events coming from outside the LCCI Faults: A disturbance of the LCCI generated by events coming from the components that are part of the LCCI Definition of the meaning of attacks and faults

IRRIIS Meaning of attacks and faults Attacks: Natural disaster (earthquake, flood, etc) Premeditated terrorist attack Cyber attacks (cyber-intrusion) Operator errors ………….…. Faults: Physical component failure (aging, stress, etc.) Software component failure (bug, wrong istal. etc) Wrong component activation ………….….

IRRIIS Normal behavior & fault behavior in SimCIP Activation event t1 Start Comp. 1 Start Comp. 2 End Start Comp. 3 t2 Comp. 3 End Activation event Normal behavior consists in an initial state and a sequence of events represented in form of a petri net oriented graph

IRRIIS Initiating event t1 Failure of Comp. 1 t2 Failure of Comp. 2 t3 Restart Comp. 1 t4 t5 Loss of service 2 Normal behavior & fault behavior in SimCIP Loss of Service 1 Fault behavior may be represented in a similar way Fault events In LCCI-1 Failure of Comp. 2 t6 Failure of Comp. 1 t7 Fault events In LCCI-2

IRRIIS For a certain LCCI normal behaviors are well known and their number is limited the number and the combinations of fault behaviors are very high and not always known in advance how to design fault behaviors? how to select fault behaviors? utilisation of a model based on attack/fault trees seem useful to formalise and manage the knowledge needed to generate attack/fault behaviour Normal behavior & fault behavior in SimCIP

IRRIIS G0 A1 A2 The root of the tree (G) represents an event that could significantly harm the infrastructure’s mission. The terminal leafs (A) of the tree represent the actions to execute for reaching the high level goals Every path in the attack tree represents a unique type of attack Goal G0 AND A1 A2 A3 Goal G0 OR A1 A2 A3 The attack trees could be visualized also in textual form G0 A1 A2 A3 Every node could be decomposed inside lower level nodes using, and decomposition types AND OR Modelling attack knowledge attack/fault trees

IRRIIS G0 S1 A2 S2 A3 A4 A5 A6 The tree generate the following two attack patterns The “terminal leafs” of the tree (A1..An) represent the actions steps needed to execute the attack The “intermediate nodes” (S1..Sn) represent the steps in which a decision has to be taken The attack tree generates attack patterns (attack behaviors), composed by sequences of actions. Attack goal Modelling attack knowledge attack/fault trees

IRRIIS TE S1 C2 S3 C11 C12 C31 C32 The tree generate the following two fault patterns The “terminal leafs” of the tree (C..) represent the elementary failures of the single components of LCCI. The “intermediate nodes” (S…) represent failures of subsystems or services for which the components contribute The fault tree generates fault patterns (fault behaviors), composed by sequences of elementary failures. Top event Fault trees Modelling attack knowledge attack/fault trees

IRRIIS And gate Or gate OR gate AND gate Example of attack tree to model an attack in a local area network (tree structure) The reference model take in account the: Fault Tree Handbook of US Nuclear Regulatory Commission

IRRIIS And gate Or gate OR gate AND gate Example of attack tree to model an attack in a local area network (tree structure) Verify the accessibility to a subnet

IRRIIS And gate Or gate OR gate AND gate Example of attack tree to model an attack in a local area network (tree structure) Discover the target locations & addresses

IRRIIS And gate Or gate OR gate AND gate Example of attack tree to model an attack in a local area network (tree structure) Make sniffing activity or damages

IRRIIS And gate Or gate OR gate AND gate Example of attack tree to model an attack in a local area network (tree structure) Generated behaviours table Attack behaviour 0 Attack behaviour 1 Attack behaviour 2 Attack behaviour 3 Attack behaviour 4 Attack behaviour 5 Attack behaviour 6 Attack behaviour

IRRIIS Example of attack tree to model an attack: associating difficulties to the actions OR gate AND gate = maximum difficulty 1.0 = minimum difficulty Generated behaviours table ordered by action difficulties Attack behaviour 0 with 0,39 of difficulty Attack behaviour 2 with 0,24 of difficulty Attack behaviour 1 with 0.12 of difficulty Attack behaviour 3 with 0.08 of difficulty Attack behaviour 4 with 0.08 of difficulty Attack behaviour 6 with 0.05 of difficulty Attack behaviour 5 with 0.03 of difficulty Attack behaviour 7 with 0.02 of difficulty

IRRIIS Macro scenarios: how to compose attack and fault trees Attack tree Fault tree Attack tree Wait for malfunction

IRRIIS Composite attack and fault behavior t1 Basic Action 0 t2 Basic Action 2 Final Action 0 t3 t4 Final Action 1 Network malfunction Basic Event 0 Attack behavior Attack behavior Fault behavior Attack escalation

IRRIIS Testing MIT components (meaning) REQUIREMENTS: Risk Ass. (1) - The Risk estimator assessment of cascading and escalating effects shall be performed in near real-time. Risk Ass. (2) - The Risk estimator assessment of cascading and escalating effects shall be performed in a predictive way. Risk Ass. (3) - The Risk estimator shall estimate immediate risk to the LCCI. Risk Ass. (4) - The Risk estimator may estimate expected risk to the LCCI. Risk Ass. (5) - The Risk estimator shall estimate potential cascading effects. Objective of the TEST: validate the requirements Risk Ass. (1) - OK Risk Ass. (2) - OK Risk Ass. (3) - OK Risk Ass. (4) - NOT OK Risk Ass. (5) - NOT OK

IRRIIS Testing MIT components (meaning) One of the main objective of the MIT components test inside SimCIP simulated environment is the evaluation of the rate of false/true alarms. The second is to evaluate how much the rate of false alarms may be acceptable for the LCCIs operators

IRRIIS Detecting interdependency alarms Real states Predicted states AlarmNo Alarm P(Alarm) AB P(No Alarm) CD A = Number of alarm states correctly predicted D = Number of no alarm states correctly predicted B = Number of no alarm states predicted as true (FALSE POSITIVE) C = Number of alarm states not predicted (FALSE NEGATIVE) The goal is: max(A + D), min(B + C) Evaluation Table

IRRIIS Detecting interdependency alarms Real states Predicted states AlarmNo Alarm P(Alarm) AB P(No Alarm) CD Fn = C / ( C + D ) Observed False Negative Ratio (FNR) Fp = B / ( A + B ) Observed False Positive Ratio (FPR)

IRRIIS Be not afraid to discover false alarms during the tests. This is the tests objective!! In many cases false alarms could be simply reduced tuning the “sensitivity” level of a MIT component. To evaluate true/false alarms ratio is not sufficient a single attack/fault behavior. Many alternative behaviors are needed!! Logging facilities are very important during experimentations, are the tests results must be archived and documented Detecting interdependency alarms

IRRIIS Proposed testing strategy IRRIIS testing operator Attack/Fault tree editor Design or modify a scenario tree GA S1 A2 S2 A3 A4 A5 A6 Fault behaviors editor Generate & modify fault behaviors, insert timing information etc Documentation console View logs Edit test documents Logs Test documents Fault behavior execution Execute behavious, sets monitors Attacks/faults execution in SimCIP Test design entry point Test design exit point Test design

IRRIIS Proposed testing strategy IRRIIS testing operator Attack/Fault tree editor Design or modify a scenario tree GA S1 A2 S2 A3 A4 A5 A6 Fault behaviors editor Generate & modify fault behaviors, insert timing information etc Documentation console View logs Edit test documents Logs Test documents Fault behavior execution Execute behavious, sets monitors Attacks/faults execution in SimCIP Test execution entry point Test execution exit point Fast testing

IRRIIS Proposed testing strategy IRRIIS testing operator Attack/Fault tree editor Design or modify a scenario tree GA S1 A2 S2 A3 A4 A5 A6 Fault behaviors editor Generate & modify fault behaviors, insert timing information etc Documentation console View logs Edit test documents Logs Test documents Fault behavior execution Execute behavious, sets monitors Attacks/faults execution in SimCIP Test entry point Test exit point Exhaustive testing

IRRIIS Physical TESTBED Configurations LAMPSSys RTI GUI Logger Tool 1 Electricity Simulator LCCI Data Com Simulator Tool 2 Agent / Scenario Behaviours Analysis 1Analysis 2 Fault / Attack Tool MIT Analysis 3 SimCIP Architecture

IRRIIS Physical TESTBED Configurations GUI Logger LAMPSSys RTI Agent / Scenario Behaviours Electricity Simulator Com Simulator LCCI Electricity Data Base Tool 1 Tool 2 Analysis 1, 2, 3.. LCCI Telecom Data Base Simple SimCIP configuration

IRRIIS Physical TESTBED Configuration LAMPSSys RTI Agent / Scenario Behaviours Electricity Simulator Com Simulator LCCI Electricity Data Base Fault /Attack Tool Tool 1 Tool 2 Analysis 1, 2, 3.. LCCI Telecom Data Base SimCIP for testing attacks and faults without MIT GUI Logger

IRRIIS Physical TESTBED Configuration GUI Logger LAMPSSys RTI Agent / Scenario Behaviours Electricity Simulator Com Simulator LCCI Electricity Data Base LCCI Telecom Data Base MT communication Electricity Add-onTelecom Add-on SimCIP for testing MIT with normal behaviors (detect false positive alarms)

IRRIIS Physical TESTBED Configuration GUI Logger LAMPSSys RTI Agent / Scenario Behaviours Electricity Simulator Com Simulator LCCI Electricity Data Base LCCI Telecom Data Base MT communication Electricity Add-onTelecom Add-on SimCIP for testing MIT in presence of attacks/faults (detect false negative alarms) Fault /Attack Tool Tool 1 Tool 2 Analysis 1, 2, 3..

IRRIIS Conclusions Testing of MIT components will be a continuous and iterative process It is necessary to distinguish between the fast tests of the more simple requirements and the exhaustive test process aimed to evaluate the MIT efficiency in detecting interdependency alarms Test designing, reports logging/archiving in a standard way and with the support of a common tool, will help to have sets of comparable tests also if produced in different SimCIP installations. The testing environment will be one of the major a research product of the project, where experimentation may continue also after the end of the project. QUESTIONS?