Static and Dynamic Analysis at JPL Klaus Havelund

2 Mars Science Laboratory (MSL) planned launch 2011 biggest rover so far to be sent to Mars programmer team of 30 testing team of 10+ people programming language is C, 3 M LOC highly multi-threaded (over 160 threads)

3 program inputoutput specification and programming Let’s see … a command should always succeed …

4 program specification inputoutput specification and programming formal

5 specification and programming program specification relationship inputoutput formal

6 specification and programming program specification refinement inputoutput code generation: -from state machines -graphical -textual -from data formats (XML) formal

7 specification and programming program specification static analysis inputoutput formal abstraction [](request -> <>response) model checking

8 specification and programming program specification static analysis inputoutput Good practice analysis using commercial static analyzers. formal

9 a checkable Java coding standard

10 why a standard? two perspectives more reliable code code that is easier to read, leading to –code that is easier to develop –code that is easier to maintain –code that is easier to share style

11 the choices industry standard organizational standard project standard personal standard no standard good bad

12 basic questions 1. do we need to agree on a standard? carefully designed? lots of rules, which can be turned on/off? or 2. should it be 100% checkable? 3. do we care about naming, style, doc? 4. how many rules?

13 C coding standard

14

15

16

17 free tools coding standard checkers coding error checker

18

19

20

21 program specification dynamic analysis inputoutput specification and programming formal

model-based testing

23 model-based testing do :: mkdir dir :: cd dir :: rm file ::.. … od SPIN abstract state abstract real file system reference file system

24 program specification runtime verification inputoutput specification and programming formal

tool-based log file analysis

26 problems with testing FSW flight software engineers work under tight schedules: hard to access. system = hardware + software: it is cumbersome to run. difficult to determine what events to monitor..h.c AB

27 separation of concerns log

28 architecture log LogMaker [e 1,e 2,…,e n ] LogScope spec violations

29 command execution commanddispatchsuccess dispatch failure

30... COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7309 { message := "Dispatched immediate command POWER_HOUSEKEEPING: number=4, seconds= , subseconds= " Dispatch := "POWER_HOUSEKEEPING" Time := name := "CMD_DISPATCH" level := "COMMAND" Number := "4” }... EVR 7311 { name := "POWER_SEND_REQUEST" Time := message := "power_queue_card_request- sending request to PAM 0." level := "DIAGNOSTIC” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” } EVR 7313 { name := "PWR_REQUEST_CALLBACK" Time := message := "power_card_request - FPGA request successfully sent to RPAM A." level := "DIAGNOSTIC” } CHANNEL 7314 { channelId := "PWR-3049" DNChange := 67 dnUnsignedValue := 1600 type := "UNSIGNED_INT" Time := ChannelName := "PWR-BCB1-AMP” }... COMMAND 9626 { Args := ['set_device(1)', 'TRUE'] Time := Stem := "RUN_COMMAND" Number := "18" type := "FSW” } EVR 9627 { message := "Validation failed for command RUN_COMMAND: number=18." DispatchFailure := "RUN_COMMAND" Time := name := "CMD_DISPATCH_VALIDATION_FAILURE" level := "COMMAND" Number := "18” }... example log ?

31... COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7309 { message := "Dispatched immediate command POWER_HOUSEKEEPING: number=4, seconds= , subseconds= " Dispatch := "POWER_HOUSEKEEPING" Time := name := "CMD_DISPATCH" level := "COMMAND" Number := "4” }... EVR 7311 { name := "POWER_SEND_REQUEST" Time := message := "power_queue_card_request- sending request to PAM 0." level := "DIAGNOSTIC” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” } EVR 7313 { name := "PWR_REQUEST_CALLBACK" Time := message := "power_card_request - FPGA request successfully sent to RPAM A." level := "DIAGNOSTIC” } CHANNEL 7314 { channelId := "PWR-3049" DNChange := 67 dnUnsignedValue := 1600 type := "UNSIGNED_INT" Time := ChannelName := "PWR-BCB1-AMP” }... COMMAND 9626 { Args := ['set_device(1)', 'TRUE'] Time := Stem := "RUN_COMMAND" Number := "18" type := "FSW” } EVR 9627 { message := "Validation failed for command RUN_COMMAND: number=18." DispatchFailure := "RUN_COMMAND" Time := name := "CMD_DISPATCH_VALIDATION_FAILURE" level := "COMMAND" Number := "18” }... example log ?

32... COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7309 { message := "Dispatched immediate command POWER_HOUSEKEEPING: number=4, seconds= , subseconds= " Dispatch := "POWER_HOUSEKEEPING" Time := name := "CMD_DISPATCH" level := "COMMAND" Number := "4” }... EVR 7311 { name := "POWER_SEND_REQUEST" Time := message := "power_queue_card_request- sending request to PAM 0." level := "DIAGNOSTIC” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” } EVR 7313 { name := "PWR_REQUEST_CALLBACK" Time := message := "power_card_request - FPGA request successfully sent to RPAM A." level := "DIAGNOSTIC” } CHANNEL 7314 { channelId := "PWR-3049" DNChange := 67 dnUnsignedValue := 1600 type := "UNSIGNED_INT" Time := ChannelName := "PWR-BCB1-AMP” }... COMMAND 9626 { Args := ['set_device(1)', 'TRUE'] Time := Stem := "RUN_COMMAND" Number := "18" type := "FSW” } EVR 9627 { message := "Validation failed for command RUN_COMMAND: number=18." DispatchFailure := "RUN_COMMAND" Time := name := "CMD_DISPATCH_VALIDATION_FAILURE" level := "COMMAND" Number := "18” }... example log ?

33 specification languages for trace analysis programming languages (Python at JPL) state machines regular expressions temporal logic grammars most commonly used “formal” trace logics

34 the first scripture look:DRILL_DMP\ evr(CMD_DISPATCH,positive)\ evr(CMD_COMPLETED_SUCCCESS,positive)\ evr(CMD_COMPLETED_FAILURE,negative)\ chan(id:CMD-0004,positive,contains opcode of last immediate command)\ chan(id:CMD-0007,positive)\ chan(id:CMD-0001,negative)\ chan(id:CMD-0009,negative)\ prod(name:DrillAll,1,*) trigger consequences

35 property P 1 P 1 : Whenever a flight software command is issued, then eventually an EVR should indicate success of that command P 1 : Whenever a flight software command is issued, then eventually an EVR should indicate success of that command

36 recall log COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” }......

37 COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” } recall log P 1 : Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, then eventually an EVR should occur, with the field Success mapped to x and the Number field mapped to y. P 1 : Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, then eventually an EVR should occur, with the field Success mapped to x and the Number field mapped to y.

38 COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” } recall log pattern P1: COMMAND{Type:"FSW", Stem:x, Number:y} => EVR{Success:x, Number:y}

39 pattern P1: COMMAND{Type:"FSW", Stem:x, Number:y} => EVR{Success:x, Number:y}

40 pattern syntax pattern ::= 'pattern' NAME ':' event '=>' consequence consequence ::= event | '!' event | '[' consequence 1,...,consequence n ']’ | ‘{' consequence 1,...,consequence n ‘}' pattern ::= 'pattern' NAME ':' event '=>' consequence consequence ::= event | '!' event | '[' consequence 1,...,consequence n ']’ | ‘{' consequence 1,...,consequence n ‘}'

41 pattern P2: COMMAND{Type:"FSW", Stem:x, Number:y} => ! EVR{Failure:x, Number:y} P 2 : Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, Then an EVR should thereafter not occur, with the field Failure mapped to x and the Number field mapped to y. P 2 : Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, Then an EVR should thereafter not occur, with the field Failure mapped to x and the Number field mapped to y. property P 2

42 pattern syntax pattern ::= 'pattern' NAME ':' event '=>' consequence consequence ::= event | '!' event | '[' consequence 1,...,consequence n ']’ | ‘{' consequence 1,...,consequence n ‘}' pattern ::= 'pattern' NAME ':' event '=>' consequence consequence ::= event | '!' event | '[' consequence 1,...,consequence n ']’ | ‘{' consequence 1,...,consequence n ‘}'

43 property P 3 P 3 : Whenever a flight software command is issued, there should follow a dispatch and then exactly one success. No dispatch failure before the dispatch, and no failure between dispatch and success. P 3 : Whenever a flight software command is issued, there should follow a dispatch and then exactly one success. No dispatch failure before the dispatch, and no failure between dispatch and success.

44 formalization pattern P3: COMMAND{Type:"FSW", Stem:x, Number:y} => [ ! EVR{DispatchFailure:x, Number:y}, EVR{Dispatch:x, Number:y}, ! EVR{Failure:x, Number:y}, EVR{Success:x, Number:y}, ! EVR{Success:x, Number:y} ]

45 expressed in first order LTL

46 Python predicate definitions {: def within(t1,t2,max): return (t2-t1) <= max :} pattern P6: COMMAND{Type:"FSW",Stem:x,Number:y,Time:t1} where {: x.startswith("PWR_”) :} => EVR{Success:x, Number:y, Time:t2} where within(t1,t2,10000)

47 scoped version of P4 pattern P9: COMMAND{Type:"FSW", Stem:x, Number:y} => { EVR{Dispatch:x, Number:y}, [ EVR{Success:x, Number:y}, ! EVR{Success:x, Number:y} ], ! EVR{DispatchFailure:x, Number:y}, ! EVR{Failure:x, Number:y} } upto COMMAND{Type: "FSW"}

48 from patterns to automata temporal patterns are translated into parameterized universal automata automaton language more expressive user can use both, in practice only temporal patterns have been used for testing MSL

49 recall P3 pattern P3: COMMAND{Type:"FSW", Stem:x, Number:y} => [ ! EVR{DispatchFailure:x, Number:y}, EVR{Dispatch:x, Number:y}, ! EVR{Failure:x, Number:y}, EVR{Success:x, Number:y}, ! EVR{Success:x, Number:y} ]

50 automaton A_P3 { always S1 { COMMAND{Type:"FSW",Stem:x,Number:y} => S2(x,y) } hot state S2(x,y) { EVR{DispatchFailure:x, Number:y} => error EVR{Dispatch:x, Number:y} => S3(x,y) } hot state S3(x,y) { EVR{Failure:x, Number:y} => error EVR{Success:x, Number:y} => S4(x,y) } state S4(x,y) { EVR{Success:x, Number:y} => error }

51

52... COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW” } EVR 7309 { message := "Dispatched immediate command POWER_HOUSEKEEPING: number=4, seconds= , subseconds= " Dispatch := "POWER_HOUSEKEEPING" Time := name := "CMD_DISPATCH" level := "COMMAND" Number := "4” }... EVR 7311 { name := "POWER_SEND_REQUEST" Time := message := "power_queue_card_request- sending request to PAM 0." level := "DIAGNOSTIC” } EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4” } EVR 7313 { name := "PWR_REQUEST_CALLBACK" Time := message := "power_card_request - FPGA request successfully sent to RPAM A." level := "DIAGNOSTIC” } CHANNEL 7314 { channelId := "PWR-3049" DNChange := 67 dnUnsignedValue := 1600 type := "UNSIGNED_INT" Time := ChannelName := "PWR-BCB1-AMP” }... COMMAND 9626 { Args := ['set_device(1)', 'TRUE'] Time := Stem := "RUN_COMMAND" Number := "18" type := "FSW” } EVR 9627 { message := "Validation failed for command RUN_COMMAND: number=18." DispatchFailure := "RUN_COMMAND" Time := name := "CMD_DISPATCH_VALIDATION_FAILURE" level := "COMMAND" Number := "18” }... recall log

53 log = [ { "OBJ_TYPE" : "COMMAND", "Args" : ['CLEAR_RELAY_PYRO_STATUS'], "Time" : , "Stem" : "POWER_HOUSEKEEPING", "Number" : "4", "Type" : "FSW" }, … { "OBJ_TYPE" : "EVR", "name" : "PWR_REQUEST_CALLBACK", "Time" : , "message" : "power_card_request -\ FPGA request successfully sent to\ RPAM A.", "level" : "DIAGNOSTIC" }, … ]

54 running LogScope import logscope log = extractLog(…) obs = logscope.Observer("specs/rv-tutorial") obs.monitor(log)

55

56 specification learning writing specs is time consuming often hard come up with properties one approach is to use already generated log files to “get ideas” in the extreme case, specifications can be automatically generated from log files

57 architecture learnermonitor logs spec yes no: … spec

58 learner API import logscope log1 = … ; log2 = … ; learner = logscope.ConcreteLearner(“P”) learner.learnlog(log1) learner.learnlog(log2) learner.dumpSpec(sfile) log3 = … ; learner = logscope.ConcreteLearner(“P”,sfile) learner.learnlog(log3) learner.dumpSpec(sfile) log4 = … ; obs = logscope.Observer(sfile) obs.monitor(log4)

59

60 using step and success states

61 conclusions use of static analysis is now required for flight missions. model-based testing is used by LaRS group to test file system – for real. log analysis support shows promise. what’s next?: runtime verification, interaction between static and dynamic analysis, specification learning, trace visualization.

62 monitor the proof obligations static analyzer combining static and dynamic analysis program + spec proof obligations

63 RMOR monitor FileMonitor { symbol open = after call(main.c:openfile); symbol read = after call(main.c:readfile); symbol close = after call(main.c:closefile); symbol send = before call(main.c:senddata); symbol end = before call(main.c:finish); state closed {when open -> unread;} state unread {when read -> read;} state read {when send => error;} super opened [unread,read] { when end => error; when close -> closed; } AspectC like instrumentation state machine specification

64 thanks for listening!