1. Model-based Software Testing and Analysis with C#
Introduction
Slides adapted from ones by Margus Veanes from Microsoft Research

2. Outline Overview Systems with Finite Models
Brief introduction to NModel
Exercises and discussion

3. Overview Describe, analyze, test Why we need model-based testing
Model programs
Model-based analysis
Model-based testing
Model programs in the software process
Why we need model-based testing
Why we need model-based analysis
Further reading

4. Model Program
A program that describes the behavior of another program, system, or component, called the implementation. A model program can act as a specification or a design.
We distinguish between
contract model programs (in C#)
represent all of the allowed behaviors (runs or traces) of the implementation
scenario model programs (in FSM)
represent a collection of runs that are related in some way: a test suite, or a temporal property to check during analysis

5. Sample model program (library model program)
namespace NewsReader {
enum Page { Topics, Messages };
enum Style { WithText, TitlesOnly };
enum Sort { ByFirst, ByMostRecent };
static class NewsReaderUI
static Page page = Page.Topics;
static Style style = Style.WithText;
static Sort sort = Sort.ByMostRecent;
[Action]
static void SelectMessages()
{ page = Page.Messages; }
static bool SelectMessagesEnabled()
{ return (page == Page.Topics); }
static void SelectTopics()
{ page = Page.Topics; }
static bool SelectTopicsEnabled()
{ return (page == Page.Messages); } …
Model Program name
State variables
Action SelectMessages
Enabling condition of SelectMessages

6. Sample model program (FSM model program)
SelectTopics()
SelectMessages() 1
FSM(0, AcceptingStates(1), Transitions(t(0,SelectMessages(),1),
t(1, SelectTopics(), 1))

7. Model-based analysis
Using a model program to debug and improve specifications and designs. The following are model-based analysis techniques:
Simulation,
exploration, and
composition

8. Model-based testing
Using a model program to generate test cases or act as an oracle. Model-based testing includes both
offline testing and
on-the-fly (online) testing
An oracle is the authority which provides the correct result used to make a judgment about the outcome of a test case - whether the test passed or failed.

9. V-diagram for software development

10. V-diagram for model-based testing and analysis

11. Why we need model-based testing
Unit testing is not enough
We need meaningful sequences of actions calls that provide systematic behavioral coverage
In software industry model-based testing is being adapted as an integrated part of the testing process
In academia several model-based testing conferences have been started
Several commercial tools have been developed for model-based testing

12. Why we need model-based analysis
We need to relate the formal model to the natural language specification
Model needs to be validated
design flaws
inconsistencies

13. Systems with finite models

14. Systems with finite models
Finite model programs
Exploring finite model programs
Composing finite model programs

15. Finite model programs (Chapter 5)
States, actions, transitions
Case study: user interface
Preliminary analysis
Coding the model program
Simulation

16. States
A state is a collection of all state variables and their values at a given point in time
Initial state is the initial assignment of values to state variables
namespace NewsReader {
static class NewsReaderUI
static Page page = Page.Topics;
static Style style = Style.WithText;
static Sort sort = Sort.ByMostRecent; …

17. Actions An action is a discrete atomic execution step.
namespace NewsReader {
static class NewsReaderUI
[Action]
static void SelectMessages()
{ page = Page.Messages; }
static bool SelectMessagesEnabled()
{ return (page == Page.Topics); } …

18. Transitions
A transition is the effect of executing an action from a given state.
page = Page.Topics
style = Style.WithText
sort = Sort.ByMostRecent
page = Page.Messages
style = Style.WithText
sort = Sort.ByMostRecent
SelectMessages()

19. Case study: user interface
For our first model program we model the user interface for a web-based news reader, where users select a “group” (news about a particular subject), and can then read and post messages about topics in that group.

20. News Reader Topics page in WithText mode

21. News Reader Topics page in TitlesOnly mode

22. News reader Messages page

23. Preliminary analysis Decide on the purpose
Choose a level of abstraction
Select the actions
Write sample traces
Select the state variables

24. 1. Decide on the purpose
Modeling is a goal-directed activity; every model program has a purpose. We write a model program in order to answer some questions about the implementation (by testing or analysis). We have to formulate those questions before we can see how to write the model program.

25. 1. Decide on the purpose (cont.)
In this example, our questions arise from the observation that the newsreader presents several kinds of pages that can be formatted in various styles, and several options for selecting pages and styles. Our questions are:
What options are available when a particular page and style is displayed?
What page and style appears when the user selects a particular option?

26. 2. Choose a level of abstraction
You should always write your model program at the highest level of abstraction that achieves your purpose (that can answer the questions you have formulated).
Much of the skill in modeling is understanding what can be left out in order to achieve the right level of abstraction.
We use here two kinds of abstraction
Data abstraction deals with state variables.
Behavioral abstraction deals with actions.

27. 2. Choose a level of abstraction (cont.)
In this example we choose a high level of data abstraction where the variables only represent the kind of page that is displayed its style and whether topics are sorted

28. 3. Select the actions We select these actions:
SelectMessages selects the page with messages
SelectTopics selects the page with topics
ShowTitles shows only titles in the topics page
ShowText shows message text in the topics page
SortByFirst and SortByMostRecent select the sort order of the topics page when the titles only are shown.
The name of an action is called an action symbol.
The entire collection of action symbols used by a model program is called its action vocabulary.

29. 4. Write sample traces
A trace is a sequence of actions from the vocabulary we identified.
Drawing on our observations and understanding of how the system is supposed to behave, we write down some traces
This is a sample trace that the system can execute: ShowTitles(),SortByFirst(),SortByMostRecent(),ShowText()
Not all sequences are allowed, for example the system cannot execute: ShowTitles(),SortByFirst(),SortByFirst(),ShowText(),

30. 5. Select the state variables
We must infer the state variables from the intended behavior; e.g., using the sample traces we wrote.
The state variables must store just enough information to determine two things:
control state, what actions are possible (enabled) in each state, and
data state, what data the program can provide to its user or client.
(Sometimes the control state and the data state are mixed, as we will see later on during the course.)

31. 5. Select the state variables (cont.)
We select these state variables:
page indicates which kind of page is displayed,
style indicates the appearance of the entries on the topics page,
sort indicates the sort order of the topics page when titles only are shown
For this example we chose a high level of data abstraction where we ignore the page contents (the titles and messages). Therefore this model program has no data state.

32. Coding the model program
The are two main styles of coding a model program that are supported by NModel:
library model programs are the most general way of coding model programs
FSM model programs provide a convenient way to do scenario control (introduced later)
In NModel a model program extends the ModelProgram base class

33. Library model program Program structure State variables Action methods
Enabling conditions

34. Program structure of a library model program
A model program must include using statements to use the namespaces of the NModel library. A typical model program uses the NModel, NModel.Attributes, and NModel.Execution namespaces. All of these are provided by the library NModel.dll.
A model program is compiled to a library, for example by a command like >csc /t:library /r:NModel.dll NewsReaderUI.cs assuming NModel.dll is in the current directory
The namespace name is the model program name: NewsReader in this example.
A model program consists of the actions and variables defined by the types declared in its namespace.
A model program can include more than one class (or other types).

35. State variables
The state variables of a model program are the static fields and instance fields declared in the model program's namespace. Public and nonpublic fields are included.

36. State variables (cont.)
In our example there are only three state variables. Each has only a few values so we declare an enumerated type for each. In our examples all of the state variables are static variables.
We declare all three, and assign the initial state in the declarations.
namespace NewsReader {
enum Page { Topics, Messages };
enum Style { WithText, TitlesOnly };
enum Sort { ByFirst, ByMostRecent };
static class NewsReaderUI
static Page page = Page.Topics;
static Style style = Style.WithText;
static Sort sort = Sort.ByMostRecent;
...

37. Action methods
The actions of the model program are methods annotated with the [Action] attribute. We call these methods action methods. Each action method must express how the action changes the state.
Here is the ShowTitles action method from this example.
namespace NewsReader { …
static class NewsReaderUI
[Action]
static void ShowTitles()
style = Style.TitlesOnly; }

38. Enabling conditions
In addition to action methods, a model program has methods that check when each action method is enabled (allowed to be invoked). These are called enabling conditions, preconditions or guards.
Writing enabling conditions is the only way to control the sequence of actions that can occur when a model program executes. When coding enabling conditions, it is helpful to refer to the sample traces you wrote during preliminary analysis.
The enabling conditions of an action method have the same name as the action method, with the added suffix Enabled. An action method that has no enabling condition is always enabled.

39. Enabling conditions (cont.)
In this example, the enabling condition for the ShowTitles action method is ShowTitlesEnabled. It expresses that ShowTitles is enabled in all states where the page is a Topics page and the style is WithText.
Q: How many states are there is this model program where ShowTitles is enabled?
namespace NewsReader {
static class NewsReaderUI
static bool ShowTitlesEnabled()
return (page == Page.Topics &&
style == Style.WithText); } …

40. Simulation
One can now execute the model program, e.g. by writing unit tests for it.
Here we try out the Model Program Viewer mpv and use exploration instead as an introduction to the next topic of the course.

41. Exploring finite model programs
Finite State Machines
Exploration
Analysis

42. Finite State Machines (FSMs)
A finite collection of state transitions, along with an identification of an initial state and any accepting states. An FSM can express a scenario.
Can be represented as a table, where each row corresponds to a transition (initial and accepting states are indicated separately)

43. Runs of an FSM
A run of an FSM is a sequence of actions that begins in the initial state and should stop in an accepting state.
Suppose that all states are accepting in the previous table and the initial state is the start state of the first transition (row). Then this is a run of that FSM:
ShowTitles(),SortByFirst(),SortByMostRecent(),ShowText()

44. Scenario FSMs and true FSMs
An FSM whose purpose is to define a scenario is called a scenario FSM. It is possible to define many different scenario FSMs for any model program; each different scenario FSM defines a different collection of runs.
There is one FSM that can generate all of the runs of a finite model program: its true FSM.

45. NewsReader scenario FSM

46. NewsReader true FSM

47. FSM model programs
One can create a model program in NModel from an FSM. The FSM can be input from a file.
For example create a file named test.txt with the following content: FSM(0, AcceptingStates(2), Transitions(t(0,a(),1), t(1,b(),2), t(2,a(),0))) and invoke >mpv /fsm:test.txt

48. Exploration
Exploration automatically generates an FSM from a model program. The FSM can then be used for visualization, analysis, and offline test generation.
We explain how this works
Accessing the model program
Exhaustive exploration
Interactive exploration
An exploration algorithm

49. Accessing the model program
To work with the library or any tool created from it (including mpv), a model program must provide a factory method that creates a ModelProgram object from the compiled model program assembly.
This is the factory method from our news reader model program:
namespace NewsReader {
public static class Factory
public static ModelProgram Create()
return new LibraryModelProgram(typeof(Factory).Assembly,
"NewsReader"); }
...

50. Accessing the model program (cont.)
To invoke a tool on a model program, reference the assembly that contains the compiled model program, and provide the fully qualified name of the factory method in that assembly.
This command invokes mpv on the NewsReader model:
> mpv /r:NewsReaderUI.dll NewsReader.Factory.Create

51. Exhaustive exploration
When mpv is invoked on a finite model program, it exhaustively explores the model program, generates its true FSM, and displays its state transition graph.
For example, invoking mpv on NewsReader as above produces its true FSM. (Try this out)
The graph generated by mpv looks similar to the true FSM shown before but is not exactly the same - mpv simply labels each node with a number. The underlying state variables in a state can be inspected by using the state viewer in mpv.

52. Interactive exploration
The mpv tool can also explore interactively, executing and displaying a few transitions at a time, under the user's control. This provides a more convenient way to simulate particular runs than coding and executing main methods or unit tests.
The option initialTransitions of mpv can be used to specify how many transitions are explored initially. Try the command:
>mpv /r:NewsReaderUI.dll NewsReader.Factory.Create /initialTransitions:0

53. Exploration Algorithm used in mpv
ModelProgramSimple m = ModelName.Factory.Create();
Sequence<State> frontier = new Sequence<State>(m.InitialState);
Sequence<State> explored = Sequence<State>.EmptySequence;
Set<Transition> transitions = Set<Transition>().EmptySequence;
while (!frontier.IsEmpty) {
State current = frontier.Head; // choose first element of frontier
frontier = frontier.Tail; // all but first element of frontier
explored = explored.AddLast(current); // append current to explored
foreach (Action a in m.GetActions(current))
State next = m.GetNextState(current, a);
if (!frontier.Contains(next) && !explored.Contains(next))
frontier = frontier.AddLast(next); // append for breadth-first }
transitions = transitions.Add(Transition(current, a, next));

54. Safety analysis
Safety analysis checks whether anything bad can happen. We express safety requirements by writing a Boolean expression called an invariant, which is supposed to be true in every state that can be reached. An unsafe state is a state where an invariant is false. In a library model program, an invariant is a Boolean field, property, or method labeled with the [StateInvariant] attribute.
For example, we can add the following dummy invariant to the news reader model program: (using mpv we can turn safety analysis on/off)
[StateInvariant]
public static bool Safe() {
return (style == Style.TitlesOnly ||
sort == Sort.ByMostRecent); }

55. Liveness analysis
Liveness analysis checks whether something good will happen. It searches for dead states from which an accepting state cannot be reached.. We express liveness requirements by identifying accepting states by a predicate labeled with the [AcceptingStateCondition] attribute.