1. Issue# UMLTP21-3 Related issues
UTP test log facility ought to enable logging on procedural element level
Issue# UMLTP21-3 Related issues

2. Purpose of a test log Calculate coverage and determine progress
Analysis of test results (as part of test evaluation activities)
Basis for confirmation test (running the very same test with the exact same values again)
Comparison of subsequent test runs of the very same test case
Post-execution comparison
Post-execution comparison calculates the verdict after test execution
Requires detailed test logs; arbitration can be done much later
Trend analysis
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

3. RPF (excerpt)
ISTQB/ISO 24765: „A chronological record of relevant details about the execution of tests”
IS (Test Execution Log): document that records details of the execution of one or more test procedures
Execution of (at least) the test actions is (IMHO) a relevant detail about the execution of tests, first an foremost, CreateStimulusAction and ExpectResponseAction
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

4. No mechanism to log the execution details of a ProceduralElement
Current situation
COP for test design - web site, forum, group, learning structure, topics project, career
No mechanism to log the execution details of a ProceduralElement
Doc#:

5. This is possible… Doc#:
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

6. This is not possible (but ought to be, though)
COP for test design - web site, forum, group, learning structure, topics project, career
Execution of CreateStimulusAction
Execution of ExpectResponseAction
Doc#:

7. Proposal
Create a minimal test log model that is capable of recording the execution details of the following procedural elements
CreateStimulusAction
ExpectResponseAction
CheckPropertyAction
LogAction
SuggestVerdictAction
ProcedureInvocation (including actual parameter)
Mainly for logging the invocation of parameterized test cases from test execution logs
Logging other procedural elements (such as loops, parallels, alternatives etc.) are of lesser interest for the time being
Minimal log model must be easily extandable with further log entries
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

8. Proposal
TestLogEntries shall be based on InstanceSpecifications (and Classes for additional details), since there is no such thing as an instance of an Event, MessageOccurrenceSpecification or Action in UML
A set of standardised information shall be associated with every TestLogEntry (similar to TestLog)
Timestamp
Duration
Executing entity
User-specific extension of that standard information must be possible to enable customization
Possible through Classes of «TestLogEntry» InstanceSpecification that add additional information as Properties and values as Slots
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

9. Proposal
The proposal introduces the minimal test log model in the following order:
Extending test log with test log entries
Basic Test Log Entries
Refinement of Test Log Entries for Test Actions
Declaration of Event structures
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

10. 1. Extending UTP test log with test log entries
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

11. 1. Extending UTP test log with test log entries
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

12. 1. Extending UTP test log with test log entries
TestLogEntry is a TestLog (of a ProceduralElement execution)
Has a starting point in time
A duration
And an (optional) executing entity
TestLogEntry may add additional (non-standardised) structural information by using TestLogEntryStructure
TestLogEntry is an InstanceSpecification that can be classified by a «TestLogEntryStructure» Class
Used to define formal and actual parameter for invocation events later on
TestCaseLog and TestSetLog have and ordered set of TestLogEntries
Ordered by timestamp
Represents the sequential execution of events during test case execution
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

13. 1. Extending UTP test log with test log entries
Examples
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

14. 2. Basic Test Log Entries Doc#:
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

15. 2. Basic Test Log Entries Doc#:
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

16. 2. Basic Test Log Entries Doc#:
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

17. 3. Refinement of Test Log Entries for Test Actions
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

18. 3. Refinement of Test Log Entries for Test Actions
Derived from corresponding InvocationActionLogEntry
COP for test design - web site, forum, group, learning structure, topics project, career
Derived from corresponding InvocationActionLogEntry
Doc#:

19. 4. Declaration of event structures
Log entries for InvocationEvents usually require additional information
Invoked Operation/Method (BehavioralFeature/Behavior in UML)
Values that were passed into or received from the callee (i.e., actual parameter values)
ProceduralElements that invoke something
ProcedureInvocationAction
CreateStimulusAction
ExpectResponseAction
Respective TestLogEntries already exist!
Event-specific structural information (i.e. parameters) of the invocation added through specific TestLogEntryStructures
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

20. 4. Declaration of event structures
3 possible invocation events that can be recorded during test execution
All declare formal parameters
2 posses only input parameter
3 TestLogEntryStructures shall be defined an reused (instantiated) for a concrete test log
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

21. 4. Declaration of event structures
Derived from underlying Slot.value
Order determined by order of Parameter of ‚parameterizedInvocation‘
Derived from underlying Slot.definingFeature
COP for test design - web site, forum, group, learning structure, topics project, career
Parameter direction kind derived from ‚formalParameter‘ directionKind
Doc#:

22. A real example
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

23. Example (UTEST)
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

24. Example (UTEST)
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

25. Example (UTEST)
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

26. Example (UTEST) derived automatically derived automatically
COP for test design - web site, forum, group, learning structure, topics project, career
derived automatically
Doc#:

27. Example (UTEST) Instance of Represents execution of Instance of
Only «ActualParameterValue»s that correspond to in-Kind-Parameter are allowed for CreateStimulusLogEntry
Instance of
COP for test design - web site, forum, group, learning structure, topics project, career
Represents execution of
Instance of
Only «ActualParameterValue»s that correspond to out-Kind-Parameter are allowed for ActualResponseLogEntry
Represents execution of
Doc#:

28. Example (UTEST) - Tooling
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#:

29. Summary
UTP 2 shall support test logs an procedural element level (according to RFP)
Test logging on procedural element level unifies the evaluation of test logs and adheres by the ideo of comprehensive test specifications
Test runs from multiple test execution tools can be uniquely accessed and evaluated
Post-execution comparison is possible
Trend analysis based on UTP 2
Test log entries can be easily extended with user-specific information using UML‘s InstanceSpecification framework
The most important procedural elements are supported for the time being; further procedural elements can be incorporated in the exact same way in a later RTF
COP for test design - web site, forum, group, learning structure, topics project, career
Doc#: