Issue# UMLTP21-3 Related issues UTP test log facility ought to enable logging on procedural element level Issue# UMLTP21-3 Related issues

Purpose of a test log Calculate coverage and determine progress Analysis of test results (as part of test evaluation activities) Basis for confirmation test (running the very same test with the exact same values again) Comparison of subsequent test runs of the very same test case Post-execution comparison Post-execution comparison calculates the verdict after test execution Requires detailed test logs; arbitration can be done much later Trend analysis COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

RPF (excerpt) ISTQB/ISO 24765: „A chronological record of relevant details about the execution of tests” IS 29119 (Test Execution Log): document that records details of the execution of one or more test procedures Execution of (at least) the test actions is (IMHO) a relevant detail about the execution of tests, first an foremost, CreateStimulusAction and ExpectResponseAction COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

No mechanism to log the execution details of a ProceduralElement Current situation COP for test design - web site, forum, group, learning structure, topics project, career No mechanism to log the execution details of a ProceduralElement Doc#:

This is possible… Doc#: COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

This is not possible (but ought to be, though) COP for test design - web site, forum, group, learning structure, topics project, career Execution of CreateStimulusAction Execution of ExpectResponseAction Doc#:

Proposal Create a minimal test log model that is capable of recording the execution details of the following procedural elements CreateStimulusAction ExpectResponseAction CheckPropertyAction LogAction SuggestVerdictAction ProcedureInvocation (including actual parameter) Mainly for logging the invocation of parameterized test cases from test execution logs Logging other procedural elements (such as loops, parallels, alternatives etc.) are of lesser interest for the time being Minimal log model must be easily extandable with further log entries COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Proposal TestLogEntries shall be based on InstanceSpecifications (and Classes for additional details), since there is no such thing as an instance of an Event, MessageOccurrenceSpecification or Action in UML A set of standardised information shall be associated with every TestLogEntry (similar to TestLog) Timestamp Duration Executing entity User-specific extension of that standard information must be possible to enable customization Possible through Classes of «TestLogEntry» InstanceSpecification that add additional information as Properties and values as Slots COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Proposal The proposal introduces the minimal test log model in the following order: Extending test log with test log entries Basic Test Log Entries Refinement of Test Log Entries for Test Actions Declaration of Event structures COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

1. Extending UTP test log with test log entries COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

1. Extending UTP test log with test log entries COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

1. Extending UTP test log with test log entries TestLogEntry is a TestLog (of a ProceduralElement execution) Has a starting point in time A duration And an (optional) executing entity TestLogEntry may add additional (non-standardised) structural information by using TestLogEntryStructure TestLogEntry is an InstanceSpecification that can be classified by a «TestLogEntryStructure» Class Used to define formal and actual parameter for invocation events later on TestCaseLog and TestSetLog have and ordered set of TestLogEntries Ordered by timestamp Represents the sequential execution of events during test case execution COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

1. Extending UTP test log with test log entries Examples COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

2. Basic Test Log Entries Doc#: COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

2. Basic Test Log Entries Doc#: COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

2. Basic Test Log Entries Doc#: COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

3. Refinement of Test Log Entries for Test Actions COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

3. Refinement of Test Log Entries for Test Actions Derived from corresponding InvocationActionLogEntry COP for test design - web site, forum, group, learning structure, topics project, career Derived from corresponding InvocationActionLogEntry Doc#:

4. Declaration of event structures Log entries for InvocationEvents usually require additional information Invoked Operation/Method (BehavioralFeature/Behavior in UML) Values that were passed into or received from the callee (i.e., actual parameter values) ProceduralElements that invoke something ProcedureInvocationAction CreateStimulusAction ExpectResponseAction Respective TestLogEntries already exist! Event-specific structural information (i.e. parameters) of the invocation added through specific TestLogEntryStructures COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

4. Declaration of event structures 3 possible invocation events that can be recorded during test execution All declare formal parameters 2 posses only input parameter 3 TestLogEntryStructures shall be defined an reused (instantiated) for a concrete test log COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

4. Declaration of event structures Derived from underlying Slot.value Order determined by order of Parameter of ‚parameterizedInvocation‘ Derived from underlying Slot.definingFeature COP for test design - web site, forum, group, learning structure, topics project, career Parameter direction kind derived from ‚formalParameter‘ directionKind Doc#:

A real example COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Example (UTEST) COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Example (UTEST) COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Example (UTEST) COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Example (UTEST) derived automatically derived automatically COP for test design - web site, forum, group, learning structure, topics project, career derived automatically Doc#:

Example (UTEST) Instance of Represents execution of Instance of Only «ActualParameterValue»s that correspond to in-Kind-Parameter are allowed for CreateStimulusLogEntry Instance of COP for test design - web site, forum, group, learning structure, topics project, career Represents execution of Instance of Only «ActualParameterValue»s that correspond to out-Kind-Parameter are allowed for ActualResponseLogEntry Represents execution of Doc#:

Example (UTEST) - Tooling COP for test design - web site, forum, group, learning structure, topics project, career Doc#:

Summary UTP 2 shall support test logs an procedural element level (according to RFP) Test logging on procedural element level unifies the evaluation of test logs and adheres by the ideo of comprehensive test specifications Test runs from multiple test execution tools can be uniquely accessed and evaluated Post-execution comparison is possible Trend analysis based on UTP 2 Test log entries can be easily extended with user-specific information using UML‘s InstanceSpecification framework The most important procedural elements are supported for the time being; further procedural elements can be incorporated in the exact same way in a later RTF COP for test design - web site, forum, group, learning structure, topics project, career Doc#: