Markov Reward Models By H. Momeni Supervisor: Dr. Abdollahi Azgomi

Markov Reward Models 2 31 Contents  Modeling Taxonomy  Markov Reward Models Definition  Reliability measures  Availability measures  Performance measures  Conclusion

Markov Reward Models 3 31 MODELING TAXONOMY “All Models are Wrong; Some Models are Useful” George Box Modeling Simulation Analytic modeling Non-State-Space Method State-Space Method

Markov Reward Models 4 31 Non-State-Space Modeling Taxonomy Non-State-Space method Performance modelsDependability models Queuing models Reliability Block Diagram models Fault Tree models

Markov Reward Models 5 31 State Space Modeling Taxonomy Markovian models Non-Markovian models discrete-time Markov chains continuous-time Markov chains Markov reward models Semi-Markov process Markov regenerative process Non-Homogeneous Markov State space models

Markov Reward Models 6 31 Motivation  Extension of CTMC to Markov reward models make them even more useful  Markov reward models is used as a means to obtain performance and dependability measures.

Markov Reward Models 7 31 Dependability Concepts DEPENDABILITY ATTRIBUTES AVAILABILITY RELIABILITY SAFETY CONFIDENTIALITY INTEGRITY MAINTAINABILITY FAULT PREVENTION FAULT REMOVAL FAULT TOLERANCE FAULT FORECASTING MEANS THREATS FAULTS ERRORS FAILURES SECURITY Faults are the cause of errors that may lead to failures FaultErrorFailure

Markov Reward Models 8 31 MRM Formal Definition  A Markov reward model consists of a continuous time Markov chain X={X(t), t  0)} with a finite state space S, and a reward function r where r:S   Usually, for each state i  S, r(i) represents the reward obtained per unit time in that state  With MRMs, rewards can assign to states or transitions  The reward rates are defined based on the system requirements (availability, reliability, performance, … )

Markov Reward Models 9 31 Formal Definitions  is the system reward rate at time t  Accumulated reward in the interval [0, t) is denoted as  The expected accumulated reward is Li(t) denotes the expected total time the CTMC spends in state i during the interval [0, t]

Markov Reward Models Formal Definition (cont’d)  Let  i be the steady state probability for state i  The expected steady-state reward rate is  The expected instantaneous reward rate is

Markov Reward Models Example  A three state Markov Reward model  The reward rate vector is r=(3,1,0)  Initial probability vector is

Markov Reward Models 12 31

Markov Reward Models Case Study  Consider a multiprocessor system with n processor elements processing a given workload

Markov Reward Models System Availability  Definition: The availability of a system at time t (A(t)) is the probability that the system is accessible to perform its tasks correctly  Availability measures are based on a binary reward structure  One processor is sufficient for the system to be up, otherwise it is considered as being down  Set of states where and  Reward rate 1 is attached to the states in U and a reward rate 0 to those in D

Markov Reward Models System Availability  Reward function r is:  Instantaneous availability is : Availability reward rates

Markov Reward Models System Availability  Unavailability can be calculated with a reverse reward assignment to that for availability  Steady state availability

Markov Reward Models System Availability  There are related measures that do not rely on the binary reward structure (e.g. uptime, number of repair calls)  Mean transient uptime Mean uptimes reward rates

Markov Reward Models System Availability  Very important measures related to the frequency of certain events of interest (e.g. average number of repair calls in [0,t) )  With repair rate the transient average number of repair call and steady-state Reward rates for average number of repair calls in [0,t)

Markov Reward Models System Reliability  Definition : The reliability of a system at time t (R(t)) is the probability that the system operation is proper throughout the interval [0,t]  A binary reward function r is defined that assigns reward rates 1 to up states and reward rates 0 to down states.

Markov Reward Models System Reliability  Reliability is the likelihood that an unwanted event has not yet occurred since the beginning of the system operation.  T is the time to the next occurrence of an unwanted (failure) event Reward rates for reliability

Markov Reward Models System Reliability 

Markov Reward Models System Reliability  Mean time to the occurrence of an unwanted (failure) event is given by:  Unreliability follows as the complement:  The unreliability also could be calculated based on a reward assignment complementing the one in Table

Markov Reward Models System Reliability  Related to Reliability measures, the expected number of catastrophic events C(t) in [o,t) is important Reward assignment for predicting the number of catastrophic incidents

Markov Reward Models System and Task Performance  Definition: measure of responsiveness  The use of reward rates is not restricted to availability, reliability and performability models  This concept can also be used in pure (failure-free) performance models (e.g. throughput, response time, utilization, total task loss probability)

Markov Reward Models System and Task Performance  The values are used to characterize the percentage loss of tasks arriving at the system in state Reward rates for computing the total loss probability Reward rates for throughput

Markov Reward Models System and Task Performance  The expected total loss probability, TLP, in the steady state an transient state TLP(t) are:

Markov Reward Models System and Task Performance  Throughput can be achieved by assigning state transition rates corresponding to departure from a queue (service completion) as reward rates  Mean response time can be achieved by assigning number of customers present in a state as a reward rate  Utilization is based on binary reward structure, if a particular resource is occupied in a given state, reward rate 1 is assigned, otherwise reward rate 0, indicates the idleness of the resources.

Markov Reward Models System and Task Performance Mean number of customers reward rates Throughput reward ratesUtilization reward rates  imagine customers arriving at a system with λ, service time is μ  Single server

Markov Reward Models Performance’s Measures  Throughput  Mean number of customers  Mean response time –Use Little ’ s law  Utilization

Markov Reward Models Conclusion  MRM is State space model  MRM is more useful than CTMC to obtain Performance and dependability measures  Reward Rates are assigned based on system requirements  Structure of Reward rate can be various (usually binary)  Stochastic Reward Nets (SRN) are an extension on SPN that assign reward rate to transitions

Markov Reward Models 31 References  Gunter Bluch et al, Queuing network and markov chain, 2nd Ed., John Wiley and Sons, 2006  J.c. Laprie, Fundamental Concepts of Dependability, IEEE Transaction, 2004  K. Trivedi, Probability and Statistics with Reliability, Queuing, and Computer Science Applications, 2nd Ed., John Wiley and Sons, New York, 2001  B. Haverkort et al, Performability Modeling, John Wiley, 2001