Unità di Perugia e di Roma “Tor Vergata” "Uncertain production systems: optimal feedback control of the single site and extension to the multi-site case" workshop Ottimizzazione e Controllo delle Supply Chain Siena, Certosa di Pontignano, ottobre 2005 Francesco Martinelli Fabio Piedimonte Università di Roma “Tor Vergata” Mauro Boccadoro Paolo Valigi Università di Perugia

Unità di Perugia e di Roma "Tor Vergata" 2/31 x(t)  (t) u(t) backlog/inventory level at time t (fluid model) x(t):  (t): The machine is failure prone,  (t)=1 if the machine is up at time t,  (t)= 0 if the machine is down, with failures and working times characterized by some deterministic or random law, depending on the production control d

Unità di Perugia e di Roma "Tor Vergata" 3/31 Two main objectives: In the literature, in the Markov case, it has been observed (mainly numerically) a relevant difference between the case the failure rate is a convex function of the production rate and the case it is concave [Hu Vakili Yu, 1994; Liberopoulos Caramanis, 1994] Explore this analytically in the Markovian and in the non Markovian (deterministic) case Several papers on single failure prone machines: Explore the multi-site case where the production of each site may be increased by the production of the other, with some penalty (modeling for example transportation costs)

Unità di Perugia e di Roma "Tor Vergata" 4/31 Minimize u(t) 0 x d cpcp cmcm g(x) Backlog Inventory

Unità di Perugia e di Roma "Tor Vergata" 5/31 01 Machine down Machine up ququ q d (u) Markov The site is modeled as a failure prone machine with a failure-repair process which can be: Deterministic Deterioration rate: The machine is stopped for a repair/maintainance operation when z(t)=1 The single site case

Unità di Perugia e di Roma "Tor Vergata" 6/31 Optimal policy: hedging point policy (Kimemia and Gershwin, 1983; Bielecki and Kumar, 1988) t x(t) z Single site, Markov: the homogeneous case (q d constant)

Unità di Perugia e di Roma "Tor Vergata" 7/31 u q d (u) U  q d1 q d2 d Single site, Markov: a non homogeneous case (q d =q d (u))

Unità di Perugia e di Roma "Tor Vergata" 8/31 (OPT) t x(t) Z X Single site, Markov: a non homogeneous case (q d =q d (u))

Unità di Perugia e di Roma "Tor Vergata" 9/31 Single site, Markov: a non homogeneous case (q d =q d (u)) Procedure followed for the proof and for the computation of the optimal thresholds X* and Z* Take X  Z and apply policy (OPT). At steady state the buffer level is a random variable with pdf: where: and

Unità di Perugia e di Roma "Tor Vergata" 10/31 Single site, Markov: a non homogeneous case (q d =q d (u)) For the level x=Z, there is a point mass probability  (X,Z):=K 0 (X,Z)d/q d2 Z  X have to be properly selected to minimize: Once X* and Z* have been found and the optimal J* has been derived, compute the cost-to-go functions solving the HJB equations where the min operation has been replaced by the (supposed) optimal policy u*(x):

Unità di Perugia e di Roma "Tor Vergata" 11/31 Single site, Markov: a non homogeneous case (q d =q d (u)) Once the cost-to-go functions V 0 (x) and V 1 (x) have been computed, show that these functions, with the policy considered to compute them, satisfy the following HJB equations: If these equations are satisfied and the cost-to-go functions are C 1 and bounded by a quadratic function, then the considered policy is optimal.

Unità di Perugia e di Roma "Tor Vergata" 12/31 Single site, Markov: a non homogeneous case (q d =q d (u)) Computation of X* and Z*

Unità di Perugia e di Roma "Tor Vergata" 13/31 Single site, Markov: a non homogeneous case (q d =q d (u))

Unità di Perugia e di Roma "Tor Vergata" 14/31 Single site, Markov: a non homogeneous case (q d =q d (u))

Unità di Perugia e di Roma "Tor Vergata" 15/31 Single site, Markov: a non homogeneous case (q d =q d (u))  =30; U=22; q d1 =0.06; d=20; c m =100; c p =1; q u =0.5 Example

Unità di Perugia e di Roma "Tor Vergata" 16/31 Single site, Markov: a general heuristic approach for the non homogeneous case In the general case we propose the following heuristic approach: discretize q d (u) obtaining a multi-value failure rate function with production levels U i and corresponding failure rates q di apply the results of the two level failure rate case to the multi-value case by considering each couple (U i, U j ) and the corresponding q di and q dj : this gives a threshold X * ij, such that select the longest sequence of all the X * ij computed Example: x

Unità di Perugia e di Roma "Tor Vergata" 17/31 For multi-value failure rate functions (as the ones obtained by discretizing q d (u) = a u  + b), Liberopoulos and Caramanis (IEEE TAC 1994) numerically found that: if  ≤1, the optimal feedback policy will operate the machine at maximum rate until a safety stock Z * is reached (i.e. it is a hedging point policy) if  >1, the optimal feedback policy will operate the machine progressively reducing the production rate from its maximum value as the inventory level increases The heuristic proposed above confirms these findings. Z*Z*  x u * (x)  x Z*Z* Single site, Markov: a general heuristic approach for the non homogeneous case

Unità di Perugia e di Roma "Tor Vergata" 18/31 Single site, Markov: a general heuristic approach for the non homogeneous case Example  =50; d=1; c m =1000; c p =1; q u =0.5

Unità di Perugia e di Roma "Tor Vergata" 19/31 Single site, Markov: a general heuristic approach for the non homogeneous case Example For q d2 =0.01 the points (U i,q di ) lie on a line. U 1 =50; U 2 =25; U 3 =5; q d1 =0.02; q d3 =0.002; d=1; c m =1000; c p =1; q u =0.5

Unità di Perugia e di Roma "Tor Vergata" 20/31 The discussion above seems in conflict with the results of Hu, Vakili and Yu (IEEE TAC, 1994) where hedging policy is proved optimal iff  =0 or 1. Remark. This is not a conflict: if 0<  <1 the optimal policy probably is a switched non-feedback policy, with the hedging point policy remaining optimal among feedback policies. Single site, deterministic To clarify this we have considered a deterministic system and approached it through the Maximum Principle. g(x) =c x 2 To simplify the analysis we have considered a symmetric system and a quadratic cost function Deterioration rate: The machine is stopped when z(t)=1. After each repair z=0. The system is stable if and only if there exists a constant production rate (not larger than  ) which is large enough to meet the demand

Unità di Perugia e di Roma "Tor Vergata" 21/31 The analysis of this case confirms the heuristic and the numerical results of the Markov system: Single site, deterministic If  =0 or  =1 (affine case) the optimal policy is  -d-  (similar to the hedging point policy) x(t) 0 If 0<  1, the optimal policy looks macroscopically like the  -d-  but an infinite number of switches between 0 and  is performed to obtain a production rate equal to d If  >1, the optimal policy reduces the production rate around 0 0  lim 0 x(t) 0

Unità di Perugia e di Roma "Tor Vergata" 22/31 Multi site, Markov, homogeneous Each site is like the one considered by the classical paper of Bielecki and Kumar, for which the optimal policy is optimal. x  (t)   (t) u  (t) x  (t)   (t) u  (t) u  (t) d d u  (t) A penalty cost (a) is incurred whenever a site receives items produced by the other site A two site system

Unità di Perugia e di Roma "Tor Vergata" 23/31 Multi site, Markov, homogeneous Using a dynamical programming approach, in the s=(1,1) operational state, it is possible to expect the following regions, whose shape in the state space (x 1,x 2 ) is usually very complex to derive: V 11 (x) being the cost-to-go function in the operational state (1,1)

Unità di Perugia e di Roma "Tor Vergata" 24/31 Multi site, Markov, homogeneous Through a numerical integration of the HJB equations (for a finite inventory system with loss cost R,  x=0.1), we have derived the following solutions, corresponding to the s=(1,1) state (arrows denote the production flow): a=10 a=50 a=  System parameters:  =5, d=4, q u =1, q d =0.01, c m =50, c p =1, R=2500

Unità di Perugia e di Roma "Tor Vergata" 25/31 Multi site, Markov, homogeneous In the case the operational state is s=(0,1) and a=50:

Unità di Perugia e di Roma "Tor Vergata" 26/31 Multi site, Markov, homogeneous Single site theoretical values: z * =3.8, J*=7.73 Hedging point and total cost as a function of the cost parameter a:

Unità di Perugia e di Roma "Tor Vergata" 27/31 Multi site, Markov, homogeneous Numerical solution through Hamilton Jacobi Bellman (HJB) equations Performance index to minimize Optimal value: J * J s (k) (x) The minimum average expected cost on a time horizon k  t, starting in (s,x), hence it is 0 for k=0 for all s and x Iterative equation (discretized space): lim k !1 J s (k) (x) = J * It gives the optimal minimum cost J * but not the optimal policy

Unità di Perugia e di Roma "Tor Vergata" 28/31 Multi site, Markov, homogeneous Applying a stable stationary policy, let at steady state J=E[g(x,u)] Then define a differential cost: The total (not average) expected cost in [0,T] from x(0)=x and s(0)=s can be written as J T + V s (x). For the optimal policy, J=J * and we have for its differential cost: V s (k) (x) The minimum expected differential cost on a time horizon k  t, starting in (s,x), hence it is 0 for k=0 for all s and x Iterative equation (discretized space): lim k !1 V s (k) (x) = V * s (x) From V* s (x) it is straightforward to get the optimal policy

Unità di Perugia e di Roma "Tor Vergata" 29/31 Multi site, Markov, homogeneous

Unità di Perugia e di Roma "Tor Vergata" 30/31 A single site and a multi site system have been considered in this research. As for the single site problem: A similar behavior has been observed in a deterministic scenario where the machine is characterized by a deterioration rate which is a deterministic function of the production rate The optimal analytical solution for a non homogeneous Markov failure prone system has been completely derived This solution has been used to investigate (through a heuristic approach) the property observed in the literature that a major difference arises when the failure rate of the machine is a concave or a convex function of the production rate As for the multi site problem, a HJB approach has been used to analyze a Markov, homogeneous, two site system, and the optimal solution has been completely derived numerically for some examples

Unità di Perugia e di Roma "Tor Vergata" 31/31 The general Markov non homogeneous case could be better analyzed, improving the heuristic and studying its validity The deterministic case should be generalized and possibly approached through a numerical algorithm to solve the maximum principle equations As for the single site problem: As for the multi site problem: More general models to describe some typical dynamical phenomena of supply chains are under investigation