Transition probabilities

The probabilities of transition of a Markov chain $\xi (t)$ from a state $i$ into a state $j$ in a time interval $[s,t]$:

$$p_{ij}(s,t)=\mathsf{P}\{\xi (t)=j\mid \xi (s)=i\},s<t.$$

In view of the basic property of a Markov chain, for any states $i,j\in S$ (where $S$ is the set of all states of the chain) and any $s<t<u$,

$$p_{ij}(s,u)=\sum _{k\in S}{p}_{ik}(s,t)p_{kj}(t,u).$$

One usually considers homogeneous Markov chains, for which the transition probabilities $p_{ij}(s,t)$ depend on the length of $[s,t]$ but not on its position on the time axis:

$$p_{ij}(s,t)=p_{ij}(t-s).$$

For any states $i$ and $j$ of a homogeneous Markov chain with discrete time, the sequence $p_{ij}(n)$ has a Cesàro limit, i.e.

$$\underset{n\to \mathrm{\infty }}{lim}\frac{1}{n}\sum _{k=1}^n{p}_{ij}(k)\ge 0.$$

Subject to certain additional conditions (and also for chains with continuous time), the limit exists also in the usual sense. See Markov chain, ergodic; Markov chain, class of positive states of a.

The transition probabilities $p_{ij}(t)$ for a Markov chain with discrete time are determined by the values of $p_{ij}(1)$, $i,j\in S$; for any $t>0$, $i\in S$,

$$\sum _{j\in S}{p}_{ij}(t)=1.$$

In the case of Markov chains with continuous time it is usually assumed that the transition probabilities satisfy the following additional conditions: All the $p_{ij}(t)$ are measurable as functions of $t\in (0,\mathrm{\infty })$,

$$\underset{t\downarrow 0}{lim}{p}_{ij}(t)=0(i\ne j),\underset{t\downarrow 0}{lim}{p}_{ii}(t)=1,i,j\in S.$$

Under these assumptions the following transition rates exist:

$$\begin{array}{}\text{(1)}& {\lambda }_{ij}=\underset{t\downarrow 0}{lim}\frac{1}{t}(p_{ij}(t)-p_{ij}(0))\le \mathrm{\infty },i,j\in S;\end{array}$$

if all the ${\lambda }_{ij}$ are finite and if $\sum _{j\in S}{\lambda }_{ij}=0$, $i\in S$, then the $p_{ij}(t)$ satisfy the Kolmogorov–Chapman system of differential equations

$$\begin{array}{}\text{(2)}& p_{ij}^{\prime }(t)=\sum _{k\in S}{\lambda }_{ik}{p}_{kj}(t),p_{ij}^{\prime }(t)=\sum _{k\in S}{\lambda }_{kj}{p}_{ik}(t)\end{array}$$

with the initial conditions $p_{ii}(0)=1$, $p_{ij}(0)=0$, $i\ne j$, $i,j\in S$ (see also Kolmogorov equation; Kolmogorov–Chapman equation).

If a Markov chain is specified by means of the transition rates (1), then the transition probabilities $p_{ij}(t)$ satisfy the conditions

$$p_{ij}(t)\ge 0,\sum _{j\in S}{p}_{ij}(t)\le 1,i,j\in S,t>0;$$

chains for which $\sum _{j\in S}{p}_{ij}(t)<1$ for certain $i\in S$ and $t>0$ are called defective (in this case the solution to (2) is not unique); if $\sum _{j\in S}{p}_{ij}(t)=1$ for all $i\in S$ and $t>0$, the chain is called proper.

Example. The Markov chain $\xi (t)$ with set of states $\{0,1,\dots \}$ and transition densities

$${\lambda }_{i,i+1}=-{\lambda }_{ii}={\lambda }_i>0,{\lambda }_{ij}=0(i\ne j\ne i+1)$$

(i.e., a pure birth process) is defective if and only if

$$\sum _{i=0}^{\mathrm{\infty }}\frac{1}{{\lambda }_i}<\mathrm{\infty }.$$

Let

$${\tau }_{0n}=inf\{t>0:\xi (t)=n(\xi (0)=0)\},$$

$$\tau =\underset{n\to \mathrm{\infty }}{lim}{\tau }_{0n};$$

then

$$\mathsf{E}\tau =\sum _{i=1}^{\mathrm{\infty }}\frac{1}{{\lambda }_i}$$

and for $\mathsf{E}\tau <\mathrm{\infty }$ one has $\mathsf{P}\{\tau <\mathrm{\infty }\}=1$, i.e. the path of $\xi (t)$" tends to infinity in a finite time with probability 1" (see also Branching processes, regularity of).

References[edit]

Comments[edit]

For additional references see also Markov chain; Markov process.

In (1), ${\lambda }_{ij}\ge 0$ if $i\ne j$ and ${\lambda }_{ii}\le 0$.

References[edit]