2.4.2 Continuous time Markov chains (CTMCs)

In a CTMC, the process makes a transition from one state to another, after it has spent an amount of time on the state it starts from. This amount of time is defined as the state holding time. In a DTMC the holding time is geometrically distributed, while in a CTMC it is exponentially distributed. All DTMC definitions apply for CTMCs as well. In the same way that we build the probability transition matrix for a DTMC, we construct the infinitesimal generator matrix ${\mathbf{Q}}$ of a CTMC. The entries of the infinitesimal generator matrix ${\mathbf{Q}}$ are the rates at which the process jumps from state to state. By definition, the diagonal entries of ${\mathbf{Q}}$ are equal to minus the total rate out of the state that corresponds to that row, $q_{i,i} = - \sum_{j\neq i}^{\infty} q_{i,j}$. This implies that the row sums of ${\mathbf{Q}}$ equal $0$:

$${\mathbf{Q}}\stackrel{\rm def}{=} \left[ \begin{array}{c c c... ...\vdots &\vdots & \cdots & \vdots & \cdots \end{array}\right] .$$ (2.12)

Similar to DTMCs, the following proposition holds for CTMCs.

Proposition[65] The stationary probability vector $\mbox{\boldmath {$\pi$}}$ of an irreducible CTMC in an ergodic set of states is unique and satisfies

$$\mbox{\boldmath {$\pi$}}\cdot {\mathbf{Q}}= 0$$ (2.13)

and the normalization condition

$$\mbox{\boldmath {$\pi$}}\cdot {\mathbf{1}}^T= 1.$$ (2.14)

When the CTMC process is in steady state, the property of flow balance holds, and Eq.(2.13) represents all the flow balance equations of the CTMC.