2.4 Markov processes

A Markov process^2.4 is a stochastic process that has a limited form of ``historical'' dependency [65]. Let $\{ X(t)~:~t\in \rm {\cal{T}} \}$ be defined on the parameter set $\rm {\cal{T}}$ and assume that it represents time. The values that $X(t)$ can obtain are called states, and all together they define the state space ${\cal {S}}$ of the process. A stochastic process is a Markov process if it satisfies

$$P [ X(t_0+t_1)\leq x~\vert~X(t_0)=x_0,~X(\tau), -\infty<\tau... ...= P [ X(t_0+t_1)\leq x~\vert~X(t_0)=x_0 ],~\forall t_0,t_1>0 .$$ (2.6)

Let $t_0$ be the present time. Eq.(2.6) states that the evolution of a Markov process at a future time, conditioned on its present and past values, depends only on its present value [65]. The condition of Eq.(2.6) is also known as the Markov property. Markov chains are classified as discrete or continuous.