\[ \newcommand{\Exg}{\operatorname{\mathbb{E}}} \newcommand{\Ex}{\mathbb{E}} \newcommand{\Ind}{\mathbb{I}} \newcommand{\Var}{\operatorname{Var}} \newcommand{\Cov}{\operatorname{Cov}} \newcommand{\Corr}{\operatorname{Corr}} \newcommand{\ee}{\mathrm{e}} \]

18  Markov chains in the long run

Summary:

• The \(n\)-step transition probabilities \(p^{(n)}(x,y) = \mathbb{P}(X_{i+n} = y \mid X_i = x)\) can be found from the \(n\)th matrix power \(\mathsf P^n\).

• A stationary distribution \(\pi\) for a Markov chain satisfies the detailed balance equations \(\pi(y)\,p(y,x) = \pi(x)\,p(x,y)\).

• The ergodic theorem says that (under certain technical conditions) \(\frac{1}{n}\sum_{i=1}^n \phi(X_i)\), where \(X_i\) is Markov chain, tends to \(\operatorname{\mathbb E}\phi(X)\), where the PMF of \(X\) is the unique stationary distribution of the Markov chain.

Read more:

• Voss, An Introduction to Statistical Computing, Subsections 2.3.1 and 4.1.2

• my notes for MATH2750 Introduction to Markov Processes, Lectures 7 and 9–11