Proofs of convergence of random variables

This article is supplemental for “Convergence of random variables” and provides proofs for selected results.

Several results will be established using the portmanteau lemma: A sequence {X_n} converges in distribution to X if and only if any of the following conditions are met:

1. [math]\displaystyle{ \mathbb{E}[f(X_n)]\to\mathbb{E}[f(X)] }[/math] for all bounded, continuous functions [math]\displaystyle{ f }[/math];
2. [math]\displaystyle{ \mathbb{E}[f(X_n)]\to\mathbb{E}[f(X)] }[/math] for all bounded, Lipschitz functions [math]\displaystyle{ f }[/math];
3. [math]\displaystyle{ \limsup\operatorname{Pr}(X_n\in C)\leq\operatorname{Pr}(X\in C) }[/math] for all closed sets [math]\displaystyle{ C }[/math];

Convergence almost surely implies convergence in probability

[math]\displaystyle{ X_n\ \overset\mathrm{as}\rightarrow\ X \quad\Rightarrow\quad X_n\ \overset{p}\rightarrow\ X }[/math]

Proof: If [math]\displaystyle{ \{X_n\} }[/math] converges to [math]\displaystyle{ X }[/math] almost surely, it means that the set of points [math]\displaystyle{ O=\{\omega\mid\lim X_n(\omega)\neq X(\omega)\} }[/math] has measure zero. Now fix [math]\displaystyle{ \varepsilon \gt 0 }[/math] and consider a sequence of sets

[math]\displaystyle{ A_n = \bigcup_{m\geq n} \left \{ \left |X_m-X \right |\gt \varepsilon \right\} }[/math]

This sequence of sets is decreasing ([math]\displaystyle{ A_n\supseteq A_{n+1}\supseteq\ldots }[/math]) towards the set

[math]\displaystyle{ A_{\infty} = \bigcap_{n \geq 1} A_n. }[/math]

The probabilities of this sequence are also decreasing, so [math]\displaystyle{ \lim\operatorname{Pr}(A_n)=\operatorname{Pr}(A_\infty) }[/math]; we shall show now that this number is equal to zero. Now for any point [math]\displaystyle{ \omega }[/math] outside of [math]\displaystyle{ O }[/math] we have [math]\displaystyle{ \lim X_n(\omega)=X(\omega) }[/math], which implies that [math]\displaystyle{ \left| X_n(\omega) - X(\omega) \right| \lt \varepsilon }[/math] for all [math]\displaystyle{ n \geq N }[/math] for some [math]\displaystyle{ N }[/math]. In particular, for such [math]\displaystyle{ n }[/math] the point [math]\displaystyle{ \omega }[/math] will not lie in [math]\displaystyle{ A_n }[/math], and hence won't lie in [math]\displaystyle{ A_\infty }[/math]. Therefore, [math]\displaystyle{ A_\infty\subseteq O }[/math] and so [math]\displaystyle{ \operatorname{Pr}(A_\infty)=0 }[/math].

Finally, by continuity from above,

[math]\displaystyle{ \operatorname{Pr}\left(|X_n-X|\gt \varepsilon\right) \leq \operatorname{Pr}(A_n) \ \underset{n\to\infty}{\rightarrow} 0, }[/math]

which by definition means that [math]\displaystyle{ X_n }[/math] converges in probability to [math]\displaystyle{ X }[/math].

Convergence in probability does not imply almost sure convergence in the discrete case

If X_n are independent random variables assuming value one with probability 1/n and zero otherwise, then X_n converges to zero in probability but not almost surely. This can be verified using the Borel–Cantelli lemmas.

Convergence in probability implies convergence in distribution

[math]\displaystyle{ X_n\ \xrightarrow{p}\ X \quad\Rightarrow\quad X_n\ \xrightarrow{d}\ X, }[/math]

Proof for the case of scalar random variables

Lemma. Let X, Y be random variables, let a be a real number and ε > 0. Then

[math]\displaystyle{ \operatorname{Pr}(Y \leq a) \leq \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr}(|Y - X| \gt \varepsilon). }[/math]

Proof of lemma:

[math]\displaystyle{ \begin{align} \operatorname{Pr}(Y\leq a) &= \operatorname{Pr}(Y\leq a,\ X\leq a+\varepsilon) + \operatorname{Pr}(Y\leq a,\ X\gt a+\varepsilon) \\ &\leq \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr}(Y-X\leq a-X,\ a-X\lt -\varepsilon) \\ &\leq \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr}(Y-X\lt -\varepsilon) \\ &\leq \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr}(Y-X\lt -\varepsilon) + \operatorname{Pr}(Y-X\gt \varepsilon)\\ &= \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr}(|Y-X|\gt \varepsilon) \end{align} }[/math]

Shorter proof of the lemma:

We have

[math]\displaystyle{ \begin{align} \{Y \leq a\}\subset\{X \leq a + \varepsilon\}\cup \{|Y-X|\gt \varepsilon\} \end{align} }[/math]

for if [math]\displaystyle{ Y\leq a }[/math] and [math]\displaystyle{ |Y-X|\leq \varepsilon }[/math], then [math]\displaystyle{ X\leq a+\varepsilon }[/math]. Hence by the union bound,

[math]\displaystyle{ \begin{align} \operatorname{Pr}(Y\leq a) \leq \operatorname{Pr}(X \leq a + \varepsilon) + \operatorname{Pr}(|Y-X|\gt \varepsilon). \end{align} }[/math]

Proof of the theorem: Recall that in order to prove convergence in distribution, one must show that the sequence of cumulative distribution functions converges to the F_X at every point where F_X is continuous. Let a be such a point. For every ε > 0, due to the preceding lemma, we have:

[math]\displaystyle{ \begin{align} \operatorname{Pr}(X_n\leq a) &\leq \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr}(|X_n-X|\gt \varepsilon) \\ \operatorname{Pr}(X\leq a-\varepsilon)&\leq \operatorname{Pr}(X_n\leq a) + \operatorname{Pr}(|X_n-X|\gt \varepsilon) \end{align} }[/math]

So, we have

[math]\displaystyle{ \operatorname{Pr}(X\leq a-\varepsilon) - \operatorname{Pr} \left (\left |X_n-X \right |\gt \varepsilon \right ) \leq \operatorname{Pr}(X_n\leq a) \leq \operatorname{Pr}(X\leq a+\varepsilon) + \operatorname{Pr} \left (\left |X_n-X \right |\gt \varepsilon \right ). }[/math]

Taking the limit as n → ∞, we obtain:

[math]\displaystyle{ F_X(a-\varepsilon) \leq \lim_{n\to\infty} \operatorname{Pr}(X_n\leq a) \leq F_X(a+\varepsilon), }[/math]

where F_X(a) = Pr(X ≤ a) is the cumulative distribution function of X. This function is continuous at a by assumption, and therefore both F_X(a−ε) and F_X(a+ε) converge to F_X(a) as ε → 0⁺. Taking this limit, we obtain

[math]\displaystyle{ \lim_{n\to\infty} \operatorname{Pr}(X_n \leq a) = \operatorname{Pr}(X \leq a), }[/math]

which means that {X_n} converges to X in distribution.

Proof for the generic case

The implication follows for when X_n is a random vector by using this property proved later on this page and by taking Y_n = X.

Convergence in distribution to a constant implies convergence in probability

[math]\displaystyle{ X_n\ \xrightarrow{d}\ c \quad\Rightarrow\quad X_n\ \xrightarrow{p}\ c, }[/math] provided c is a constant.

Proof: Fix ε > 0. Let B_ε(c) be the open ball of radius ε around point c, and B_ε(c)^c its complement. Then

[math]\displaystyle{ \operatorname{Pr}\left(|X_n-c|\geq\varepsilon\right) = \operatorname{Pr}\left(X_n\in B_\varepsilon(c)^c\right). }[/math]

By the portmanteau lemma (part C), if X_n converges in distribution to c, then the limsup of the latter probability must be less than or equal to Pr(c ∈ B_ε(c)^c), which is obviously equal to zero. Therefore,

[math]\displaystyle{ \begin{align} \lim_{n\to\infty}\operatorname{Pr}\left( \left |X_n-c \right |\geq\varepsilon\right) &\leq \limsup_{n\to\infty}\operatorname{Pr}\left( \left |X_n-c \right | \geq \varepsilon \right) \\ &= \limsup_{n\to\infty}\operatorname{Pr}\left(X_n\in B_\varepsilon(c)^c\right) \\ &\leq \operatorname{Pr}\left(c\in B_\varepsilon(c)^c\right) = 0 \end{align} }[/math]

which by definition means that X_n converges to c in probability.

Convergence in probability to a sequence converging in distribution implies convergence to the same distribution

[math]\displaystyle{ |Y_n-X_n|\ \xrightarrow{p}\ 0,\ \ X_n\ \xrightarrow{d}\ X\ \quad\Rightarrow\quad Y_n\ \xrightarrow{d}\ X }[/math]

Proof: We will prove this theorem using the portmanteau lemma, part B. As required in that lemma, consider any bounded function f (i.e. |f(x)| ≤ M) which is also Lipschitz:

[math]\displaystyle{ \exists K \gt 0, \forall x,y: \quad |f(x)-f(y)|\leq K|x-y|. }[/math]

Take some ε > 0 and majorize the expression |E[f(Y_n)] − E[f(X_n)]| as

[math]\displaystyle{ \begin{align} \left|\operatorname{E}\left[f(Y_n)\right] - \operatorname{E}\left [f(X_n) \right] \right| &\leq \operatorname{E} \left [\left |f(Y_n) - f(X_n) \right | \right ]\\ &= \operatorname{E}\left[ \left |f(Y_n) - f(X_n) \right |\mathbf{1}_{\left \{|Y_n-X_n|\lt \varepsilon \right \}} \right] + \operatorname{E}\left[ \left |f(Y_n) - f(X_n) \right |\mathbf{1}_{\left \{|Y_n-X_n|\geq\varepsilon \right \}} \right] \\ &\leq \operatorname{E}\left[K \left |Y_n - X_n \right |\mathbf{1}_{\left \{|Y_n-X_n|\lt \varepsilon \right \}}\right] + \operatorname{E}\left[2M\mathbf{1}_{\left \{|Y_n-X_n|\geq\varepsilon \right \}}\right] \\ &\leq K \varepsilon \operatorname{Pr} \left (\left |Y_n-X_n \right |\lt \varepsilon\right) + 2M \operatorname{Pr} \left( \left |Y_n-X_n \right |\geq\varepsilon\right )\\ &\leq K \varepsilon + 2M \operatorname{Pr} \left (\left |Y_n-X_n \right |\geq\varepsilon \right ) \end{align} }[/math]

(here 1_{...} denotes the indicator function; the expectation of the indicator function is equal to the probability of corresponding event). Therefore,

[math]\displaystyle{ \begin{align} \left |\operatorname{E}\left [f(Y_n)\right ] - \operatorname{E}\left [f(X) \right ]\right | &\leq \left|\operatorname{E}\left[ f(Y_n) \right ]-\operatorname{E} \left [f(X_n) \right ] \right| + \left|\operatorname{E}\left [f(X_n) \right ]-\operatorname{E}\left [f(X) \right] \right| \\ &\leq K\varepsilon + 2M \operatorname{Pr}\left (|Y_n-X_n|\geq\varepsilon\right )+ \left |\operatorname{E}\left[ f(X_n) \right]-\operatorname{E} \left [f(X) \right ]\right|. \end{align} }[/math]

If we take the limit in this expression as n → ∞, the second term will go to zero since {Y_n−X_n} converges to zero in probability; and the third term will also converge to zero, by the portmanteau lemma and the fact that X_n converges to X in distribution. Thus

[math]\displaystyle{ \lim_{n\to\infty} \left|\operatorname{E}\left [f(Y_n) \right] - \operatorname{E}\left [f(X) \right ] \right| \leq K\varepsilon. }[/math]

Since ε was arbitrary, we conclude that the limit must in fact be equal to zero, and therefore E[f(Y_n)] → E[f(X)], which again by the portmanteau lemma implies that {Y_n} converges to X in distribution. QED.

Convergence of one sequence in distribution and another to a constant implies joint convergence in distribution

Main page: Slutsky's theorem

[math]\displaystyle{ X_n\ \xrightarrow{d}\ X,\ \ Y_n\ \xrightarrow{p}\ c\ \quad\Rightarrow\quad (X_n,Y_n)\ \xrightarrow{d}\ (X,c) }[/math] provided c is a constant.

Proof: We will prove this statement using the portmanteau lemma, part A.

First we want to show that (X_n, c) converges in distribution to (X, c). By the portmanteau lemma this will be true if we can show that E[f(X_n, c)] → E[f(X, c)] for any bounded continuous function f(x, y). So let f be such arbitrary bounded continuous function. Now consider the function of a single variable g(x) := f(x, c). This will obviously be also bounded and continuous, and therefore by the portmanteau lemma for sequence {X_n} converging in distribution to X, we will have that E[g(X_n)] → E[g(X)]. However the latter expression is equivalent to “E[f(X_n, c)] → E[f(X, c)]”, and therefore we now know that (X_n, c) converges in distribution to (X, c).

Secondly, consider |(X_n, Y_n) − (X_n, c)| = |Y_n − c|. This expression converges in probability to zero because Y_n converges in probability to c. Thus we have demonstrated two facts:

[math]\displaystyle{ \begin{cases} \left| (X_n, Y_n) - (X_n,c) \right|\ \xrightarrow{p}\ 0, \\ (X_n,c)\ \xrightarrow{d}\ (X,c). \end{cases} }[/math]

By the property proved earlier, these two facts imply that (X_n, Y_n) converge in distribution to (X, c).

Convergence of two sequences in probability implies joint convergence in probability

[math]\displaystyle{ X_n\ \xrightarrow{p}\ X,\ \ Y_n\ \xrightarrow{p}\ Y\ \quad\Rightarrow\quad (X_n,Y_n)\ \xrightarrow{p}\ (X,Y) }[/math]

Proof:

[math]\displaystyle{ \begin{align} \operatorname{Pr}\left(\left|(X_n,Y_n)-(X,Y)\right|\geq\varepsilon\right) &\leq \operatorname{Pr}\left(|X_n-X| + |Y_n-Y|\geq\varepsilon\right) \\ &\leq\operatorname{Pr}\left(|X_n-X|\geq\varepsilon/2\right) + \operatorname{Pr}\left(|Y_n-Y|\geq\varepsilon/2\right) \end{align} }[/math]

where the last step follows by the pigeonhole principle and the sub-additivity of the probability measure. Each of the probabilities on the right-hand side converge to zero as n → ∞ by definition of the convergence of {X_n} and {Y_n} in probability to X and Y respectively. Taking the limit we conclude that the left-hand side also converges to zero, and therefore the sequence {(X_n, Y_n)} converges in probability to {(X, Y)}.

See also

• Convergence of random variables

References

0.00 (0 votes) Original source: https://en.wikipedia.org/wiki/Proofs of convergence of random variables. Read more