Information, amount of

An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let $\xi$ and $\eta$ be random variables defined on a probability space $(\mathrm{\Omega },\mathfrak{A},\mathsf{P})$ and taking values in measurable spaces (cf. Measurable space) $(\mathfrak{X},S_{\mathfrak{X}})$ and $(\mathfrak{Y},S_{\mathfrak{Y}})$, respectively. Let $p_{\xi \eta }(C)$, $C\in S_{\mathfrak{X}}\times S_{\mathfrak{Y}}$, and $p_{\xi }(A)$, $A\in S_{\mathfrak{X}}$, $p_{\eta }(B)$, $B\in S_{\mathfrak{Y}}$, be their joint and marginale probability distributions. If $p_{\xi \eta }(\cdot )$ is absolutely continuous with respect to the direct product of measures $p_{\xi }\times p_{\eta }(\cdot )$, if $a_{\xi \eta }(\cdot )$ is the (Radon–Nikodým) density of $p_{\xi \eta }(\cdot )$ with respect to $p_{\xi }\times p_{\eta }(\cdot )$, and if $i_{\xi \eta }(\cdot )=\log a_{\xi \eta }(\cdot )$ is the information density (the logarithms are usually taken to base 2 or $e$), then, by definition, the amount of information is given by

$$I(\xi ,\eta )=\underset{\mathfrak{X}\times \mathfrak{Y}}{\int }{i}_{\xi \eta }(x,y)p_{\xi \eta }(dx,dy)=$$

$$=\underset{\mathfrak{X}\times \mathfrak{Y}}{\int }{a}_{\xi \eta }(x,y)\log a_{\xi \eta }(x,y)p_{\xi }(dx)p_{\eta }(dy).$$

If $p_{\xi \eta }(\cdot )$ is not absolutely continuous with respect to $p_{\xi }\times p_{\eta }(\cdot )$, then $I(\xi ,\eta )=+\mathrm{\infty }$, by definition.

In case the random variables $\xi$ and $\eta$ take only a finite number of values, the expression for $I(\xi ,\eta )$ takes the form

$$I(\xi ,\eta )=\sum _{i=1}^n\sum _{j=1}^m{p}_{ij}\log \frac{p_{ij}}{p_i{q}_i},$$

where

$$\{p_i{\}}_{i=}{1}^n,\{q_j{\}}_{j=}{1}^m,\{p_{ij}:i=1\dots n;j=1\dots m\}$$

are the probability functions of $\xi$, $\eta$ and the pair $(\xi ,\eta )$, respectively. (In particular,

$$I(\xi ,\xi )=-\sum _{i=1}^n{p}_i\log p_i=H(\xi )$$

is the entropy of $\xi$.) In case $\xi$ and $\eta$ are random vectors and the densities $p_{\xi }(x)$, $p_{\eta }(y)$ and $p_{\xi \eta }(x,y)$ of $\xi$, $\eta$ and the pair $(\xi ,\eta )$, respectively, exist, one has

$$I(\xi ,\eta )=\int p_{\xi \eta }(x,y)\log \frac{p_{\xi \eta }(x,y)}{p_{\xi }(x)p_{\eta }(y)}dxdy.$$

In general,

$$I(\xi ,\eta )=supI(\varphi (\xi ),\psi (\eta )),$$

where the supremum is over all measurable functions $\varphi (\cdot )$ and $\psi (\cdot )$ with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission.

For references, see , ,

to Information, transmission of.