Continuous Bernoulli distribution

Template:Infobox probability distribution 2 In probability theory, statistics, and machine learning, the continuous Bernoulli distribution^[1][2][3] is a family of continuous probability distributions parameterized by a single shape parameter ${\displaystyle \lambda \in (0,1)}$, defined on the unit interval ${\displaystyle x\in [0,1]}$, by:

${\displaystyle p(x|\lambda )\propto {\lambda }^x(1-\lambda {)}^{1-x}.}$

The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders,^[4][5] for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, ${\displaystyle [0,1]}$-valued data.^[6][7][8][9] This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, ${\displaystyle \{0,1\}}$-valued data.

The continuous Bernoulli also defines an exponential family of distributions. Writing ${\displaystyle \theta =\log (\lambda /(1-\lambda ))}$ for the natural parameter, the density can be rewritten in canonical form: ${\displaystyle p(x|\theta )\propto \exp (\theta x)}$. ^[10]

Given an independent sample of ${\displaystyle n}$ points ${\displaystyle x_1,\dots ,x_n}$ with ${\displaystyle x_i\in [0,1]\mathrm{\forall }i}$ from continuous Bernoulli, the log-likelihood of the natural parameter ${\displaystyle \theta }$ is

${\displaystyle {ℒ}(\theta )=\theta {\displaystyle \sum _{i=1}^n}{x}_i-n\log \{(e^{\theta }-1)/\theta \}}$

and the maximum likelihood estimator of the natural parameter ${\displaystyle \theta }$ is the solution of ${\displaystyle {{ℒ}}^{\prime }(\theta )=0}$, that is, ${\displaystyle \hat{\theta }}$ satisfies

${\displaystyle \frac{e^{\hat{\theta }}}{e^{\hat{\theta }}-1}-\frac{1}{\hat{\theta }}=\frac{1}{n}{\displaystyle \sum _{i=1}^n}{x}_i}$

where the left hand side ${\displaystyle e^{\hat{\theta }}/(e^{\hat{\theta }}-1)-{\hat{\theta }}^{-1}}$ is the expected value of continuous Bernoulli with parameter ${\displaystyle \hat{\theta }}$. Although ${\displaystyle \hat{\theta }}$ does not admit a closed-form expression, it can be easily calculated with numerical inversion.

The entropy of a continuous Bernoulli distribution is

${\displaystyle \mathrm{H}[X]=\{\begin{array}{ll}0& \text{ if }\lambda =\frac{1}{2}\\ \frac{\lambda \log \left(\lambda \right)-(1-\lambda )\log (1-\lambda )}{1-2\lambda }-\log \left(\frac{2{\tanh }^{-1}(1-2\lambda )}{e(1-2\lambda )}\right)& \text{ otherwise}\end{array}}$

The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set ${\displaystyle \{0,1\}}$ by the probability mass function:

${\displaystyle p(x)=p^x(1-p{)}^{1-x},}$

where ${\displaystyle p}$ is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval ${\displaystyle [0,1]}$ results in the continuous Bernoulli probability density function, up to a normalizing constant.

The Uniform distribution between the unit interval [0,1] is a special case of continuous Bernoulli when ${\displaystyle \lambda =1/2}$ or ${\displaystyle \theta =0}$.

An exponential distribution with rate ${\displaystyle \mathrm{\Lambda }}$ restricted to the unit interval [0,1] corresponds to a continuous Bernoulli distribution with natural parameter ${\displaystyle \theta =-\mathrm{\Lambda }<0}$.

The multivariate generalization of the continuous Bernoulli is called the continuous-categorical.^[11]

0.00 (0 votes) Original source: https://en.wikipedia.org/wiki/Continuous Bernoulli distribution. Read more