Joint and conditional probability

Suppose that outcome can be either of events A or B (but never both) with probabilities 0.4 and 0.6 correspondingly in case event X happens. If mutually disjoint to X, event Y occurs instead then probabilites of A-B distribute evenly, like .5 and .5. These data can be summarized in a Markov matrix:

X Y A P ( A | X ) P ( A | Y ) B P ( B | X ) P ( B | Y ) = X Y A .4 .5 B .6 .5 {\displaystyle {\begin{matrix}&X&Y\\\A&P(A|X)&P(A|Y)\\\B&P(B|X)&P(B|Y)\end{matrix}}\quad =\quad {\begin{matrix}&X&Y\\\A&.4&.5\\\B&.6&.5\end{matrix}}}

Here, P(A|X) stands for probability of event A provided that X has occurred. A|X generally denotes a conditional probability of event A under condition X.

Note that the sum of columns adds up to 1 since their entries represent mutually exclusive events.

Now, suppose that X can occur with probability .8 and Y has probability of .2. We multiply the first unit/column with .8 and second with .2 so that the joint distribution breaks down into

1 * .8 + 1 * .2 = 1 * (.8 +.2) = 1 * 1 = 1 = 1 * .8 + 1 * .2 = (.4 + .6) * .8 + (.5+.5)/5 = (.32 + .48) + (.1 + .1)

where first parenthesis is a sum of event probabilities under X and .1 + .1 are probabilities under event Y

This can be represented again by matrix again

X | P ( X ) Y | P ( Y ) A P ( A ∧ X ) P ( A ∧ Y ) B P ( B ∧ X ) P ( B ∧ Y ) = X | P ( X ) Y | P ( Y ) A P ( X ∧ A ) P ( Y ∧ A ) B P ( X ∧ B ) P ( Y ∧ B ) = X | .8 Y | .2 A .32 .1 B .48 .1 {\displaystyle {\begin{matrix}&X|_{P(X)}&Y|_{P(Y)}\\\A&P(A\land X)&P(A\land Y)\\\B&P(B\land X)&P(B\land Y)\end{matrix}}\quad =\quad {\begin{matrix}&X|_{P(X)}&Y|_{P(Y)}\\\A&P(X\land A)&P(Y\land A)\\\B&P(X\land B)&P(Y\land B)\end{matrix}}\quad =\quad {\begin{matrix}&X|_{.8}&Y|_{.2}\\\A&.32&.1\\\B&.48&.1\end{matrix}}}

Note that columns now add up to .8 and .2 correpsondingly whereas all table adds up to .8+.2 = 1. We have got a 2-dimensional distribution of probability. In every cell we have the joint probability of pair of events occurring, e.g. P(A∩X) = .32. The probability of conjunction A∩X is less than the probability of components (A|X) and X alone because probability under X, probability of every column, added up to 1 in the conditional probability table but it adds to P(X) ≤ 1 in the joint distribution table.

This fact, that column  P ( A 1 ∧ X i ) + P ( A 2 ∧ X i ) + … = P ( X i ) {\displaystyle P(A_{1}\land X_{i})+P(A_{2}\land X_{i})+\ldots =P(X_{i})} adds up to marginal probability of the column Xi, that is a probability that randomly drawn event ends up in the column i, enables us to recover the conditional probabilities. We just need to divide every  P ( A j ∧ X i ) {\displaystyle P(A_{j}\land X_{i})} in the column i by P(Xi):

[ P ( A | X ) P ( B | X ) ] = [ P ( A ∧ X ) P ( B ∧ X ) ] 1 P ( X ) = [ .32 .48 ] 1 .8 = [ .4 .6 ] {\displaystyle {\begin{bmatrix}P(A|X)\\\P(B|X)\end{bmatrix}}={\begin{bmatrix}P(A\land X)\\\P(B\land X)\end{bmatrix}}{1 \over P(X)}={\begin{bmatrix}.32\\\\.48\end{bmatrix}}{1 \over .8}={\begin{bmatrix}.4\\\\.6\end{bmatrix}}}

The relationship

P ( A ∧ X ) = P ( X ) ⋅ P ( A | X ) {\displaystyle P(A\land X)=P(X)\cdot P(A|X)}

is a basis for famous w:Bayes' theorem P ( X ) ⋅ P ( A | X ) = P ( A ) ⋅ P ( X | A ) {\displaystyle P(X)\cdot P(A|X)=P(A)\cdot P(X|A)} because we can symmetrically condition the probabilities within the rows by probabilities of observing the rows:

[ ( .32 + .1 ) / p a ( .48 + .1 ) / p b ] = [ 1 1 ] = [ ( .32 + .1 ) / .42 ( .48 + .1 ) / .58 ] = [ .76 + .24 .83 + .17 ] = [ P ( X | A ) + P ( Y | A ) P ( X | B ) + P ( Y | B ) ] {\displaystyle {\begin{bmatrix}(.32+.1)/p_{a}\\\\(.48+.1)/p_{b}\end{bmatrix}}={\begin{bmatrix}1\\\1\end{bmatrix}}={\begin{bmatrix}(.32+.1)/.42\\\\(.48+.1)/.58\end{bmatrix}}={\begin{bmatrix}.76+.24\\\\.83+.17\end{bmatrix}}={\begin{bmatrix}P(X|A)+P(Y|A)\\\P(X|B)+P(Y|B)\end{bmatrix}}}

That is, conditional probability P(X|A) = .76.