08. Joint Distribution

$\gdef \N{\mathbb{N}} \gdef \Z{\mathbb{Z}} \gdef \Q{\mathbb{Q}} \gdef \R{\mathbb{R}} \gdef \C{\mathbb{C}} \gdef \setcomp#1{\overline{#1}} \gdef \sseq{\subseteq} \gdef \pset#1{\mathcal{P}(#1)} \gdef \covariant{\operatorname{Cov}} \gdef \of{\circ} \gdef \p{^{\prime}} \gdef \pp{^{\prime\prime}} \gdef \ppp{^{\prime\prime\prime}} \gdef \pn#1{^{\prime\times{#1}}}$

Definition¶

Given 2 Random Variables $X$ and $Y$, their joint distribution is the probability function $P_{X,Y}(k,l)$ for all possible values $k,l$ of $X$ and $Y$ respectively.

Example:
Balls numbered $1,2,3$ are put in a jar. Two are selected without replacement. Let $X=$ number on the first selected ball, $Y=$ number on the second selected ball.
The joint dist. function can be written as a table:

The upper right square represents $P_{X,Y}(1,3)$, and by the table, we can see that it is equal to $\underbrace{\frac13}_{\text{1st ball}}\cdot\underbrace{\frac12}_{\text{2nd ball}}=\frac16$
The sum of all probabilities in a joint distribution function must be 1.
In case that was too easy to understand, here’s the same statement in MathSpeak:

\($\sum_k\sum_lP_{X,Y}(k,l)\ =1$\)
The expected value of $X\cdot Y$ is equal to the sum of a…
Ok I tried, but I can’t write this in English, here’s the algorithm to do it:

1
2
3
4
5
6
7

def expected_value():
    expected=0
    for x_value in possible_values_of_X: # For every row in table
        for y_value in possible_values_of_Y: # For every cell in row
            expected+=x_value*y_value*P(X=x_value,Y=y_value)
            # Where P(X=x_value,Y=y_value) is the value in the cell
    return expected

And here’s the MathSpeak version:
\($E[YX]=\sum_k\sum_lkl\cdot P_{X,Y}(k,l)$\)

Given the joint distribution function, we can find $P_x(k)$ and $P_Y(l)$ for any $k$ or $l$.
In the above example, $P_X(2)=P_{X,Y}(2,1)+P_{X,Y}(2,2)+P_{X,Y}(2,3)=\frac16+0+\frac16=\frac13$

Marginal Probabilities¶

In general:
$P_X(k)=$ the sum of entries in row $k$ in the table.
$P_Y(l)=$ the sum of entries in col $l$ in the table.

These probabilities are called marginal probabilities and are typically written in the margin of the joint dist. table.

The sum of marginal probabilities for any random variable is always 1.

The above table, with added marginal probabilities:

X,Y are independent if and only if their joint distribution table forms a multiplication table, where each cell is equal to the product of the relevant marginal probabilities.
The above table, therefore, is not independent.

If a 0 appears anywhere inside a joint dist. table, then $X,Y$ must be dependent, because marginal probabilities are never zero.