Skip to content

09. Covariance

$\gdef \N{\mathbb{N}} \gdef \Z{\mathbb{Z}} \gdef \Q{\mathbb{Q}} \gdef \R{\mathbb{R}} \gdef \C{\mathbb{C}} \gdef \setcomp#1{\overline{#1}} \gdef \sseq{\subseteq} \gdef \pset#1{\mathcal{P}(#1)} \gdef \covariant{\operatorname{Cov}} \gdef \of{\circ} \gdef \p{^{\prime}} \gdef \pp{^{\prime\prime}} \gdef \ppp{^{\prime\prime\prime}} \gdef \pn#1{^{\prime\times{#1}}} $

Definition
Intuitive definition

\(\covariant(X,Y)\) measures the likelihood of \(X,Y\) “moving in the same direction”.
If I know \(X\) is less than expected, what does that tell me about \(Y\)?

  1. Nothing; \(Y\) is still expected to be \(E[Y] \Rightarrow \covariant(X,Y)=0\)
  2. \(Y\) is also likely to be less than expected \(\Rightarrow \covariant(X,Y)>0\)
  3. \(Y\) is now likely to be more than expected \(\Rightarrow \covariant(X,Y)<0\)
    • i.e., \(X,Y\) “tend to move” in different directions
Formal definition
\[\covariant(X,Y)=E[(X-\mu_1)\cdot(Y-\mu_2)]\]

Where \(\mu_1=E[X],\mu_2=E[Y]\)
Note that \(V(X)=\covariant(X,X)\)

A simpler formula:

\[\covariant(X,Y)=E[XY]-\mu_1\mu_2\]

Proof: Likely on the ProbabilityExam{: #ProbabilityExam .hash}

\[\begin{aligned} \covariant(X,Y)&=E[(X-\mu_1)\cdot(Y-\mu_2)] \\&=E[XY-\mu_1Y-\mu_2X+\mu_1\mu_2] \\&=E[XY]-\mu_1\underbrace{E[Y]}_{\mu_2}-\cancel{\mu_2\underbrace{E[X]}_{\mu_1}}+\cancel{\mu_1\mu_2} \\&=E[XY]-\mu_1\mu_2 \\Q.E.D \end{aligned}\]

Properties of Covariance

  1. \(\covariant(aX,bY)=ab\cdot\covariant(X,Y)\)
  2. \(\covariant(a+X,b+Y)=\covariant(X,Y)\)
  3. For any \(X,Y\): \(V(X+Y)=V(X)+V(Y)+2\covariant(X,Y)\)

Proof of Property 3: Likely on the ProbabilityExam{: #ProbabilityExam .hash}
\(\(\begin{aligned} V(X+Y)&=E[(X+Y)^2]-(\underbrace{E[X+Y]}_{\begin{aligned}(E[X]&+E[Y])\\(\mu_1&+\mu_2)\end{aligned}})^2 \\&=E[X^2+2XY+Y^2]-(\mu_1^2+2\mu_1\mu_2+\mu_2^2) \\&=E[X^2]+2E[XY]+E[Y^2]-(\mu_1^2+2\mu_1\mu_2+\mu_2^2) \\&=\textcolor{orange}{(E[X^2]-\mu_1^2)}+\textcolor{purple}{(E[Y^2]-\mu_2^2)}+\textcolor{blue}{2(E[XY]-\mu_1\mu_2)} \\&=\textcolor{orange}{V(X)}+\textcolor{purple}{V(Y)}+\textcolor{blue}{2\covariant(X,Y)} \\&Q.E.D. \end{aligned}\)\)

Corollary of Property 3:

\[\begin{aligned} V(X-Y)&=V(X+(-Y))\\ &=V(X)+\underbrace{V(-Y)}_{V(Y)}+2\underbrace{\covariant(X,-Y)}_{-\covariant(X,Y)} \\&=V(X)+V(Y)-2\covariant(X,Y) \end{aligned}\]

Pearson Correlation Coefficient

Definition

\(\rho(X,Y)\) is a normalized version of \(\covariant(X,Y)\).
\(-1\le\rho(X,Y)\le1\) (Sign of \(\rho\) is the same sign of \(\covariant\))

  • If \(\rho(X,Y)\) is close to 0, then the correlation between X,Y is weak
  • If \(\rho(X,Y)\) is close to \(1\), then \(X,Y\) are strongly positively correlated
    • \(X,Y\) have a strong tendency to move in the same direction
  • If \(\rho(X,Y)\) is close to \(-1\), then \(X,Y\) are strongly negatively correlated
    • \(X,Y\) have a strong tendency to move in opposite directions
\[\rho(X,Y)=\frac{\covariant(X,Y)}{\sigma(X)\cdot\sigma(Y)}\]