# Where have all the covariances gone?

Have you wondered ‘Where do the covariances of the predictor variables go in regression?’ or ‘How does regression analysis produce the unique effects of each predictor on the outcome variable?’ Awesome. You’re a normal psychologist asking a really important question.

The simple answer is that the covariances among predictors are accounted for during parameter estimation (which is just a fancy way of saying ‘It just does’, and you can get away with that answer as a psychological researcher). Read on to learn more about this.

Consider a regression with two predictors (${X_{1i}, X_{2i}}$) and an outcome variable (${Y_i}$), such that

Equation 1: $Y_{i} = b_0 + b_1X_{1i} + b_2X_{2i} + \epsilon_{i}$

Within the Equation 1, $b_0$ is the expected value of $Y_{i}$ when $X_{1i}$ and $X_{2i}$ equal zero, $b_1$ represents the unique effect of $X_{1i}$ on $Y_{i}$, and $b_2$ indicates the unique effect of $X_{2i}$ on $Y_{i}$.

Therefore, when fitting the model in Equation 1, how does regression analysis account for the covariance between $X_{1i}$ and $X_{2i}$?

I’m going to simplify this blog post by only considering z-score versions of each of the variables. Let

• ${z_{X_{1i}}} = \frac{X_{1i} - \bar{X_{1}}}{s_{X_1}}$
• ${z_{X_{2i}}} =\frac{X_{2i} - \bar{X_{2}}}{s_{X_2}}$
• ${z_{Y_i}} =\frac{Y_{i} - \bar{Y}}{s_{Y}}$

and substitute into Equation 1, then we have

Equation 2:  ${z_{Y_i}} = b_1{z_{X_{1i}}} +b_2{z_{X_{2i}}} + \epsilon_{i}$

Equation 2 does not include an intercept; when the predictors and outcome variable are  z-scores, the intercept equals 0.

Regression analysis selects the estimates for $b_1$ and $b_2$ by minimizing the residual variance (i.e.,  $\sigma^2_{\epsilon}$ ). In particular, regression analysis chooses  the $b_1$ and $b_2$ that minimizes $\sigma^2_{\epsilon}$.

Equation 3: $\sigma^2_{\epsilon} = \frac{1}{N-1}\sum_{i=1}^{N}\epsilon_{i}^2$.

We can rewrite the residual variance of Equation 3 in terms of the regression equation from Equation 2 as

Equation 4: $\frac{1}{N-1}\sum_{i=1}^{N}\epsilon_{i}^2 =\frac{1}{N-1}\sum_{i=1}^{N}(z_{Y_i} - b_1{z_{X_{1i}}} - b_2{z_{X_{2i}}})^2$

Let’s understand this equation a bit further. First, let’s square the summand on the right-hand side of Equation 4.

Equation 5: $\frac{1}{N-1}\sum_{i=1}^{N}\epsilon_{i}^2 = \frac{1}{N-1}\sum_{i=1}^{N}z_{Y_i}^2 + b_1^{2}z_{x_{1i}}^2 + b_2^{2}z_{x_{2i}}^2 - 2b_1z_{Y_i}z_{x_{1i}} -2b_2z_{Y_i}z_{x_{2i}} + 2b_1b_2z_{x_{1i}}z_{x_{2i}}$

Distributing the summation operator in Equation 5 produces

Equation 6:$\frac{1}{N-1}\sum_{i=1}^{N}\epsilon_{i}^2 = \frac{1}{N-1}\sum_{i=1}^{N}y_{i}^2+\frac{1}{N-1}\sum_{i=1}^{N}b_1^2x_{1i}^2 + \frac{1}{N-1}\sum_{i=1}^{N}b_2^2x_{2i}^2 -\frac{1}{N-1}\sum_{i=1}^{N}2b_1y_ix_{1i} -\frac{1}{N-1}\sum_{i=1}^{N}2b_2y_ix_{2i} +\frac{1}{N-1}\sum_{i=1}^{N}2b_1b_2x_{1i}x_{2i}$

We can extract out each of the regression coefficients from the summands on the right hand size of equation 6  to obtain

Equation 7:$\frac{1}{N-1}\sum_{i=1}^{N}\epsilon_{i}^2 = \frac{1}{N-1}\sum_{i=1}^{N}y_{i}^2+b_1^2\frac{1}{N-1}\sum_{i=1}^{N}x_{1i}^2 +b_2^2\frac{1}{N-1}\sum_{i=1}^{N}x_{2i}^2 -2b_1\frac{1}{N-1}\sum_{i=1}^{N}y_ix_{1i} -2b_2\frac{1}{N-1}\sum_{i=1}^{N}y_ix_{2i} +2b_1b_2\frac{1}{N-1}\sum_{i=1}^{N}x_{1i}x_{2i}$

Many of the terms within Equation 7 can be simplified. We know that

• $\frac{1}{N-1}\sum_{i=1}^{N}\epsilon_{i}^2 = \sigma^2_{\epsilon} = 1$
• $\frac{1}{N-1}\sum_{i=1}^{N}y_{i}^2 = \sigma^2_Y = 1$
• $\frac{1}{N-1}\sum_{i=1}^{N}x_{1i}^2 = \sigma^2_{X_1} = 1$
• $\frac{1}{N-1}\sum_{i=1}^{N}x_{2i}^2 = \sigma^2_{z_{X_2}} = 1$
• $\frac{1}{N-1}\sum_{i=1}^{N}y_ix_{1i}= \sigma_{Y,X_1}$
• $\frac{1}{N-1}\sum_{i=1}^{N}y_ix_{2i}=\sigma_{Y,X_2}$
• $\frac{1}{N-1}\sum_{i=1}^{N}x_{1i}x_{2i}= \sigma_{X_1,X_2}$

Therefore, we can simplify Equation 7 to

Equation 7: $latex\sigma^2_{\epsilon} = \sigma^2_Y +b_1^2(\sigma^2_{X_1}) +b_2^2(\sigma^2_{X_2}) -2b_1\sigma_{Y,X_1}-2b_2\sigma_{Y,X_2} +2b_1b_2\sigma_{X_1,X_2}&bg=ffffff$

Thus, by squaring all of the terms within the regression model (i.e., Equations 4-5), the covariances are among the predictors variables (e.g., $\sigma_{X_1,X_2}$) are directly accounted for.

Note that this is all way, way clearer within an SEM framework****.

*Ok, it was just Simine.

**Mean-centering. No interactions. Mind. Blown.

***So far I’ve just been answering the fake question.

****I smell another blog post….

Posted in Uncategorized