Bayesian Estimation and Shrinkage

We study Bayes estimators of the regression coefficients, and show how these can be viewed as shrunken versions of the OLS estimator. We discuss two commonly used priors, the \(g\)-prior of Zellner (1986) and an independent prior. In the latter case, the posterior mean is equivalent to the classic ridge regression estimator of Hoerl and Kennedy. The posterior mean in both cases may offer improvements over OLS. Rigde regression in particular can reduce variance and estimation error in cases where the predictors are highly collinear.

Readings:

• Christensen Chapter 2.9 and Chapter 15

• Seber & Lee Chapter 3.12 and Chapter 12

• Hoerl, A.E. and Kennard, R.W. (1970) Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67

• Zellner, A. (1986) On assessing prior distributions and Bayesian regression analysis with \(g\)-prior distributions. In Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti, eds. P. K. Goel and A. Zellner, 233–243. Amsterdam: North-Holland