STA721-F24: Linear Models – Optimal Shrinkage/Selection and Oracle Properties

Outline

Bounded Influence and Posterior Mean
Shrinkage properties and nonconcave penalties
conditions for optimal shrinkage and selection . . .
Readings (see reading link)
- Tibshirani (JRSS B 1996)
- Carvalho, Polson & Scott (Biometrika 2010)
- Armagan, Dunson & Lee (Statistica Sinica 2013)
- Fan & Li (JASA 2001)

Horseshoe Priors

Carvalho, Polson & Scott (2010) propose an alternative shrinkage prior $\begin{aligned} β ∣ ϕ & \sim N (0_{p}, \frac{diag (τ^{2})}{ϕ}) \\ τ ∣ λ & \overset{i i d}{\sim} C^{+} (0, λ) \\ λ & \sim C^{+} (0, 1 / ϕ) \\ p (α, ϕ) & \propto 1 / ϕ \end{aligned}$

$C^{+} (0, λ)$ is the half-Cauchy distribution with scale $λ$ $p (τ ∣ λ) = \frac{2}{π} \frac{λ}{λ^{2} + τ_{j}^{2}}$
$C^{+} (0, 1 / ϕ)$ is the half-Cauchy distribution with scale $1 / ϕ$

Special Case: Orthonormal Regression

In the case $λ = ϕ = 1$ and with $X^{t} X = I$ , $Y^{*} = X^{T} Y$ $\begin{aligned} E [β_{i} ∣ Y] & = E_{κ_{i} ∣ Y} [E_{β_{i} ∣ κ_{i}, Y} [β_{i} ∣ Y] \\ = \int_{0}^{1} (1 - κ_{i}) y_{i}^{*} p (κ_{i} ∣ Y) d κ_{i} \\ = (1 - E [κ ∣ y_{i}^{*}]) y_{i}^{*} \end{aligned}$ where $κ_{i} = 1 / (1 + τ_{i}^{2})$ is the shrinkage factor (like in James-Stein)

Half-Cauchy prior induces a Beta(1/2, 1/2) distribution on $κ_{i}$ a priori (change of variables)
marginal prior (after integrating out )

Bounded Influence ( $X^{T} X = I$ )

Posterior mean of $β_{i}$ may be written as $E [β_{i} ∣ y_{i}^{*}] = y_{i}^{*} + \frac{d}{d y} \log m (y_{i}^{*})$ where $m (y)$ is the predictive density $y_{i}^{*}$ under the prior (known $λ$ )
Bounded Influence of the prior (in this setting) means that $lim_{| y | \to \infty} \frac{d}{d y} \log m (y) = c$
For HS $lim_{| y | \to \infty} \frac{d}{d y} \log m (y) = 0$
$lim_{| y_{i}^{*} | \to \infty} E [β_{i} ∣ y_{i}^{*}) \to y_{i}^{*}$ (the MLE)
unbiasedness for large $| y_{i}^{*} |$

DE has bounded influence, but bound does not decay to zero in tails so the posterior mean does not shrink to the MLE (bounded away)

Comparison

Diabetes data (from the lars package)
64 predictors: 10 main effects, 2-way interactions and quadratic terms

sample size of 442
split into training and test sets
compare MSE for out-of-sample prediction using OLS, lasso and horseshoe priors
Root MSE for prediction for left out data based on 25 different random splits with 100 test cases
both Lasso and Horseshoe much better than OLS

Duality for Modal Estimators

Model $Y = X β + ϵ$ with $X^{T} X = I_{p}$ and $\hat{β} = X^{T} Y \equiv Y^{*}$ (take $σ^{2} = 1$ )
Penalized Least Squares ${\hat{β}}_{j}^{λ} = \underset{β}{argmin} \frac{1}{2} ‖ Y - X \hat{β} ‖^{2} + \frac{1}{2} \sum_{j} (β_{j} - {\hat{β}}_{j})^{2} + \sum_{j} {pen}_{λ} (β_{j})$
Bayes posterior mode (conditional) with prior $p (β ∣ λ) = \prod_{j} p (β_{j} ∣ λ)$ $\begin{aligned} {\hat{β}}_{j}^{λ} & = \underset{β}{argmax} - \frac{1}{2} ‖ Y - X \hat{β} ‖^{2} - \frac{1}{2} \sum_{j} (β_{j} - {\hat{β}}_{j})^{2} + \sum_{j} \log (p (β_{j} ∣ λ)) \\ = \underset{β}{argmin} \frac{1}{2} ‖ Y - X \hat{β} ‖^{2} + \frac{1}{2} \sum_{j} (β_{j} - {\hat{β}}_{j})^{2} - \sum_{j} \log (p (β_{j} ∣ λ)) \\ = \underset{β}{argmin} \frac{1}{2} ‖ Y - X \hat{β} ‖^{2} + \frac{1}{2} \sum_{j} (β_{j} - {\hat{β}}_{j})^{2} + \sum_{j} {pen}_{λ} (β_{j}) \end{aligned}$

Properties for Modal Estimates

Fan & Li (JASA 2001) discuss variable selection via nonconcave penalties and oracle properties in the context of penalized likelihoods in this setting

with duality of the negative log prior as their penalty we can extend to Bayesian modal estimates where the prior is a function of $| β_{j} |$ $\frac{1}{2} \sum (β_{i} - y_{i}^{*})^{2} + \frac{1}{2} \sum_{j} (β_{j} - {\hat{β}}_{j})^{2} + \sum_{j} {pen}_{λ} (| β_{j} |)$
Requirements on penality
- Unbiasedness: The resulting estimator is nearly unbiased when the true unknown parameter is large (avoid unnecessary modeling bias).
- Sparsity: thresholding rule sets small coefficients to 0 (avoid model complexity)
- Continuity: continuous in the data ${\hat{β}}_{j} = y_{i}^{*}$ (avoid instability in model prediction)

Conditions for Unbiasedness

To find the optimal estimator take derivative of $\frac{1}{2} \sum_{j} (β_{j} - {\hat{β}}_{j})^{2} + \sum_{j} {pen}_{λ} (| β_{j} |)$ componentwise and set to zero

Derivative is $\begin{aligned} \frac{d}{d β_{j}} {\frac{1}{2} (β_{j} - {\hat{β}}_{j})^{2} + {pen}_{λ} (| β_{j} |)} & = (β_{j} - {\hat{β}}_{j}) + sgn (β_{j}) {pen}_{λ}^{'} (| β_{j} |) \\ = sgn (β_{j}) {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)} - {\hat{β}}_{j} \end{aligned}$
setting derivative to zero gives ${\hat{β}}_{j} = sgn (β_{j}) {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)}$
if $lim_{| β_{j} | \to \infty} {pen}_{λ}^{'} (| β |) = 0$ then ${\hat{β}}_{j} = sgn (β_{j}) | β_{j} | = β_{j}$
for large $| β_{j} |$ , $| {\hat{β}}_{j} |$ is large with high probability
as MLE is unbiased, the optimal estimator is approximately unbiased for large $| β_{j} |$

Conditions for Thresholding & Continuity

As sufficient condition for a thresholding rule ${\hat{β}}_{j}^{λ} = 0$ is if $0 < min {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)}$

if $| {\hat{β}}_{j} | < min {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)}$ then the derivative is positive for all positive $β_{j}$ and negative for all negative $β_{j}$ so ${\hat{β}}_{j}^{λ} = 0$ is a local minimum
if $| {\hat{β}}_{j} | > min {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)}$ multiple crossings (local roots)
a sufficient and necessary condition for continuity is that the minimum of $| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)$ is obtained at zero

Example: Gaussian Prior

Prior $N (0, 1 / λ^{2})$
Penalty: ${pen}_{λ} (| β_{j} |) = \frac{1}{2} λ | β_{j} |^{2}$
Unbiasedness: for large $| β_{j} |$ ?
- Derivative of ${pen}_{λ} (| β_{j} |) = λ β_{j} = sgn (β_{j}) λ | β_{j} |$
- does not go to zero as $| β_{j} | \to \infty$
- No! (bias towards zero)
not a thresholding rule as $min {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)} = (1 + λ) | β_{j} |$ is zero
is continuous as minimum is at zero

Example: Lasso Prior

Penalty: ${pen}_{λ} (| β_{j} |) = λ | β_{j} |$
Unbiasedness: for large $| β_{j} |$ ?
- Derivative of ${pen}_{λ} (| β_{j} |) = λ sgn (β_{j})$
- does not go to zero as $| β_{j} | \to \infty$
- No! (bias towards zero)
Is a thresholding rule as $min {| β_{j} | + {pen}_{λ}^{'} (| β_{j} |)} = (| β_{j} | + λ) > 0$
is continuous as minimum is at $β_{j} = 0$

Generalized Double Pareto Prior

The Generalized Double Pareto of Armagan, Dunson & Lee (2013)
has a prior density for $β_{j}$ of the form $p (β_{j} ∣ ξ, α) = \frac{1}{2 ξ} {(1 + \frac{β_{j}}{α ξ})}^{- (1 + α)}$

express as $β_{j} ∣ ξ, α \sim GDP (ξ, α)$
Scale mixtures of Normals representation $\begin{aligned} β ∣ τ_{j} & \sim N (0, τ_{j}) \\ τ_{j} ∣ λ_{j} & \sim Exp (λ_{j}^{2} / 2) \\ λ_{j} & \sim Gamma (α, η) \\ β_{j} & \sim GDP (ξ = η / α, α) \end{aligned}$
is this a thresholding rule? unbiasedness? continuity?
for all parameters or are there restrictions?

Choice of Penalty/Prior and Conditions

Ridge: none
Lasso: does not satisfy conditions for unbiasedness
GDP: Can show that Generalized Double Pareto does for some choices of hyperparameters
Horseshoe: need marginal distribution of $β_{j}$ for penalty
- marginal generally not available in closed form
- can show for a special case where there is an analytic expression for the marginal density ( $λ = ϕ = 1$ ) $p (β) = k \exp (β^{2} / 2) E_{1} (β^{2} / 2)$
where $E_{n} (x) = \int_{1}^{\infty} \frac{e^{- x t}}{t^{n}} d t$ for $n = 1, 2, \dots$
$E_{n}^{'} (x) = - E_{n - 1} (x)$ for $n = 1, 2, \dots$

Shrinkage Estimators

The literature on shrinkage estimators (with or without selection) is extensive

Ridge
Lasso
Elastic Net (Zou & Hastie 2005)
SCAD (Fan & Li 2001)
Generalized Double Pareto Prior (Armagan, Dunson & Lee 2013)
Spike-and-Slab Lasso (Rockova & George 2018)

For Bayes, choice of estimator

posterior mean (easy via MCMC)
posterior mode (optimization)
posterior median (via MCMC)

Selection and Uncertainty

Prior/Posterior do not put any probability on the event $β_{j} = 0$
Uncertainty that the coefficient is zero?
Selection solved as a post-analysis decision problem
Selection part of model uncertainty
- add prior probability that $β_{j} = 0$
- combine with decision problem