STA721-F24: Linear Models – Confidence & Credible Regions

Outline

Confidence Interverals from Test Statistics
Pivotal Quantities
Confidence intervals for parameters
Prediction Intervals
Bayesian Credible Regions and Intervals

Readings:

Christensen Appendix C, Chapter 3

Goals

For the regression model $Y = X β + ϵ$ we usually want to do more than just testing that $β$ is zero

what is a plausible range for $β_{j}$ ?
what is a plausible set of values for $β_{j}$ and $β_{k}$ ?
what is a a plausible range of values for $x^{T} β$ for a particular $x$ ?
what is a plausible range of values for $Y_{n + 1}$ for a given value of $x_{n + 1}$ ?

Look at confidence intervals, confidence regions, prediction regions and Bayesian credible regions/intervals

Confidence Sets

For a random variable $Y \sim P \in {P_{θ} : θ \in Θ}$

Definition: Confidence Region

A set valued function

C

is a

(1 - α) \times 100 %

confidence region for

θ

if

P_{θ} ({θ \in C (Y)}) = 1 - α \forall θ \in Θ

In this case we say $C (Y)$ is a $1 - α$ confidence region for the parameter $θ$
there is some true value of $θ$ , and the confidence region will cover it with probability $1 - α$ no matter what it is.
the randomness is due to $Y$ and $C (Y)$
once we observe $Y$ everything is fixed, so region may not include the true $θ$

Hypothesis Tests and Rejection/Acceptance Regions

Recall for a level $α$ test of a point null hypothesis

we reject $H$ with probability $α$ when $H$ is true
for each test we can construct:
- a rejection region $R (θ) \subset Y$ , the $Y$ values for which we reject $H$
- an acceptance region $A (θ) \subset Y$ , the $Y$ values for which we accept $H$
these sets are complements of each other (for non-randomized tests)

$Pr (Y \in A (θ) ∣ θ) = 1 - α$

Duality of Hypothesis-Testing/Confidence Regions

Suppose we have a level $α$ test for every possible valuse of $θ$

for each $θ \in Θ$ , let $A (θ)$ be the acceptance region of the test $Y \sim P_{θ}$
then $P (Y \in A (θ) ∣ θ) = 1 - α$ for each $θ \in Θ$
This collection of hypothesis tests can be “inverted” to construct a confidence region for θ, as follows:
define $C (Y) = {θ \in Θ : Y \in A (θ)}$
this is the set of $θ$ values that are not rejected when $Y = y$ is observed
then $C$ is a $1 - α$ confidence region for $θ$

Confidence Intervals for Regression Parameters

For the linear model $Y \sim N (X β, σ^{2} I)$ , confidence intervals for $β_{j}$ can be constructed from inverting the approriate $t$ -test.

suppose you are testing $H : β_{j} = 0$
if $H$ is true, then
- ${\hat{β}}_{j} - β_{j} \sim N (0, σ^{2} v_{j j})$ where $v_{j j}$ is the $j$ th diagonal element of $(X^{T} X)^{- 1}$
- $s^{2} \sim σ^{2} χ_{n - p}^{2} / (n - p)$
- ${\hat{β}}_{j}$ and $s^{2}$ are independent
therefore if $H$ is true $t_{j} = \frac{{\hat{β}}_{j} - β_{j}}{s \sqrt{v_{j j}}} \sim t_{n - p}$

Acceptance Region & Confidence Interval

define the acceptance region $A (β_{j}) = {{\hat{β}}_{j}, s^{2} : | t_{j} | < t_{n - p, 1 - α / 2}}$
we have that $H$ is accepted if $t_{j} \in A (β_{j}) \Leftrightarrow \frac{| {\hat{β}}_{j} - β_{j} |}{s \sqrt{v_{j j}}} < t_{n - p, 1 - α / 2}$
Now construct a confidence interval for the true value by inverting the tests: $\begin{aligned} C ({\hat{β}}_{j}, s^{2}) & = (\hat{β_{j}}, s^{2}) \in A (β_{j}) \\ = {β_{j} : | {\hat{β}}_{j} - β_{j} | < s \sqrt{v_{j j}} t_{n - p, 1 - α / 2}} \\ = {β_{j} : {\hat{β}}_{j} - s \sqrt{v_{j j}} t_{n - p, 1 - α / 2} < β_{j} < {\hat{β}}_{j} + s \sqrt{v_{j j}} t_{n - p, 1 - α / 2}} \\ = {\hat{β}}_{j} \pm s \sqrt{v_{j j}} t_{n - p, 1 - α / 2} \end{aligned}$
for $α = 0.05$ and large $n$ , $t_{n - p, 0.975} \approx 2$ , so CI is approximately ${\hat{β}}_{j} \pm 2 s \sqrt{v_{j j}}$

Confidence Intervals for Linear Functions

For a linear function of the parameters $λ = a^{T} β$ we can construct a confidence interval by inverting the appropriate $t$ -test

most important example $a^{T} β = x^{T} β = E [Y ∣ x]$
suppose you are testing $H : a^{T} β = m$
If $H$ is true, $a^{T} β - m \sim N (0, σ^{2} v)$ where $v = a^{T} (X^{T} X)^{- 1} a)$
$s^{2} \sim σ^{2} χ_{n - p}^{2} / (n - p)$ independent of $a^{T} \hat{β}$
then $t = \frac{a^{T} \hat{β} - m}{s \sqrt{v}} \sim t_{n - p}$
a $1 - α$ confidence interval for $a^{T} β$ is $a^{T} \hat{β} \pm s \sqrt{v} t_{n - p, 1 - α / 2}$

Prediction Regions and Intervals

Related to CI for $E [Y ∣ x] = x^{T} β$ , we may wish to construct a prediction interval for a new observation $Y^{*}$ at $x_{*}$

a $1 - α$ prediction interval for $Y^{*}$ is a set valued function of $Y$ , $C (Y)$ such that $Pr (Y^{*} \in C (Y) ∣ β, σ^{2}) = 1 - α$ where the distribution is computed using the distribution of $Y^{*}$
this use the idea of a pivotal quantity: a function of the data and the parameters that has a known distribution that does not depend on any unknown parameters.
for prediction, $Y^{*} = x_{*}^{T} β + ϵ^{*}$ where $ϵ^{*} \sim N (0, σ^{2})$ independent of $ϵ$

$\begin{aligned} E [Y^{*} - x_{*}^{T} \hat{β}] & = x_{*}^{T} β - x_{*}^{T} β = 0 \\ Var (Y^{*} - x_{*}^{T} \hat{β}) & = Var (ϵ^{*}) + Var (x_{*}^{T} \hat{β}) = σ^{2} + σ^{2} x_{*}^{T} (X^{T} X)^{- 1} x_{*} \\ Y^{*} - x_{*}^{T} \hat{β} & \sim N (0, σ^{2} (1 + x_{*}^{T} (X^{T} X)^{- 1} x_{*})) \end{aligned}$

Pivotal Quantity and Prediction Intervals

Since $\hat{β}$ and $s^{2}$ are independent, we can construct a pivotal quantity for $Y^{*} - x_{*}^{T} \hat{β}$ : $\frac{Y^{*} - x_{*}^{T} \hat{β}}{s \sqrt{1 + x_{*}^{T} (X^{T} X)^{- 1} x_{*}}} \sim t_{n - p}$

therefore $Pr (\frac{| Y^{*} - x_{*}^{T} \hat{β} |}{s \sqrt{1 + x_{*}^{T} (X^{T} X)^{- 1} x_{*}}} < t_{n - p, 1 - α / 2}) = 1 - α$
Rearranging gives a $1 - α$ prediction interval for $Y^{*}$ : $x_{*}^{T} \hat{β} \pm s \sqrt{1 + x_{*}^{T} (X^{T} X)^{- 1} x_{*}} t_{n - p, 1 - α / 2}$

Joint Confidence Regions for $β$

we can construct a joint confidence region for $β$ based on inverting a test $H : β = β_{0}$ . Recall:

$\begin{aligned} \hat{β} - β & \sim N (0, σ^{2} (X^{T} X)^{- 1}) \\ (X^{T} X)^{- 1 / 2} (\hat{β} - β) & \sim N (0, σ^{2} I) \\ (\hat{β} - β)^{T} (X^{T} X)^{- 1} (\hat{β} - β) & \sim σ^{2} χ_{p}^{2} \end{aligned}$

since $s^{2}$ is independent of $\hat{β}$ we can construct a CI based on the $F$ -distribution $\frac{(\hat{β} - β_{0})^{T} (X^{T} X)^{- 1} (\hat{β} - β_{0}) / p}{s^{2}} \sim F_{p, n - p}$
inverting the $F$ -test gives a $1 - α$ confidence region for $β$ : ${β : (\hat{β} - β_{0})^{T} (X^{T} X)^{- 1} (\hat{β} - β_{0}) / s^{2} < p F_{p, n - p, 1 - α}}$

Bayesian Credible Regions

In a Bayesian setting, we have a posterior distribution for $β$ given the data $Y$
a set $C \in R^{p}$ is a $1 - α$ posterior credible region (sometimes called a Bayesian confidence region) if $Pr (β \in C ∣ Y) = 1 - α$
lots of sets have this property, but we usually want the most probable values of $β$ given the data
this motivates looking at the highest posterior density (HPD) region which is a $1 - α$ credible set $C$ such that the values in $C$ have higher posterior density than those outside of $C$
the HPD region is the smallest region that contains $1 - α$ of the posterior probability

Bayesian Credible Regions

For a normal prior and normal likelihood, the posterior for $β$ conditional on $σ^{2}$ is normal with say posterior mean $b_{n}$ and posterior precision $Φ_{n}$
the posterior density as a function of $β$ for a fixed $σ^{2}$ is $p (β ∣ Y) \propto \exp {- (β - b_{n})^{T} Φ_{n} (β - b_{n}) / 2}$
so a highest posterior density region has the form $C = {β : (β - b_{n})^{T} Φ_{n}^{- 1} (β - b_{n}) < q}$

$\begin{aligned} β - β_{n} ∣ σ^{2} & \sim N (0, Φ_{n}^{- 1}) \\ Φ_{n}^{1 / 2} (β - β_{n}) ∣ σ^{2} & \sim N (0, I) \\ (β - β_{n})^{T} Φ_{n} (β - β_{n}) ∣ σ^{2} & \sim χ_{p}^{2} \end{aligned}$

setting $q = χ_{p, 1 - α}^{2}$ gives a Credible Region for $Pr (β \in C ∣ Y) = 1 - α$

Bayesian HPD Regions For Unknown $σ^{2}$

For unknown $σ^{2}$ we need to integrate out $σ^{2}$ to get the marginal posterior for $β$
for conjugate priors, $β ∣ ϕ \sim N (b_{0}, (ϕ Φ_{0})^{- 1})$ and $ϕ \sim G (a_{0} / 2, b_{0} / 2)$ , then $\begin{aligned} β ∣ ϕ, Y & \sim N (b_{n}, (ϕ Φ_{n})^{- 1}) \\ ϕ ∣ Y & \sim G (a_{n} / 2, b_{n} / 2) \\ β ∣ Y & \sim St (a_{n}, b_{n}, {\hat{σ}}^{2} Φ_{n}^{- 1}) \end{aligned}$ where $St (a_{n}, b_{n}, {\hat{σ}}^{2} Φ_{n}^{- 1})$ is a multivariate Student-t distribution with $a_{n}$ degrees of freedom location $b_{n}$ and scale matrix ${\hat{σ}}^{2} Φ_{n}^{- 1}$ with ${\hat{σ}}^{2} = b_{n} / a_{n}$
density of $β$ is $p (β ∣ Y) \propto {(1 + \frac{(β - b_{n})^{T} Φ_{n} (β - b_{n})}{a_{n} {\hat{σ}}^{2}})}^{- (a_{n} + p) / 2}$

Reference Posterior Distribution

For the reference prior $π (β, ϕ) \propto 1 / ϕ$ and the likelihood $p (Y ∣ β)$ , the posterior is proportional to the likelihood times $ϕ^{- 1}$

(generalized) posterior distribution: $\begin{aligned} β ∣ ϕ, Y & \sim N (\hat{β}, (ϕ X^{T} X)^{- 1}) \\ ϕ ∣ Y & \sim G ((n - p) / 2, SSE / 2) \end{aligned}$ if $n > p$
marginal posterior distribution for $β$ is multivariate Student-t with $n - p$ degrees of freedom, location $\hat{β}$ and scale matrix ${\hat{σ}}^{2} X^{T} X^{- 1}$

Duality

the posterior density $β$ is a monotonically decreasing function of $Q (β) \equiv (β - \hat{β})^{T} X^{T} X (β - \hat{β})$ so contours of $p (β ∣ Y)$ are ellipsoidal in the parameter space of $β$
the quantity $Q (β) / p {\hat{σ}}^{2}$ is distributed a posteriori $Q (β) / p {\hat{σ}}^{2} \sim F (p, n - p)$ and the ellipsoidal contour of $p (β ∣ Y)$ is defined as $\frac{Q (β)}{p {\hat{σ}}^{2}} = F (p, n - p, α)$ . (Box & Tiao 1973)
then HPD regions for $β$ are the same as confidence regions for $β$ based on the $F$ -distribution
marginals of $β_{j}$ , $x^{T} β$ and $Y^{*}$ are also univariate Student-t with $n - p$ degrees of freedom
difference is in the interpretation of the regions i.e posterior probability that $β$ is in the given the data vs the probability a priori that the region covers the true $β$