A method, based on approximate message passing (AMP), that for \(\delta
= 0\) recovers estimation and inferential performance by appropriately rescaling \(\hat\beta\), whenever that exists.
Empirical evidence that this continues to hold in high dimensions
\[\LARGE\Downarrow\]
\[\beta_{P} = 0\]
fou + kar VS fou using MDYPL
Maximum Diaconis-Ylvisaker prior penalized likelihood (MDYPL)
Adjust responses to \(\displaystyle y_i^* = \alpha y_i + \frac{1 - \alpha}{2}\) with \(\alpha \in [0, 1)\)1
Use ML estimation with the adjusted responses
full_mdypl <-glm(y ~ ., data = full_train_adj, family = binomial)nest_mdypl <-glm(y ~ ., data = nest_train_adj, family = binomial)
Penalized likelihood ratio test
Df
PLR
64
64.36
fou + kar VS fou using MDYPL
Simulation setup
\(1000 \times 140\) matrix \(X_{\rm fou + kar}\) is fixed
\(\beta_{\rm fou + kar} = (\beta_{\rm fou}^\top, 0_{64}^\top)^\top\)
\(\beta_{\rm fou}\) from i.i.d \(N(0, 1)\), rescaled so that \[
\widehat{\rm var}(X_{\rm fou + kar} \beta_{\rm fou + kar}) = \gamma^2
\]
Intercept: \(\delta \in \{-3, -2, -1, 0\}\)
Signal strength: \(\gamma^2 \in \{1, 2, 4, 8, 16\}\)
\(500\) response vectors per \((\delta, \gamma^2)\)
MDYPL: estimation and inference
Aggregate asymptotic behaviour of MDYPL estimator
Assume that \((\alpha,\kappa,\gamma)\) are such that \(\| \hat{\beta}^{\textrm{\small DY}} \|_2 = \mathcal{O}(n^{1/2})\) almost surely. Then for any function \(\psi: \Re^2 \to \Re\) that is pseudo-Lipschitz of order 2, \[
\frac{1}{p} \sum_{j=1}^{p} \psi(\hat{\beta}^{\textrm{\small DY}}_j - {\mu}_{\star}\beta_{0,j}, \beta_{0,j}) \overset{\textrm{a.s.}}{\longrightarrow} \mathop{\mathrm{E}}\left[\psi({\sigma}_{\star}G, \bar{\beta})\right], \quad \textrm{as } n \to \infty
\] where \(G \sim \mathop{\mathrm{N}}(0,1)\) is independent of \(\bar{\beta} \sim \pi_{\bar{\beta}}\)
Amount of shrinkage (value of \(\alpha\)) that results in \(\hat{\beta}^{\textrm{\small DY}}/ {\mu}_{\star}\) being asymptotically unbiased
Optimal shrinkage: Minimum MSE
Amount of shrinkage (value of \(\alpha\)) that results in \(\hat{\beta}^{\textrm{\small DY}}/ {\mu}_{\star}\) having minimal asymptotic MSE
DY prior penalized likelihood ratio test statistics
Assume that \((\alpha,\kappa,\gamma)\) are such that \(\| \hat{\beta}^{\textrm{\small DY}} \|_2 = \mathcal{O}(n^{1/2})\) almost surely. Define the DY prior penalized likelihood ratio test statistic for \(H_0: \beta_{0,i_1} = \ldots = \beta_{0,i_k} = 0\) as \[
\Lambda_I = \underset{\beta \in \Re^p}{\max} \, \ell(\beta \,;\,y^*, X) - \underset{\substack{\beta \in \Re^p: \\ \beta_j = 0, \, j \in I }}{\max} \, \ell(\beta \,;\,y^*, X)\,, \quad I = \{i_1,\ldots,i_k\}
\]
Classical theory implies \(Z \stackrel{d}{\rightarrow} \mathop{\mathrm{N}}(0, 1)\) when \(\beta_{k} = 0\)
Estimating unknowns
Dimension
\(\hat\kappa = p / n\)
Signal strength
Adaptation of the Signal Strength Leave-One-Out Estimator (SLOE) of Yadlowsky et al. (2021), that consistently estimates \(\upsilon^2 = \lim\limits_{n \to \infty}\mathop{\mathrm{var}}(x_j^\top \hat{\beta}^{\textrm{\small DY}}) = {\mu}_{\star}^2 \gamma^2 +
\kappa {\sigma}_{\star}^2\)
\[
\hat \upsilon^2 = \frac{\sum_{i = 1}^n (s_i - \bar{s})^2}{n} \quad \text{with} \quad s_i = \hat{\eta}^{\textrm{\small DY}}_i - \frac{\hat{h}^{\textrm{\small DY}}_i}{1 - \hat{h}^{\textrm{\small DY}}_i} (\hat{z}^{\textrm{\small DY}}_i - \hat{\eta}^{\textrm{\small DY}}_i)
\] where \(\hat{h}^{\textrm{\small DY}}_i\) is the hat value for the \(i\)th observation
Estimating unknowns
Estimates of \({\mu}_{\star}\), \(b_{\star}\), \({\sigma}_{\star}\)
Replace \(\gamma\) in the state evolution equations (1) with \(\hat\gamma = (\hat\upsilon^2 - \hat\kappa {\sigma}_{\star}^2) / {\mu}_{\star}^2\)
Solve (1) to get estimates for \({\mu}_{\star}\), \(b_{\star}\), \({\sigma}_{\star}\)
Conditional variance of covariates
For adjusted Wald tests, \(\tau_j^2 = \mathop{\mathrm{var}}(x_{ij} \mid x_{i, -j})\) can be estimated using the residual sums of squares from the regressions of each covariate on all the others
Suppose that \(x_{i} \stackrel{iid}{\sim} \mathop{\mathrm{N}}(0, \Sigma)\), and that \(\mathop{\mathrm{var}}(x_i^\top \beta_0) \to \gamma_0^2\), as \(p / n \to \kappa \in
(0, 1)\).
Then, the mJPL estimator \((\tilde\delta, \tilde\beta^\top)^\top\) satisfies \[
\frac{1}{p} \sum_{j = 1}^p (\tilde\beta_j - \alpha_\star \beta_{0,j}) \overset{\textrm{a.s.}}{\longrightarrow} 0
\] with \(\alpha_\star \le 1\). For small to moderate values of \(\delta_0^2 / \gamma_0^2\), the scaling \(\alpha_\star\) can be approximated by \[
q(\kappa, \gamma, \gamma_0; b) = \left\{
\begin{array}{ll}
1 \,, & \text{if } \kappa < h_{\rm MLE}(\delta, \gamma_0) \\
\kappa^{b_1} \gamma^{b_2} \gamma_0^{b_3}\,, & \text{if } \kappa > h_{\rm MLE}(\delta, \gamma_0)
\end{array}
\right.
\] where \(b_1 < -1\), \(b_2 < -1\), \(b_3 > 0\).
Predictions for \(b_1\), \(b_2\), \(b_3\) are -1.172, -1.869, 0.817, respectively (Kosmidis & Zietkiewicz, 2025)
Sterzinger P, Kosmidis I (2024). Diaconis-Ylvisaker prior penalized likelihood for \(p / n \to \kappa \in (0, 1)\) logistic regression. arXiv: 2311.11290
References
Candès, E. J., & Sur, P. (2020). The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. Annals of Statistics, 48(1), 27–42. https://doi.org/10.1214/18-AOS1789
Cordeiro, G. M., & McCullagh, P. (1991). Bias correction in generalized linear models. Journal of the Royal Statistical Society, Series B: Methodological, 53(3), 629–643. https://doi.org/10.1111/j.2517-6161.1991.tb01852.x
Kosmidis, I., & Zietkiewicz, P. (2025). Jeffreys-prior penalty for high-dimensional logistic regression: A conjecture about aggregate bias. Statistical Science (to Appear). https://arxiv.org/abs/2311.11290
Lewis, R. M., & Battey, H. S. (2024). On inference in high-dimensional logistic regression models with separated data. Biometrika, 111(3), 989–1011. https://doi.org/10.1093/biomet/asad065
Rigon, T., & Aliverti, E. (2023). Conjugate priors and bias reduction for logistic regression models. Statistics & Probability Letters, 202, 109901. https://doi.org/10.1016/j.spl.2023.109901
Sur, P., & Candès, E. J. (2019). A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences, 116(29), 14516–14525. https://doi.org/10.1073/pnas.1810420116
Zhao, Q., Sur, P., & Candès, E. J. (2022). The asymptotic distribution of the MLE in high-dimensional logistic models: Arbitrary covariance. Bernoulli, 28(3). https://doi.org/10.3150/21-BEJ1401
