Jackknife and Jackknife+ Prediction Intervals for Machine Learning: Simulation and Susenas 2023 Household Expenditure
Abstract
This study evaluated and compared jackknife and jackknife+ conformal prediction intervals for three machine learning regression models, namely random forest, gradient boosting, and support vector regression. We used two settings: a simulation that varied the predictor-to-sample-size ratio from 0.2 to 2.0 across ten scenarios, each with 100 replications, using synthetic data drawn from a semiparametric Gaussian copula, and an application to Susenas 2023 household expenditure for Bogor City. We assessed six method-model combinations at 90% nominal coverage using coverage rate, interval width, Winkler score, computation time, and peak memory. In the simulation, median coverage reached 91-92%, but the first quartile fell to 86-88%, and interval width was governed mainly by the base model, not the interval method. On 815 households, coverage ranged from 90.80% to 92.64%, with random forest and jackknife+ closest to nominal. Coverage failures concentrated in the right tail, so these dataset-specific findings require cautious generalization.
Keywords
Full Text:
PDFReferences
L. Breiman, “Random Forest,” Mach. Learn., vol. 45, pp. 1–33, 2001.
J. H. Friedman, “Greedy Function Approximation: A Gradient Boosting Machine,” The Annals of Statistics, vol. 29, pp. 1189–1232, 2001, https://doi.org/10.1214/aos/1013203451.
V. Vapnik, The Nature of Statistical Learning Theory, Second Edition. 1995.
H. Drucker, C. J. C. Surges, L. Kaufman, A. Smola, and V. Vapnik, “Support vector regression machines,” Adv. Neural Inf. Process. Syst., vol. 1, pp. 155–161, 1997.
T. Hastie, R. Tibshirani, and J. Friedman, “Springer Series in Statistics The Elements of Statistical Learning,” The Mathematical Intelligencer, vol. 27, no. 2, pp. 83–85, 2009.
G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning with Applications in R, vol. 7, no. 10. New York: Springer, 2013. https://doi.org/10.1007/978-1-4614-7138-7.
V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World. New York: Springer, 2005.
G. Shafer and V. Vovk, “A tutorial on conformal prediction,” Journal of Machine Learning Research, vol. 9, pp. 371–421, 2008.
H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman, “Inductive Confidence Machines for Regression BT - Machine Learning: ECML 2002,” T. Elomaa, H. Mannila, and H. Toivonen, Eds., Berlin, Heidelberg: Springer Berlin Heidelberg, 2002, pp. 345–356.
J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman, “Distribution-Free Predictive Inference for Regression,” J. Am. Stat. Assoc., vol. 113, no. 523, pp. 1094–1111, 2018, https://doi.org/10.1080/01621459.2017.1307116.
Y. Romano, E. Patterson, and E. J. Candès, “Conformalized quantile regression,” Adv. Neural Inf. Process. Syst., vol. 32, 2019.
C. Kath and F. Ziel, “Conformal prediction interval estimation and applications to day-ahead and intraday power markets,” Int. J. Forecast., vol. 37, no. 2, pp. 777–799, 2021, https://doi.org/10.1016/j.ijforecast.2020.09.006.
A. N. Angelopoulos and S. Bates, “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification,” 2022.
M. Fontana, G. Zeni, and S. Vantini, “Conformal prediction: A unified review of theory and new challenges,” Bernoulli, vol. 29, no. 1, pp. 1–23, 2023, https://doi.org/10.3150/21-BEJ1447.
R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani, “Predictive inference with the jackknife+,” Ann. Stat., vol. 49, no. 1, pp. 486–507, 2021, https://doi.org/10.1214/20-AOS1965.
L. Steinberger and H. Leeb, “Conditional predictive inference for high-dimensional stable algorithms,” pp. 1–75, 2018.
A. N. Angelopoulos, R. F. Barber, and S. Bates, “Theoretical Foundations of Conformal Prediction,” 2026.
V. Manokhin, “Machine Learning for Probabilistic Prediction,” Royal Holloway University of London, 2022.
N. Jean, M. Burke, M. Xie, W. M. Davis, D. B. Lobell, and S. Ermon, “Combining satellite imagery and machine learning to predict poverty,” Science (1979)., vol. 353, no. 6301, pp. 790–794, 2016, https://doi.org/10.1126/science.aaf7894.
D. Filmer and L. H. Pritchett, “Estimating wealth effects without expenditure data—or tears: An application to educational enrollments in states of India,” Demography, vol. 38, no. 1, pp. 115–132, Feb. 2001, https://doi.org/10.1353/dem.2001.0003.
E. L. Aiken, G. Bedoya, J. E. Blumenstock, and A. Coville, “Program targeting with machine learning and mobile phone data: Evidence from an anti-poverty intervention in Afghanistan,” J. Dev. Econ., vol. 161, p. 103016, 2023, https://doi.org/10.1016/j.jdeveco.2022.103016.
A. Usmanova, A. Aziz, D. Rakhmonov, and W. Osamy, “Utilities of Artificial Intelligence in Poverty Prediction: A Review,” Sustainability (Switzerland), vol. 14, no. 21, pp. 1–39, 2022, https://doi.org/10.3390/su142114238.
BPS Kota Bogor, “Kota Bogor dalam Angka 2023,” Kota Bogor, Indonesia, 2023.
Badan Pusat Statistik, Pengeluaran untuk Konsumsi Penduduk Indonesia. Jakarta, 2023. https://doi.org/10.21831/jptk.v23i1.10441.
Badan Pusat Statistik, Statistik Kesejahteraan Rakyat 2023. Jakarta: [BPS] Badan Pusat Statistik, 2023.
A. Deaton and S. Zaidi, “Guidelines for constructing consumption aggregates for welfare analysis,” in World Bank LSMS Working Paper 135, Washington, DC, USA: The International Bank for Reconstruction and Development / The World Bank, 2002.
S. L. Lohr, Design and Analysis. Duxburry Press, 2022.
R. J. A. Little and D. B. Rubin, Statistical Analysis with Missing Data, Third edit. New Delhi, India: Wiley, 2020.
T. de Waal, J. Pannekoek, and S. Scholtus, “Handbook of Statistical Data Editing and Imputation,” 2011.
M. Kuhn and K. Johnson, Applied Predictive Modeling. Groton, Connecticut, USA: Springer, 2013.
C. Cortes and V. Vapnik, “Support Vector Network,” Mach. Learn., vol. 20, pp. 273–297, 1995, https://doi.org/10.1007/BF00994018.
A. J. Smola and B. Schölkopf, “A tutorial on support vector regression,” Stat. Comput., vol. 14, no. 3, pp. 199–222, 2004, https://doi.org/10.1023/B:STCO.0000035301.49549.88.
M. H. Quenouille, “Approximate Tests of Correlation in Time-Series,” Journal of the Royal Statistical Society: Series B (Methodological), vol. 11, no. 1, pp. 68–84, Jan. 1949, https://doi.org/10.1111/j.2517-6161.1949.tb00023.x.
John W. Tukey, “Bias and Confidence in Not-quite Large Samples,” The Annals of Mathematical Statistics, vol. 29, p. 614, 1958.
R. G. Miller, “The Jackknife--A review,” 1974.
B. Efron, “Bootstrap Methods: Another Look at the Jackknife,” The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979.
R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability,” The Annals of Statistics, vol. 51, no. 2, pp. 816–845, 2023, https://doi.org/10.1214/23-AOS2276.
N. Metropolis and S. Ulam, “The Monte Carlo Method,” J. Am. Stat. Assoc., vol. 44, no. 247, pp. 335–341, Sep. 1949, https://doi.org/10.1080/01621459.1949.10483310.
B. Nowok, G. M. Raab, and C. Dibben, “synthpop: Bespoke Creation of Synthetic Data in R,” J. Stat. Softw., vol. 74, no. 11 SE-Articles, pp. 1–26, Oct. 2016, https://doi.org/10.18637/jss.v074.i11.
M. Sklar, “Fonctions de répartition à N dimensions et leurs marges To cite this version : HAL Id : hal-04094463,” hal.scienceM SklarAnnales de l’ISUP, 1959•hal.science, no. 3, pp. 229–231, 1959.
R. B. Nelsen, An Introduction to Copulas. New York: Springer, 2006. https://doi.org/10.1007/0-387-28678-0.
H. Joe, “Dependence modeling with copulas,” Dependence Modeling with Copulas, pp. 1–457, 2014, https://doi.org/10.1201/b17116.
N. J. Higham, “Computing the nearest correlation matrix—a problem from finance,” IMA Journal of Numerical Analysis, vol. 22, no. 3, pp. 329–343, Jul. 2002, https://doi.org/10.1093/imanum/22.3.329.
L. Devroye, “Non-Uniform Random Variate Generation,” Springer-Verlag, New York, 1986, https://doi.org/10.1007/978-1-4613-8643-8.
A. Khosravi, S. Nahavandi, D. Creighton, and A. F. Atiya, “Comprehensive review of neural network-based prediction intervals and new advances.,” IEEE Trans. Neural Netw., vol. 22, no. 9, pp. 1341–1356, Sep. 2011, https://doi.org/10.1109/TNN.2011.2162110.
R. L. Winkler, “A decision-theoretic approach to interval estimation,” J. Am. Stat. Assoc., vol. 67, no. 337, pp. 187–191, 1972, https://doi.org/10.1080/01621459.1972.10481224.
T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” J. Am. Stat. Assoc., vol. 102, no. 477, pp. 359–378, 2007, https://doi.org/10.1198/016214506000001437.
DOI: http://dx.doi.org/10.30829/zero.v10i2.30674
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Editorial office  .jpg) Department of Mathematics Faculty of Science and Technology UIN Sumatera Utara Medan Medan, North Sumatra, Indonesia | For authors | Contact |
Open access under CC BY-SA 4.0 · © 2017–2026 Department of Mathematics, UIN Sumatera Utara Medan