Twelve practical recommendations for developing and applying clinical predictive models

Guoshuang Feng; Huiyu Xu; Shibiao Wan; Haitao Wang; Xiaofei Chen; Robert Magari; Yong Han; Yongyue Wei; Hongqiu Gu

doi:10.59717/j.xinn-med.2024.100105

The Innovation Medicine > 2024 Vol. 2 > No. 4 > 100105

Next Article Previous Article

ARTICLE Open Access

Cite PDF

Twelve practical recommendations for developing and applying clinical predictive models

1.
Big Data Center, Beijing Children’s Hospital, Capital Medical University, National Center for Children's Health, Beijing 100045, China
2.
State Key Laboratory of Female Fertility Promotion, Center for Reproductive Medicine, Department of Obstetrics and Gynecology, Peking University Third Hospital, Beijing 100191, China
3.
Department of Genetics, Cell Biology and Anatomy, University of Nebraska Medical Center, Omaha, NE 68198, United States
4.
Thoracic Epigenetics Section, Thoracic Surgery Branch, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda, MD 20892, United States
5.
Statistical Science, Southern Methodist University, Dallas, TX 75205, United States
6.
Department of R & D, Beckman Coulter, Brea, CA 33196, United States
7.
Key Laboratory of Tumor Molecular Diagnosis and Individualized Medicine of Zhejiang Province, Zhejiang Provincial People's Hospital, Affiliated People's Hospital, Hangzhou Medical College, Hangzhou 310014, China
8.
Center for Public Health and Epidemic Preparedness & Response, School of Public Health, Peking University, Key Laboratory of Epidemiology of Major Diseases (Peking University), Ministry of Education, Beijing 100191, China
9.
China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing 100050, China

Show all affliationsShow less

More Information

Corresponding author: glxfgsh@163.com
Received Date: November 06, 2024

Accepted Date: December 02, 2024

Published Online: December 06, 2024

The Innovation Medicine 2(4), https://doi.org/10.59717/j.xinn-med.2024.100105 | Cite this article

GRAPHICAL ABSTRACT

GRAPHICAL ABSTRACT

DownLoad: Full size image

PUBLIC SUMMARY

PUBLIC SUMMARY
1. This article proposes 12 key recommendations for the application of medical prediction models in clinical practice.
  
  Developing prediction models isn't just software output; it requires avoiding pitfalls at multiple steps.
  
  Publication alone can't ensure prediction model use; clinical value and physician acceptance are essential.
  
  Promoting prediction models requires internal and external validation, impact evaluation, and expert support.

ABSTRACT

ABSTRACT

Prediction models play a pivotal role in medical practice. To ensure their clinical applicability, it is essential to guarantee the quality of predictive models at multiple stages. In this article, we propose twelve recommendations for the development and clinical implementation of prediction models. These include identifying clinical needs, selecting appropriate predictors, performing predictor transformations and binning, specifying suitable models, assessing model performance, evaluating reproducibility and transportability, updating models, conducting impact evaluations, and promoting model adoption. These recommendations are grounded in a comprehensive synthesis of insights from existing literature and our extensive clinical and statistical experience in the development and practical application of prediction models.

FullText(HTML)

REFERENCES

[1]	Wynants, L., Van Calster, B., Collins, G.S., et al. (2020). Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. BMJ 369: m1328. DOI: 10.1136/bmj.m1328. View in Article Google Scholar
[2]	Liu, Y., Feng, W., Lou, J., et al. (2023). Performance of a prediabetes risk prediction model: A systematic review. Heliyon 9: e15529. DOI: 10.1016/j.heliyon.2023.e15529. View in Article Google Scholar
[3]	Kaiser, I., Mathes, S., Pfahlberg, A.B., et al. (2022). Using the prediction model risk of bias assessment tool (PROBAST) to evaluate melanoma prediction studies. Cancers 14: 3033. DOI: 10.3390/cancers14123033. View in Article Google Scholar
[4]	Collins, G.S., Reitsma, J.B., Altman, D.G., et al. (2015). Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. Br. J. Cancer 112: 251−259. DOI: 10.1038/bjc.2014.639. View in Article CrossRef Google Scholar
[5]	Xu, H., Feng, G., Yang, R., et al. (2023). OvaRePred: Online tool for predicting the age of fertility milestones. The Innovation 4: 100490. DOI: 10.1016/j.xinn.2023.100490. View in Article Google Scholar
[6]	Xu, H., Feng, G., Shi, L., et al. (2023). PCOSt: A non-invasive and cost-effective screening tool for polycystic ovary syndrome. The Innovation 4: 100407. DOI: 10.1016/j.xinn.2023.100407. View in Article Google Scholar
[7]	Xu, H., Feng, G., Han, Y., et al. (2023). POvaStim: An online tool for directing individualized FSH doses in ovarian stimulation. The Innovation 4: 100401. DOI: 10.1016/j.xinn.2023.100401. View in Article Google Scholar
[8]	Steyerberg, E.W., and Vergouwe, Y. (2014). Towards better clinical prediction models: Seven steps for development and an ABCD for validation. Eur. Heart J. 35: 1925−1931. DOI: 10.1093/eurheartj/ehu207. View in Article CrossRef Google Scholar
[9]	Liao, Y., McGee, D.L., Cooper, R.S., et al. (1999). How generalizable are coronary risk prediction models? Comparison of Framingham and two national cohorts. Am. Heart J. 137: 837−845. DOI: 10.1016/s0002-8703(99)70407-2. View in Article CrossRef Google Scholar
[10]	Xu, H., Feng, G., Ma, C., et al. (2023). AMHconverter: An online tool for converting results between the different anti-Müllerian hormone assays of Roche Elecsys®, Beckman Access, and Kangrun. PeerJ 11: e15301. DOI: 10.7717/peerj.15301. View in Article Google Scholar
[11]	Xu, H., Feng, G., Wang, H., et al. (2020). A novel mathematical model of true ovarian reserve assessment based on predicted probability of poor ovarian response: A retrospective cohort study. J. Assist. Reprod. Genet. 37: 963−972. DOI: 10.1007/s10815-020-01700-1. View in Article CrossRef Google Scholar
[12]	Xu, H., Shi, L., Feng, G., et al. (2020). An ovarian reserve assessment model based on anti-müllerian hormone levels, follicle-stimulating hormone levels, and age: Retrospective cohort study. J. Med. Internet Res. 22: e19096. DOI: 10.2196/19096. View in Article Google Scholar
[13]	Han, Y., Xu, H., Feng, G., et al. (2022). An online tool for predicting ovarian reserve based on AMH level and age: A retrospective cohort study. Front. Endocrinol. 13: 946123. DOI: 10.3389/fendo.2022.946123. View in Article Google Scholar
[14]	Xu, H., Feng, G., Alpadi, K., et al. (2022). A model for predicting polycystic ovary syndrome using serum AMH, menstrual cycle length, body mass index and serum androstenedione in Chinese reproductive aged population: A retrospective cohort study. Front. Endocrinol. 13: 821368. DOI: 10.3389/fendo.2022.821368. View in Article Google Scholar
[15]	Zhang, X., Xu, H., Feng, G., et al. (2023). Sensitive HPLC-DMS/MS/MS method coupled with dispersive magnetic solid phase extraction followed by in situ derivatization for the simultaneous determination of multiplexing androgens and 17-hydroxyprogesterone in human serum and its application to patients with polycystic ovarian syndrome. Clin. Chim. Acta 538: 221−230. DOI: 10.1016/j.cca.2022.11.025. View in Article CrossRef Google Scholar
[16]	Xu, H., Zhang, X., Yang, R., et al. (2023). Can androgens be replaced by AMH in initial screening of Polycystic Ovary Syndrome? The Innovation Medicine 1: 100010. DOI: 10.59717/j.xinn-med.2023.100010. View in Article CrossRef Google Scholar
[17]	Vittinghoff, E., and McCulloch, C.E. (2007). Relaxing the rule of ten events per variable in logistic and cox regression. Am. J. Epidemiol. 165: 710−718. DOI: 10.1093/aje/kwk052. View in Article CrossRef Google Scholar
[18]	van Smeden, M., Moons, K.G.M., de Groot, J.A.H., et al. (2018). Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat. Methods Med. Res. 28: 2455−2474. DOI: 10.1177/0962280218784726. View in Article CrossRef Google Scholar
[19]	Steyerberg, E.W., Schemper, M., and Harrell, F.E. (2011). Logistic regression modeling and the number of events per variable: Selection bias dominates. J. Clin. Epidemiol. 64: 1464−1465. DOI: 10.1016/j.jclinepi.2011.06.016. View in Article CrossRef Google Scholar
[20]	van Smeden, M., de Groot, J.A.H., Moons, K.G.M., et al. (2016). No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med. Res. Methodol. 16: 163. DOI: 10.1186/s12874-016-0267-3. View in Article Google Scholar
[21]	Ogundimu, E.O., Altman, D.G., and Collins, G.S. (2016). Adequate sample size for developing prediction models is not simply related to events per variable. J. Clin. Epidemiol. 76: 175−182. DOI: 10.1016/j.jclinepi.2016.02.031. View in Article CrossRef Google Scholar
[22]	Austin, P.C., and Steyerberg, E.W. (2017). Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Stat. Methods Med. Res. 26: 796−808. DOI: 10.1177/0962280214558972. View in Article CrossRef Google Scholar
[23]	Wynants, L., Bouwmeester, W., Moons, K.G., et al. (2015). A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data. J. Clin. Epidemiol. 68: 1406−1414. DOI: 10.1016/j.jclinepi.2015.02.002. View in Article CrossRef Google Scholar
[24]	Riley, R.D., Ensor, J., Snell, K.I.E., et al. (2020). Calculating the sample size required for developing a clinical prediction model. BMJ 368 : m441. DOI: 10.1136/bmj.m441. View in Article Google Scholar
[25]	Riley, R.D., Snell, K.I.E., Ensor, J., et al. (2018). Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat. Med. 38: 1276−1296. DOI: 10.1002/sim.7992. View in Article CrossRef Google Scholar
[26]	Riley, R.D., Snell, K.I.E., Ensor, J., et al. (2019). Minimum sample size for developing a multivariable prediction model: Part I - Continuous outcomes. Stat. Med. 38: 1262−1275. DOI: 10.1002/sim.7993. View in Article CrossRef Google Scholar
[27]	Dhiman, P., Ma, J., Andaur Navarro, C.L., et al. (2022). Methodological conduct of prognostic prediction models developed using machine learning in oncology: A systematic review. BMC Med. Res. Methodol. 22: 101. DOI: 10.1186/s12874-022-01577-x. View in Article Google Scholar
[28]	van der Ploeg, T., Austin, P.C., and Steyerberg, E.W. (2014). Modern modelling techniques are data hungry: A simulation study for predicting dichotomous endpoints. BMC Med. Res. Methodol. 14: 137. DOI: 10.1186/1471-2288-14-137. View in Article CrossRef Google Scholar
[29]	Riley, R.D., Snell, K.I.E., Archer, L., et al. (2024). Evaluation of clinical prediction models (part 3): Calculating the sample size required for an external validation study. BMJ 384 : e074821. DOI: 10.1136/bmj-2023-074821. View in Article Google Scholar
[30]	Riley, R.D., Debray, T.P.A., Collins, G.S., et al. (2021). Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat. Med. 40: 4230−4251. DOI: 10.1002/sim.9025. View in Article CrossRef Google Scholar
[31]	Chevret, S., Seaman, S., and Resche-Rigon, M. (2015). Multiple imputation: A mature approach to dealing with missing data. Intensive Care Med. 41: 348−350. DOI: 10.1007/s00134-014-3624-x. View in Article CrossRef Google Scholar
[32]	Fletcher Mercaldo, S., and Blume, J.D. (2020). Missing data and prediction: The pattern submodel. Biostatistics 21: 236−252. DOI: 10.1093/biostatistics/kxy040. View in Article CrossRef Google Scholar
[33]	Sainani, K.L. (2015). Dealing with missing data. Pm&R 7: 990−994. DOI: 10.1016/j.pmrj.2015.07.011. View in Article CrossRef Google Scholar
[34]	Sterne, J.A., White, I.R., Carlin, J.B., et al. (2009). Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ 338: b2393. DOI: 10.1136/bmj.b2393. View in Article CrossRef Google Scholar
[35]	Steif, J., Brant, R., Sreepada, R.S., et al. (2021). Prediction model performance with different imputation strategies: A simulation study using a north American ICU registry. Pediatr. Crit. Care Med. 23: e29−e44. DOI: 10.1097/pcc.0000000000002835. View in Article CrossRef Google Scholar
[36]	Moons, K.G.M., Grobbee, D.E., Chen, Q., et al. (2009). Dealing with missing predictor values when applying clinical prediction models. Clin. Chem. 55: 994−1001. DOI: 10.1373/clinchem.2008.115345. View in Article CrossRef Google Scholar
[37]	Eekhout, I., de Boer, R.M., Twisk, J.W.R., et al. (2012). Missing data. Epidemiology 23: 729−732. DOI: 10.1097/EDE.0b013e3182576cdb. View in Article CrossRef Google Scholar
[38]	Nijman, S.W.J., Leeuwenberg, A.M., Beekers, I., et al. (2022). Missing data is poorly handled and reported in prediction model studies using machine learning: A literature review. J. Clin. Epidemiol. 142: 218−229. DOI: 10.1016/j.jclinepi.2021.11.023. View in Article CrossRef Google Scholar
[39]	Steyerberg, E.W. (2019). Dealing with missing values. (ed). Clinical prediction models: A practical approach to development, validation, and updating (Springer Nature), pp: 113-128. DOI: 10.1007/978-0-387-77244-8. View in Article Google Scholar
[40]	Zeng, H., Ran, X., An, L., et al. (2021). Disparities in stage at diagnosis for five common cancers in China: A multicentre, hospital-based, observational study. Lancet Public Health 6: e877−e887. DOI: 10.1016/s2468-2667(21)00157-2. View in Article CrossRef Google Scholar
[41]	Ciccione, L., Dehaene, G., and Dehaene, S. (2023). Outlier detection and rejection in scatterplots: Do outliers influence intuitive statistical judgments. J. Exp. Psychol. Hum. Percept. Perform. 49: 129−144. DOI: 10.1037/xhp0001065. View in Article CrossRef Google Scholar
[42]	Lakra, A., Banerjee, B., and Laha, A. (2023). A data-adaptive method for outlier detection from functional data. Stat. Comput. 34. DOI: 10.1007/s11222-023-10301-8. View in Article Google Scholar
[43]	Yang, J., Tan, X., and Rahardja, S. (2023). Outlier detection: How to select k for k-nearest-neighbors-based outlier detectors. Pattern Recogn. Lett. 174: 112−117. DOI: 10.1016/j.patrec.2023.08.020. View in Article CrossRef Google Scholar
[44]	Smiti, A. (2020). A critical overview of outlier detection methods. Comput. Sci. Rev. 38: 100306. DOI: 10.1016/j.cosrev.2020.100306. View in Article CrossRef Google Scholar
[45]	Guan, L., and Tibshirani, R. (2022). Prediction and outlier detection in classification problems. J. R. Stat. Soc. B 84: 524−546. DOI: 10.1111/rssb.12443. View in Article CrossRef Google Scholar
[46]	El-Masri, M.M., Mowbray, F.I., Fox-Wasylyshyn, S.M., et al. (2020). Multivariate outliers: A conceptual and practical overview for the nurse and health researcher. Can. J. Nurs. Res. 53: 316−321. DOI: 10.1177/0844562120932054. View in Article CrossRef Google Scholar
[47]	Sauerbrei, W., Royston, P., and Binder, H. (2007). Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat. Med. 26: 5512−5528. DOI: 10.1002/sim.3148. View in Article CrossRef Google Scholar
[48]	Binder, H., Sauerbrei, W., and Royston, P. (2013). Comparison between splines and fractional polynomials for multivariable model building with continuous covariates: A simulation study with continuous response. Stat. Med. 32: 2262−2277. DOI: 10.1002/sim.5639. View in Article CrossRef Google Scholar
[49]	Nieboer, D., Vergouwe, Y., Roobol, M.J., et al. (2015). Nonlinear modeling was applied thoughtfully for risk prediction: The prostate biopsy collaborative group. J. Clin. Epidemiol. 68: 426−434. DOI: 10.1016/j.jclinepi.2014.11.022. View in Article CrossRef Google Scholar
[50]	Ma, J., Dhiman, P., Qi, C., et al. (2023). Poor handling of continuous predictors in clinical prediction models using logistic regression: A systematic review. J. Clin. Epidemiol. 161: 140−151. DOI: 10.1016/j.jclinepi.2023.07.017. View in Article CrossRef Google Scholar
[51]	Royston, P., Altman, D.G., and Sauerbrei, W. (2006). Dichotomizing continuous predictors in multiple regression: A bad idea. Stat. Med. 25: 127−141. DOI: 10.1002/sim.2331. View in Article CrossRef Google Scholar
[52]	Altman, D.G., and Royston, P. (2006). The cost of dichotomising continuous variables. BMJ 332: 1080. DOI: 10.1136/bmj.332.7549.1080. View in Article CrossRef Google Scholar
[53]	Collins, G.S., Ogundimu, E.O., Cook, J.A., et al. (2016). Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat. Med. 35: 4124−4135. DOI: 10.1002/sim.6986. View in Article CrossRef Google Scholar
[54]	Zhou, J., You, D., Bai, J., et al. (2023). Machine learning methods in real-world studies of cardiovascular disease. Cardiovascular Innovations and Applications 7: 975. DOI: 10.15212/cvia.2023.0011. View in Article Google Scholar
[55]	Martin, S.A., Townend, F.J., Barkhof, F., et al. (2023). Interpretable machine learning for dementia: A systematic review. Alzheimer's & Dementia 19: 2135−2149. DOI: 10.1002/alz.12948. View in Article CrossRef Google Scholar
[56]	Wang, K., Tian, J., Zheng, C., et al. (2021). Interpretable prediction of 3-year all-cause mortality in patients with heart failure caused by coronary heart disease based on machine learning and SHAP. Comput. Biol. Med. 137: 104813. DOI: 10.1016/j.compbiomed.2021.104813. View in Article Google Scholar
[57]	Yi, F., Yang, H., Chen, D., et al. (2023). XGBoost-SHAP-based interpretable diagnostic framework for alzheimer’s disease. BMC Med. Inform. Decis. Mak. 23: 137. DOI: 10.1186/s12911-023-02238-9. View in Article Google Scholar
[58]	Gravesteijn, B.Y., Nieboer, D., Ercole, A., et al. (2020). Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury. J. Clin. Epidemiol. 122: 95−107. DOI: 10.1016/j.jclinepi.2020.03.005. View in Article CrossRef Google Scholar
[59]	Christodoulou, E., Ma, J., Collins, G.S., et al. (2019). A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J. Clin. Epidemiol. 110: 12−22. DOI: 10.1016/j.jclinepi.2019.02.004. View in Article CrossRef Google Scholar
[60]	Dhiman, P., Ma, J., Andaur Navarro, C.L., et al. (2023). Overinterpretation of findings in machine learning prediction model studies in oncology: A systematic review. J. Clin. Epidemiol. 157: 120−133. DOI: 10.1016/j.jclinepi.2023.03.012. View in Article CrossRef Google Scholar
[61]	Chowdhury, M.Z.I., and Turin, T.C. (2020). Variable selection strategies and its importance in clinical prediction modelling. Fam. Med. Community Health 8: e000262. DOI: 10.1136/fmch-2019-000262. View in Article Google Scholar
[62]	Heinze, G., Wallisch, C., and Dunkler, D. (2018). Variable selection – A review and recommendations for the practicing statistician. Biom. J. 60: 431−449. DOI: 10.1002/bimj.201700067. View in Article CrossRef Google Scholar
[63]	Sanchez-Pinto, L.N., Venable, L.R., Fahrenbach, J., et al. (2018). Comparison of variable selection methods for clinical predictive modeling. Int. J. Med. Inform. 116: 10−17. DOI: 10.1016/j.ijmedinf.2018.05.006. View in Article CrossRef Google Scholar
[64]	Hastie, T., Tibshirani, R., and Tibshirani, R. (2020). Best subset, forward stepwise or lasso? Analysis and recommendations based on extensive comparisons. Statist. Sci. 35: 579-592. DOI: 10.1214/19-sts733. View in Article Google Scholar
[65]	Hanke, M., Dijkstra, L., Foraita, R., et al. (2023). Variable selection in linear regression models: Choosing the best subset is not always the best choice. Biom. J. 66: e2200209. DOI: 10.1002/bimj.202200209. View in Article Google Scholar
[66]	Strandberg, R., Jepsen, P., and Hagström, H. (2024). Developing and validating clinical prediction models in hepatology – An overview for clinicians. J. Hepatol. 81: 149-162. DOI: 10.1016/j.jhep.2024.03.030. View in Article Google Scholar
[67]	Alba, A.C., Agoritsas, T., Walsh, M., et al. (2017). Discrimination and calibration of clinical prediction models: Users' guides to the medical literature. JAMA 318: 1377-1384. DOI: 10.1001/jama.2017.12126. View in Article Google Scholar
[68]	Wessler, B.S., Lai Yh, L., Kramer, W., et al. (2015). Clinical Prediction Models for Cardiovascular Disease. Circulation: Cardiovascular Quality and Outcomes 8: 368−375. DOI: 10.1161/circoutcomes.115.001693. View in Article CrossRef Google Scholar
[69]	Carrick, R.T., Park, J.G., McGinnes, H.L., et al. (2020). Clinical Predictive Models of Sudden Cardiac Arrest: A Survey of the Current Science and Analysis of Model Performances. Journal of the American Heart Association 9. DOI: 10.1161/jaha.119.017625. View in Article Google Scholar
[70]	Nahm, F.S. (2022). Receiver operating characteristic curve: overview and practical use for clinicians. Korean Journal of Anesthesiology 75: 25−36. DOI: 10.4097/kja.21209. View in Article CrossRef Google Scholar
[71]	Chicco, D., and Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21. DOI: 10.1186/s12864-019-6413-7. View in Article Google Scholar
[72]	Cowley, L.E., Farewell, D.M., Maguire, S., et al. (2019). Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagnostic and Prognostic Research 3. DOI: 10.1186/s41512-019-0060-y. View in Article Google Scholar
[73]	Yu, W., Xu, W., and Zhu, L. (2017). A Modified Hosmer–Lemeshow Test for Large Data Sets. Communications in Statistics - Theory and Methods 46. DOI: 10.1080/03610926.2017.1285922. View in Article Google Scholar
[74]	Paul, P., Pennell, M.L., and Lemeshow, S. (2013). Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets. Stat Med 32: 67−80. DOI: 10.1002/sim.5525. View in Article CrossRef Google Scholar
[75]	Austin, P.C., Harrell, F.E., and van Klaveren, D. (2020). Graphical calibration curves and the integrated calibration index (ICI) for survival models. Statistics in Medicine 39: 2714−2742. DOI: 10.1002/sim.8570. View in Article CrossRef Google Scholar
[76]	Austin, P.C., and Steyerberg, E.W. (2019). The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine 38: 4051−4065. DOI: 10.1002/sim.8281. View in Article CrossRef Google Scholar
[77]	Royston, P., and Altman, D.G. (2013). External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol 13: 33. DOI: 10.1186/1471-2288-13-33. View in Article CrossRef Google Scholar
[78]	Steyerberg, E.W., Vickers, A.J., Cook, N.R., et al. (2010). Assessing the Performance of Prediction Models. Epidemiology 21: 128−138. DOI: 10.1097/EDE.0b013e3181c30fb2. View in Article CrossRef Google Scholar
[79]	Huang, Y., Li, W., Macheret, F., et al. (2020). A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association 27: 621−633. DOI: 10.1093/jamia/ocz228. View in Article CrossRef Google Scholar
[80]	Rufibach, K. (2010). Use of Brier score to assess binary predictions. Journal of Clinical Epidemiology 63: 938−939. DOI: 10.1016/j.jclinepi.2009.11.009. View in Article CrossRef Google Scholar
[81]	Vickers, A.J., and Holland, F. (2021). Decision curve analysis to evaluate the clinical benefit of prediction models. The Spine Journal 21: 1643−1648. DOI: 10.1016/j.spinee.2021.02.024. View in Article CrossRef Google Scholar
[82]	Van Calster, B., Wynants, L., Verbeek, J.F.M., et al. (2018). Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators. European Urology 74: 796−804. DOI: 10.1016/j.eururo.2018.08.038. View in Article CrossRef Google Scholar
[83]	Justice, A.C., Covinsky, K.E., and Berlin, J.A. (1999). Assessing the generalizability of prognostic information. Ann Intern Med 130: 515−524. DOI: 10.7326/0003-4819-130-6-199903160-00016. View in Article CrossRef Google Scholar
[84]	Ramspek, C.L., Jager, K.J., Dekker, F.W., et al. (2021). External validation of prognostic models: what, why, how, when and where. Clinical Kidney Journal 14: 49−58. DOI: 10.1093/ckj/sfaa188. View in Article CrossRef Google Scholar
[85]	Collins, G.S., Dhiman, P., Ma, J., et al. (2024). Evaluation of clinical prediction models (part 1): from development to external validation. Bmj. DOI: 10.1136/bmj-2023-074819. View in Article Google Scholar
[86]	Riley, R.D., Archer, L., Snell, K.I.E., et al. (2024). Evaluation of clinical prediction models (part 2): how to undertake an external validation study. Bmj. DOI: 10.1136/bmj-2023-074820. View in Article Google Scholar
[87]	Moons, K.G.M., Kengne, A.P., Woodward, M., et al. (2012). Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 98: 683−690. DOI: 10.1136/heartjnl-2011-301246. View in Article CrossRef Google Scholar
[88]	Staffa, S.J., and Zurakowski, D. (2021). Statistical Development and Validation of Clinical Prediction Models. Anesthesiology 135: 396−405. DOI: 10.1097/aln.0000000000003871. View in Article CrossRef Google Scholar
[89]	Steyerberg, E.W., Harrell, F.E., Jr., Borsboom, G.J., et al. (2001). Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54: 774−781. DOI: 10.1016/s0895-4356(01)00341-9. View in Article CrossRef Google Scholar
[90]	Steyerberg, E.W., and Harrell, F.E. (2016). Prediction models need appropriate internal, internal–external, and external validation. Journal of Clinical Epidemiology 69: 245−247. DOI: 10.1016/j.jclinepi.2015.04.005. View in Article CrossRef Google Scholar
[91]	Steyerberg, E.W., Bleeker, S.E., Moll, H.A., et al. (2003). Internal and external validation of predictive models: A simulation study of bias and precision in small samples. Journal of Clinical Epidemiology 56: 441−447. DOI: 10.1016/s0895-4356(03)00047-7. View in Article CrossRef Google Scholar
[92]	Macleod, M.R., Bouwmeester, W., Zuithoff, N.P.A., et al. (2012). Reporting and Methods in Clinical Prediction Research: A Systematic Review. PLOS Medicine 9. DOI: 10.1371/journal.pmed.1001221. View in Article Google Scholar
[93]	Mallett, S., Royston, P., Waters, R., et al. (2010). Reporting performance of prognostic models in cancer: a review. BMC Medicine 8. DOI: 10.1186/1741-7015-8-21. View in Article Google Scholar
[94]	Steyerberg, E.W., Moons, K.G.M., van der Windt, D.A., et al. (2013). Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research. PLOS Medicine 10. DOI: 10.1371/journal.pmed.1001381. View in Article Google Scholar
[95]	Riley, R.D., Ensor, J., Snell, K.I.E., et al. (2016). External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. DOI: 10.1136/bmj.i3140. View in Article Google Scholar
[96]	Altman, D.G., Vergouwe, Y., Royston, P., et al. (2009). Prognosis and prognostic research: validating a prognostic model. BMJ 338: b605−b605. DOI: 10.1136/bmj.b605. View in Article CrossRef Google Scholar
[97]	Ramspek, C., Voskamp, P., van Ittersum, F., et al. (2017). Prediction models for the mortality risk in chronic dialysis patients: a systematic review and independent external validation study. Clinical Epidemiology Volume 9: 451−464. DOI: 10.2147/clep.S139748. View in Article CrossRef Google Scholar
[98]	Phung, M.T., Tin Tin, S., and Elwood, J.M. (2019). Prognostic models for breast cancer: a systematic review. BMC Cancer 19. DOI: 10.1186/s12885-019-5442-6. View in Article Google Scholar
[99]	Perel, P., Edwards, P., Wentz, R., et al. (2006). Systematic review of prognostic models in traumatic brain injury. BMC Medical Informatics and Decision Making 6. DOI: 10.1186/1472-6947-6-38. View in Article Google Scholar
[100]	Toll, D.B., Janssen, K.J.M., Vergouwe, Y., et al. (2008). Validation, updating and impact of clinical prediction rules: A review. Journal of Clinical Epidemiology 61: 1085−1094. DOI: 10.1016/j.jclinepi.2008.04.008. View in Article CrossRef Google Scholar
[101]	Moons, K.G.M., Kengne, A.P., Grobbee, D.E., et al. (2012). Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98: 691−698. DOI: 10.1136/heartjnl-2011-301247. View in Article CrossRef Google Scholar
[102]	Binuya, M.A.E., Engelhardt, E.G., Schats, W., et al. (2022). Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review. BMC Medical Research Methodology 22. DOI: 10.1186/s12874-022-01801-8. View in Article Google Scholar
[103]	Janssen, K.J.M., Moons, K.G.M., Kalkman, C.J., et al. (2008). Updating methods improved the performance of a clinical prediction model in new patients. Journal of Clinical Epidemiology 61: 76−86. DOI: 10.1016/j.jclinepi.2007.04.018. View in Article CrossRef Google Scholar
[104]	Su, T.L., Jaki, T., Hickey, G.L., et al. (2018). A review of statistical updating methods for clinical prediction models. Statistical Methods in Medical Research 27: 185−197. DOI: 10.1177/0962280215626466. View in Article CrossRef Google Scholar
[105]	Nieboer, D., Vergouwe, Y., Ankerst, D.P., et al. (2016). Improving prediction models with new markers: a comparison of updating strategies. BMC Medical Research Methodology 16. DOI: 10.1186/s12874-016-0231-2. View in Article Google Scholar
[106]	van Houwelingen, H.C. (2000). Validation, calibration, revision and combination of prognostic survival models. Stat Med 19: 3401−3415. DOI: 3.0.co;2-2">10.1002/1097-0258(20001230)19:24<3401::aid-sim554>3.0.co;2-2. View in Article CrossRef Google Scholar
[107]	Siregar, S., Nieboer, D., Versteegh, M.I.M., et al. (2019). Methods for updating a risk prediction model for cardiac surgery: a statistical primer. Interactive CardioVascular and Thoracic Surgery 28: 333−338. DOI: 10.1093/icvts/ivy338. View in Article CrossRef Google Scholar
[108]	Pencina, M.J., D'Agostino, R.B., and Steyerberg, E.W. (2010). Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine 30: 11−21. DOI: 10.1002/sim.4085. View in Article CrossRef Google Scholar
[109]	Steyerberg, E.W., Pencina, M.J., Lingsma, H.F., et al. (2011). Assessing the incremental value of diagnostic and prognostic markers: a review and illustration. European Journal of Clinical Investigation 42: 216−228. DOI: 10.1111/j.1365-2362.2011.02562.x. View in Article CrossRef Google Scholar
[110]	Pepe, M.S., Fan, J., Feng, Z., et al. (2014). The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets. Statistics in Biosciences 7: 282−295. DOI: 10.1007/s12561-014-9118-0. View in Article CrossRef Google Scholar
[111]	Kerr, K.F. (2023). Net Reclassification Index Statistics Do Not Help Assess New Risk Models. Radiology 306. DOI: 10.1148/radiol.222343. View in Article Google Scholar
[112]	Grunkemeier, G.L., and Jin, R. (2015). Net Reclassification Index: Measuring the Incremental Value of Adding a New Risk Factor to an Existing Risk Model. The Annals of Thoracic Surgery 99: 388−392. DOI: 10.1016/j.athoracsur.2014.10.084. View in Article CrossRef Google Scholar
[113]	Burch, P.M., Glaab, W.E., Holder, D.J., et al. (2016). Net Reclassification Index and Integrated Discrimination Index Are Not Appropriate for Testing Whether a Biomarker Improves Predictive Performance. Toxicological Sciences. DOI: 10.1093/toxsci/kfw225. View in Article Google Scholar
[114]	Kattan, M.W. (2003). Judging new markers by their ability to improve predictive accuracy. J Natl Cancer Inst 95: 634−635. DOI: 10.1093/jnci/95.9.634. View in Article CrossRef Google Scholar
[115]	Debray, T.P.A., Riley, R.D., Rovers, M.M., et al. (2015). Individual Participant Data (IPD) Meta-analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use. PLOS Medicine 12. DOI: 10.1371/journal.pmed.1001886. View in Article Google Scholar
[116]	Debray, T.P.A., Koffijberg, H., Nieboer, D., et al. (2014). Meta‐analysis and aggregation of multiple published prediction models. Statistics in Medicine 33: 2341−2362. DOI: 10.1002/sim.6080. View in Article CrossRef Google Scholar
[117]	Debray, T.P., Moons, K.G., Ahmed, I., et al. (2013). A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med 32: 3158−3180. DOI: 10.1002/sim.5732. View in Article CrossRef Google Scholar
[118]	Hickey, G.L., Grant, S.W., Caiado, C., et al. (2013). Dynamic Prediction Modeling Approaches for Cardiac Surgery. Circulation: Cardiovascular Quality and Outcomes 6: 649−658. DOI: 10.1161/circoutcomes.111.000012. View in Article CrossRef Google Scholar
[119]	Schnellinger, E.M., Yang, W., and Kimmel, S.E. (2021). Comparison of dynamic updating strategies for clinical prediction models. Diagnostic and Prognostic Research 5. DOI: 10.1186/s41512-021-00110-w. View in Article Google Scholar
[120]	McCormick, T.H., Raftery, A.E., Madigan, D., et al. (2011). Dynamic Logistic Regression and Dynamic Model Averaging for Binary Classification. Biometrics 68: 23−30. DOI: 10.1111/j.1541-0420.2011.01645.x. View in Article CrossRef Google Scholar
[121]	Jenkins, D.A., Sperrin, M., Martin, G.P., et al. (2018). Dynamic models to predict health outcomes: current status and methodological challenges. Diagnostic and Prognostic Research 2. DOI: 10.1186/s41512-018-0045-2. View in Article Google Scholar
[122]	Siregar, S., Nieboer, D., Vergouwe, Y., et al. (2016). Improved Prediction by Dynamic Modeling. Circulation: Cardiovascular Quality and Outcomes 9: 171−181. DOI: 10.1161/circoutcomes.114.001645. View in Article CrossRef Google Scholar
[123]	Moons, K.G.M., Altman, D.G., Vergouwe, Y., et al. (2009). Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338: b606−b606. DOI: 10.1136/bmj.b606. View in Article CrossRef Google Scholar
[124]	Kappen, T.H., van Klei, W.A., van Wolfswinkel, L., et al. (2018). Evaluating the impact of prediction models: lessons learned, challenges, and recommendations. Diagnostic and Prognostic Research 2. DOI: 10.1186/s41512-018-0033-6. View in Article Google Scholar
[125]	Meyer, G., Köpke, S., Bender, R., et al. (2005). Predicting the risk of falling – efficacy of a risk assessment tool compared to nurses' judgement: a cluster-randomised controlled trial [ISRCTN37794278]. BMC Geriatrics 5. DOI: 10.1186/1471-2318-5-14. View in Article Google Scholar
[126]	Foy, R., Penney, G.C., Grimshaw, J.M., et al. (2004). A randomised controlled trial of a tailored multifaceted strategy to promote implementation of a clinical guideline on induced abortion care. Bjog 111: 726−733. DOI: 10.1111/j.1471-0528.2004.00168.x. View in Article CrossRef Google Scholar
[127]	Grayling, M.J., Wason, J.M.S., and Mander, A.P. (2017). Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features. Trials 18. DOI: 10.1186/s13063-017-1783-0. View in Article Google Scholar
[128]	Li, F., and Wang, R. (2022). Stepped Wedge Cluster Randomized Trials: A Methodological Overview. World Neurosurgery 161: 323−330. DOI: 10.1016/j.wneu.2021.10.136. View in Article CrossRef Google Scholar
[129]	Huang, T., Xu, H., Wang, H., et al. (2023). Artificial intelligence for medicine: Progress, challenges, and perspectives. The Innovation Medicine 1. DOI: 10.59717/j.xinn-med.2023.100030. View in Article Google Scholar
[130]	Liu, X., Zhang, S., Shao, L., et al. (2024). Improving prediction of treatment response and prognosis in colorectal cancer with AI-based medical image analysis. The Innovation Medicine 2. DOI: 10.59717/j.xinn-med.2024.100069. View in Article Google Scholar

Cited by

ABOUT THIS ARTICLE

Cite this article:

Feng G., Xu H., Wan S., et al., (2024). Twelve practical recommendations for developing and applying clinical predictive models. The Innovation Medicine 2(4): 100105. https://doi.org/10.59717/j.xinn-med.2024.100105

Feng G., Xu H., Wan S., et al., (2024). Twelve practical recommendations for developing and applying clinical predictive models. The Innovation Medicine 2(4): 100105. https://doi.org/10.59717/j.xinn-med.2024.100105

Welcome!

To request copyright permission to republish or share portions of our works, please visit Copyright Clearance Center's (CCC) Marketplace website at marketplace.copyright.com.

PDF

Figures(3) Tables(6)

Article Metrics

Article views(16014) PDF downloads(11212)

Twelve practical recommendations for developing and applying clinical predictive models

GRAPHICAL ABSTRACT

PUBLIC SUMMARY

ABSTRACT

REFERENCES

ABOUT THIS ARTICLE

Cite this article:

Welcome!

Share

Article Metrics

Relative Articles

Cited by

Catalog

{{newsColumn.name}}

Official Account

{{newsColumn.name}}

Stay connected

{{newsColumn.name}}

Twelve practical recommendations for developing and applying clinical predictive models

GRAPHICAL ABSTRACT

PUBLIC SUMMARY

ABSTRACT

REFERENCES

ABOUT THIS ARTICLE

Cite this article:

Welcome!

Share

Article Metrics

Relative Articles

Cited by

Catalog

Export File

Citation

Format

Content

{{newsColumn.name}}

Official Account

{{newsColumn.name}}

Stay connected

{{newsColumn.name}}