Date: January 9, 2025 Reference: White et al. Evidence of questionable research practices in clinical prediction models. BMC Med 2023 Guest Skeptic: Dr. Jestin Carlson is the Program Director for the AHN-Saint Vincent EM Residency in Erie Pennsylvania. He is the former National Director of Clinical Education for US Acute Care Solutions and an American Red Cross Scientific Advisory Council member. Dr. Richard Bukata We have had the pleasure of both working for the Legend of EM, Dr. Richard Bukata. He is an amazing educator and a great human being. He has been involved in medical education for over 40 years. He helped create the Emergency Medicine and Acute Care course, a ‘year-in-review’ course where the faculty review over 200 articles from the last year in a rapid-fire, tag-team format, meaning one presents one article, the other provides additional commentary, and then they switch. Each article takes about 2-3 minutes. The faculty is amazing, and the course is held in some wonderful locations: Vail, Maui, New York City, New Orleans, Hilton Head, San Diego, and Key West. There is also a self-study option if you are not able to attend in person. Case: You are working with a fourth-year medical student who is an avid listener to the Skeptics Guide to Emergency Medicine podcast. They recently listened to an episode examining a paper that used receiver operating characteristic curves or ROC curves to determine the accuracy of a predictive model by looking at the area under the curve or AUC. The student knows from other SGEM podcasts that there has been evidence of p-hacking in the medical literature and wonders if there have been similar instances with ROC curves. They ask you if there is any evidence of ‘ROC’ or ‘AUC-hacking?’ Background: To answer that young skeptic’s question, they must understand ROC curves. The ROC is a tool used to evaluate the diagnostic performance of a test or prediction model. The curve is graphed with the true positive rate (sensitivity) on the y-axis and the false positive rate (1-specificity) on the x-axis at various threshold levels for classifying a test result as positive or negative. ROC curves help clinicians determine how well a test or model can differentiate between patients with and without a condition. A perfect test would have a point at the top-left corner of the graph (sensitivity = 1, specificity = 1). The area under the curve (AUC) is often used to summarize a prediction model's discriminatory capacity. A result of 1.0 indicates perfect discrimination, while an AUC of 0.5 suggests performance no better than chance—essentially, a coin toss. By comparing the ROC curves of different tests or models, clinicians can identify which performs better in discrimination. Interpretation of the AUC often hinges on thresholds. Values of 0.7, 0.8, and 0.9 are commonly labelled as “fair,” “good,” or “excellent.” These descriptors, while convenient, are arbitrary and lack scientific foundation. Their widespread use introduces a strong temptation for researchers to achieve “better” AUC values. This drive can lead to things like p-hacking, a questionable research practice in which investigators manipulate data or analyses to cross predefined thresholds. P-hacking is not exclusive to AUC but is a well-documented problem in broader research, particularly surrounding the 0.05 p-value significance threshold. In the context of AUC, p-hacking might include selectively reporting favourable results, re-analyzing data multiple times, or even tweaking model parameters to inflate values. Such practices risk misleading clinicians and compromising patient care by promoting overly optimistic models. Understanding the prevalence and mechanisms of AUC-related p-hacking is vital for emergency physicians who often rely on clinical prediction tools for critical decisions. As the use of these models grows, so does the importance of transparent and robust research practices. Clinical Question: Is there evidence of questionable research practices, such as data manipulation or "hacking", in studies developing clinical prediction models? Reference: White et al. Evidence of questionable research practices in clinical prediction models. BMC Med 2023 Population: PubMed abstracts that reported at least one AUC value related to clinical prediction models Excluded: Abstracts with fewer than 10 words, pharmacokinetic studies using AUC for unrelated purposes, meta-analyses or pooled analyses, and tutorial papers lacking original findings Exposure: NA Comparison: No explicit comparison group was utilized, but the study benchmarked observed AUC distributions against expected smooth patterns. Primary Outcome: Evidence of irregular distributions, such as excess AUC values just above thresholds (e.g., 0.7, 0.8 or 0.9), that could suggest data manipulation. Type of Study: Observational study based on secondary data analysis. Authors’ Conclusions: “The AUCs for some models are over-inflated, which risks exposing patients
