Comparison of Diagnostic Accuracy using Bootstrapping Methods versus a Machine Learning Algorithm using the Complex Trial Protocol (CTP)

Abstract

Because machine learning (ML) algorithms are objective and can find patterns in large data sets that might gounnoticed by the naked eye, ML often improves clinical diagnosis of disease. Since theP300-based concealed information test (CIT) also uses biomedical data to make a “diagnosis” about whether someone is knowledgeable about crime-related details, its accuracy might also improve through the use of ML. Using the beatable “three stimulus protocol” (3SP), Abootalebi, Moradi, & Khalilzadeh (2009) showed that their ML algorithm outperformed the traditional bootstrapping and cross-correlational methods for diagnosis. Here, we expand on their work by applying ML techniques to data acquired using the countermeasure-resistant Complex Trial Protocol (CTP) version of the P300-based CIT, with the goal of comparing diagnostic accuracy between this approach and our currently used bootstrapping method. As expected, grand averaged ERPs showed the CIT effect (i.e., a larger probe compared to irrelevant response) in the guilty group but not the innocent group. While results comparing diagnostic methods revealed a seemingly higher hit rate (80% vs. 73%) and area under the curve (AUC) (.872 vs. .712) using bootstrapping compared to ML, standardized z-scores did not provide evidence suggesting the superiority of one approach over the other (Z=1.06, p>.2, two-tailed). These findings offer no evidence that the ML algorithm used here increases diagnostic accuracy of the CTP, and thus do not support that the ML method should replace the currently used bootstrapping method. Limitations that may explain our findings are discussed, along with future directions.