Every data scientist knows the lingering sense of unease that accompanies a model boasting 99% accuracy. Instead of celebrating, experienced practitioners immediately grow suspicious. Nine times out of ten, the culprit is data leakage: some hidden piece of information embedded within the training data that would never actually be available at prediction time in a production environment.

To explore this persistent challenge, data scientist and StrataScratch founder Nate Rosidi recently conducted an experiment to answer two critical questions. First, how dramatic is that accuracy jump on real-world data rather than simplified toy examples? Second, if a data pipeline is described to an artificial intelligence model beforehand—with no hints that anything is fundamentally wrong—can the AI catch the leak, and does its detection capability vary depending on the specific model utilized?

To test these questions, researchers took four distinct real-world data projects and intentionally planted a well-known leakage pattern into each one. They then trained authentic machine learning models both with and without the respective leaks, recording the actual performance metrics. Separately, they described each pipeline in plain language to four different Claude models, ranging from the fastest to the strongest variants available, to see what issues would be flagged before trusting the results.

Setting Up the Experiment

Data leakage can infiltrate a pipeline through various routes. A feature might mechanically define the label itself, a feature could be constructed using future information, a dataset split might completely ignore underlying relational structures, or a vital preprocessing step could be executed in the wrong chronological order.

To evaluate how AI handles these oversights, the testing phase utilized Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5, and Claude Fable 5.1, spanning a wide spectrum of speed and capability. Each model was given a fresh session with zero memory of previous tests, relying entirely on a closed-book description of the pipeline, features, or code without any ability to execute code or run scripts. Critically, none of the AI models were told that a leak existed.

Case 1: The Feature That Already Knows the Answer

The first case study utilized datasets from a stock price direction prediction project. The core task was straightforward: predict whether Amazon’s closing price would land higher than that same day’s opening price, where the target label equals one if the closing price exceeds the opening price, and zero otherwise.

When the predictive model was fed that same day’s closing price and adjusted closing price alongside the opening, high, low, and volume figures, the validation accuracy skyrocketed to 97.0%, accompanied by an area under the curve (AUC) of 0.999. However, when the model was strictly restricted to prior-day data—omitting anything from the day actually being predicted—accuracy plummeted to 51.6% with an AUC of 0.476, essentially a coin flip. That stark contrast between 97% and 52% illustrates the profound distorting power of leakage.

When these pipelines were described to the AI models, all four caught the flaw instantly and for the correct underlying reason: the closing price directly defines the label. Haiku characterized the setup as circular, while Sonnet noted that the model could easily reconstruct the label using a single comparison. Opus went even further unprompted, pointing out that if the high equals the open, the asset never traded above the open, meaning the label must be zero. Opus also correctly established an honest performance baseline, predicting that a fair model should land in the low 50s for accuracy, mirroring the results of the clean run.

Case 2: The Dataset That Already Told You Where the Line Is

The second scenario turned to delivery duration prediction using historical data from DoorDash, containing nearly 200,000 individual orders. The objective was to predict the duration in seconds between an order being placed and delivered.

Researchers engineered a specific store-average delivery duration feature that inadvertently carried two separate leaks. First, the store average was calculated by grouping the entire dataset, combining training and testing rows together, and averaging the delivery durations before mapping that average onto every row. For stores with only a handful of orders, this meant the model was reading a smoothed copy of the actual answer. Second, because the table contained timestamps, shuffling the data randomly before an 80/20 train-test split allowed training rows to fall chronologically after test rows, violating the reality that a production model only encounters historical orders.

When scored using root mean squared error, the leaked version reported an error of 862.7 seconds compared to 949.9 seconds for the honest version. This apparent 9% performance improvement stemmed entirely from the leak rather than genuine learning.

Every AI model successfully caught the store-average leak, but only three of the four—Sonnet, Opus, and Fable—identified the critical flaw of shuffling a timestamped table instead of executing a chronological split. Haiku stopped its analysis after addressing the store-average feature and overlooked the improper data split entirely.

Case 3: The Leak You Can’t See in a Single Column

The third experiment focused on fraud detection data from Revolut, featuring a naturally one-to-many schema comprising user profiles and transaction histories. Because multiple transaction rows shared a single user-level fraud label, splitting the joined transaction rows randomly resulted in the vast majority of users having transactions on both sides of the split. Consequently, the test set frequently contained a different transaction from a user the model had already encountered during training.

A model trained using this random row-level split scored an AUC of 0.960. However, when the data was partitioned properly by user—ensuring no individual’s transactions crossed the boundary—the honest AUC dropped to 0.898, representing a solid model without artificial inflation.

All four AI models identified this structural mechanism unprompted, recognizing that because the label lived at the user level, a row-level split enabled the model to partly recognize a familiar user rather than genuinely detect fraud in new accounts. Opus delivered the most comprehensive assessment, highlighting that the true sample size corresponded to the number of users rather than the transactions, while also noting secondary administrative leaks present in account status fields.

Case 4: The Leak That Is Not in the Data at All

The final case study examined loan flagging data from Chime, where approximately 10% of loans were classified as bad. To compensate for this class imbalance, the initial pipeline duplicated minority-class rows prior to training.

Researchers evaluated a brief code snippet without any accompanying data or performance metrics, comparing an oversample-then-split approach against a properly ordered split-then-oversample pipeline. The flawed oversample-then-split method reported an inflated AUC of 0.986, while the properly sequestered split-then-oversample pipeline yielded a modest but honest AUC of 0.637. The core issue was that duplicating rows before the split allowed identical copies of the exact same loan to land in both halves, grading the model on examples it had already processed.

All four AI models flagged the methodology purely from the code snippet, requiring no performance metrics to prompt their suspicion. Opus again led the analysis by noting that stratifying on already-balanced labels was meaningless, pointing out the absence of random seeds, and questioning whether oversampling was even the optimal strategy compared to adjusting class weights or reporting recall metrics.

Conclusion and Observations

Every AI model tested—including the fastest and most economical variants—successfully caught the primary planted leak in every case when asked to review the pipeline beforehand. These findings demonstrate that common leakage patterns are readily identifiable by language models when framed with the correct questions.

However, the depth of insight varied significantly among the models. While Haiku provided concise and correct answers, its brevity occasionally caused it to miss secondary errors sitting alongside the primary leak. Sonnet and Fable consistently caught multiple compounding problems when present. Opus distinguished itself by regularly contributing contextual insights beyond the immediate leak, including realistic accuracy expectations, valid performance baselines, and critiques of broader methodological choices.

Utilizing AI to review machine learning pipelines before trusting performance metrics offers a fast, accessible safeguard capable of catching structural flaws. Ultimately, whether an AI reviewer catches every underlying issue depends heavily on the specific model employed and the comprehensiveness of the pipeline description provided.