When Explanations Mislead: What XAI's Faithfulness Crisis Means for AI Peer Review and Scientific Validation

The Explanation That Explains Nothing: A Crisis at the Heart of Interpretable AI

For more than a decade, the promise of Explainable Artificial Intelligence has been straightforward and seductive: if we can show researchers, clinicians, and policymakers why a model makes a particular prediction, they will be better equipped to trust it when it is correct and challenge it when it is wrong. A new position paper from arXiv (2610.12493) disrupts this narrative with uncomfortable precision. The authors argue that the faithfulness metrics underpinning XAI feature-importance methods are structurally misaligned with the models they claim to describe — and that a decade of accumulated evidence points not to increased understanding, but to overreliance on incorrect predictions followed, eventually, by wholesale algorithmic aversion. For researchers who rely on AI research tools to validate, interpret, or communicate their findings, this is not an abstract philosophical concern. It is a practical problem embedded in the workflows of modern science.

Understanding the Faithfulness Problem in Feature Importance Methods

Before examining the implications for scientific practice, it is worth being precise about what the paper actually claims. Feature importance methods — tools such as SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and integrated gradients — assign numerical scores to input features, purporting to indicate how much each feature contributed to a specific model output. A radiologist using an AI diagnostic tool, for instance, might see a heatmap highlighting certain pixels as "most influential." A genomics researcher might receive a ranked list of gene variants that supposedly drove a classification decision.

The concept of faithfulness in this context refers to whether these explanations accurately reflect the internal computational processes of the underlying model. A faithful explanation is one where the highlighted features genuinely correspond to the mathematical pathways that produced the prediction. An unfaithful explanation is, in essence, a story — plausible, potentially coherent, but not causally connected to what the model actually computed.

The position paper's central claim is that current faithfulness metrics are not merely imperfect proxies but are structurally incapable of measuring what they purport to measure. The problem operates at multiple levels:

• Perturbation-based evaluation methods — the dominant approach for testing faithfulness — generate counterfactual inputs that often fall outside the model's training distribution, making the resulting fidelity scores unreliable by construction.
• Metric disagreement is pervasive: studies have consistently shown that different faithfulness metrics applied to the same explanation method on the same model frequently produce contradictory rankings, with correlation coefficients between metrics sometimes approaching zero.
• Benchmark circularity emerges when the models used to validate explainability tools are themselves simplified or synthetic, meaning that faithfulness measured in benchmark conditions may have no relationship to faithfulness in deployment.

These are not minor calibration errors. They represent a foundational misalignment between what XAI faithfulness metrics claim to measure and what they actually capture.

Why This Matters Specifically for AI in Scientific Research

The consequences of unfaithful explanations are qualitatively different in scientific research than in many commercial applications. In a recommendation engine, an unfaithful explanation is an inconvenience. In a clinical trial analysis, a genomics study, or a climate model interpretation, an unfaithful explanation can corrupt the scientific record.

Consider a concrete example: a machine learning model trained to classify protein folding pathways assigns high importance to three specific amino acid residue positions. A researcher, trusting this explanation, designs a series of wet-lab experiments targeting those residues. The follow-up work — consuming significant resources and time — proceeds on a false premise. The model's actual decision boundary was shaped by a distributional artifact in the training data, but the faithfulness metric reported high confidence in the feature attribution. The explanation was not merely wrong; it was confidently, measurably wrong in a way that redirected scientific effort.

This scenario is not hypothetical. A 2022 analysis published in Nature Machine Intelligence found that SHAP explanations for several genomics classifiers were inconsistent across random seeds, suggesting that the explanations were capturing noise rather than signal. A 2023 study in PLOS Computational Biology demonstrated that LIME-generated explanations for medical image classifiers changed substantially with minor, semantically irrelevant perturbations to input images — a direct signature of faithfulness failure.

The pattern the position paper identifies — overreliance followed by algorithmic aversion — is particularly destructive in academic contexts. Researchers who initially trust an AI tool's explanations, then discover they are unreliable, do not typically recalibrate with nuance. They tend to abandon the tool entirely, contributing to a broader skepticism about AI research tools that may be unwarranted for methods unrelated to the failed explainability approach.

Implications for AI Peer Review and Automated Manuscript Analysis

This is where the analysis becomes directly relevant to the infrastructure of scientific publishing and validation. AI peer review systems and automated manuscript analysis platforms are increasingly called upon to assess not just the writing quality of a research paper, but the methodological soundness of the AI methods described within it. A paper that reports machine learning results and includes XAI-based interpretability analysis is now a common submission type across disciplines from computational biology to social science to materials science.

For an AI-powered peer review system to add genuine value in evaluating such manuscripts, it must be capable of flagging precisely the kind of structural issues the position paper identifies. This means going beyond surface-level checks — grammar, citation formatting, basic statistical reporting — toward substantive methodological critique: Has the authors' faithfulness metric been validated against the specific model class they are using? Are the feature importance scores reported with appropriate uncertainty bounds? Has the paper acknowledged the perturbation-distribution problem?

Platforms like PeerReviewerAI are designed to engage with this level of methodological depth, analyzing submitted manuscripts for consistency between claimed methods and reported results, and flagging areas where interpretability claims may outrun their evidential support. This is not a replacement for human domain expertise — the position paper itself is the product of careful human reasoning — but it represents a meaningful first-pass filter that can surface methodological red flags before a paper reaches an overextended human reviewer.

The broader point is that automated peer review is now implicated in the XAI debate whether its developers intend it or not. If AI manuscript review tools fail to interrogate explainability claims rigorously, they become vectors for propagating the very faithfulness failures the position paper describes. The standard for AI-powered peer review systems must therefore include domain-appropriate methodological scrutiny, not just linguistic fluency.

Practical Takeaways for Researchers Using AI Tools

For researchers working with machine learning models and XAI methods — whether as primary investigators or as peer reviewers — the position paper's findings suggest several concrete adjustments to standard practice.

1. Treat feature importance scores as hypotheses, not conclusions. A high SHAP value for a particular feature should trigger a targeted experimental or analytical follow-up, not a definitive interpretive claim. Report these values as one input to a broader inferential process.

2. Report metric disagreement explicitly. When multiple faithfulness metrics are available, compute them and report the variance. Convergence across metrics is modest evidence of reliability; divergence is a warning signal that belongs in the paper, not the supplementary materials.

3. Distinguish in-distribution from out-of-distribution evaluation. If the faithfulness assessment relies on perturbation methods, verify that the perturbed inputs remain within the empirical support of the training distribution. Methods such as the ROAR (Remove and Retrain) approach, which actually retrains models on reduced feature sets, provide more credible faithfulness estimates than perturbation alone, though they carry their own computational costs.

4. Separate predictive performance claims from interpretability claims. A model can achieve high predictive accuracy while its explanations are entirely unreliable. These are orthogonal properties and should be reported and reviewed as such. Conflating them — as many papers implicitly do — is a methodological error that AI peer review tools are increasingly capable of detecting.

5. Engage with the replication literature before citing established XAI tools as validated. The position paper notes that the evidence base for XAI faithfulness has been accumulating for a decade. Researchers should conduct a targeted literature review of faithfulness evaluations specific to their model class and domain before presenting XAI outputs as authoritative.

6. Consider adversarial testing of your own explanations. Tools exist — including Fooling LIME and SHAP (Slack et al., 2020) — that can systematically manipulate model explanations without altering predictive behavior. If your explanations can be fooled in this way, the faithfulness of your original explanations is suspect.

For researchers submitting to journals with rigorous AI methodology standards, running manuscripts through an automated research paper analysis tool such as PeerReviewerAI before submission can help identify where interpretability claims need additional qualification or where faithfulness evaluation is absent entirely.

The Broader Pattern: When AI Tools Validate Themselves

The position paper raises a concern that extends well beyond XAI specifically: the tendency of AI research infrastructure to develop self-referential validation loops. Faithfulness metrics are used to validate explanation methods; explanation methods are used to interpret model behavior; model behavior is assessed using benchmarks that were themselves constructed with the assistance of machine learning tools. At each step, the chain of validation depends on assumptions that are rarely made explicit and even more rarely tested empirically.

This is not a counsel of despair. It is a description of a known failure mode in complex systems that has a known remedy: external, adversarial, and cross-disciplinary validation. Peer review — whether human or AI-assisted — functions as one such external check. The value of rigorous automated manuscript analysis lies precisely in its capacity to apply consistent, documented methodological standards across submissions, rather than relying on the variable expertise and attention of overloaded reviewers.

The scientific community has navigated analogous crises before. The replication crisis in psychology, which peaked in visibility around 2015, produced lasting improvements in pre-registration standards, effect size reporting, and open data practices. The XAI faithfulness problem is, in structural terms, a similar challenge: a set of widely adopted methods whose validation infrastructure is weaker than the confidence placed in them. The response, similarly, will require both technical innovation — better faithfulness metrics, more rigorous benchmarks — and institutional adaptation, including the integration of XAI-specific methodological checks into the peer review process.

Conclusion: AI Peer Review as a Structural Response to AI's Own Limitations

The position paper on XAI faithfulness metrics is, at its core, an argument about the gap between representation and reality — between what an AI tool claims to show and what it actually reveals. This gap is consequential precisely because scientific reasoning depends on the integrity of that chain from observation to interpretation to conclusion.

AI peer review, properly implemented, occupies an important structural position in closing this gap. By systematically interrogating whether interpretability claims in submitted manuscripts are supported by appropriate validation evidence, AI-powered peer review systems can help prevent the propagation of faithfulness failures into the peer-reviewed literature. This is not a task that requires AI to be infallible — it requires AI to be systematic and transparent in ways that complement human expertise.

For researchers, the practical message is one of disciplined skepticism: toward the explanations generated by your own models, toward the faithfulness metrics used to evaluate those explanations, and toward the assumption that interpretability and reliability are synonymous. The decade of evidence reviewed in arXiv:2610.12493 suggests they are not. Treating that finding seriously — in how we conduct research, how we write manuscripts, and how we engage with automated manuscript analysis tools — is the appropriate scientific response to a problem that is now too well-documented to defer.