When Formatting Changes Everything: What the Format Sensitivity Index Means for AI Peer Review and Research Validation

The Hidden Variable That's Quietly Distorting AI Benchmarks

Imagine two researchers independently evaluating the same large language model for use in their laboratory's automated manuscript analysis pipeline. They run identical tasks, use identical datasets, and yet arrive at meaningfully different conclusions about the model's reliability — not because of any flaw in their experimental design, but because one wrapped their prompts in a JSON schema and the other used plain text. This scenario, which many practitioners have quietly suspected for years, has now been rigorously quantified. A new preprint from arXiv (2607.09665) introduces the Format Sensitivity Index (FSI) and the Parseability Sensitivity Index (PSI) — two metrics that expose just how dramatically prompt formatting alone can alter the apparent performance of language models. For the scientific community, and particularly for those building or relying on AI peer review systems, the implications are both significant and immediately actionable.

What the Research Actually Found

The study's methodology is worth examining carefully, because the scale and control of the experiment lend its findings unusual credibility. The research team generated 140,000 outputs via OpenRouter, spanning seven distinct question-answering tasks and multiple models. Crucially, they held token count constant across prompt variants — meaning any observed performance differences cannot be attributed to information density or prompt length. The only variable was formatting: how the prompt wrapper was structured, whether it imposed a schema, and how it signaled the expected response format.

The Format Sensitivity Index, as defined in the paper, measures the accuracy range induced purely by wrapper choice — in other words, the spread between a model's best and worst performance on the same underlying task when only the formatting changes. The Parseability Sensitivity Index captures the corresponding variance in how reliably a model's output can be machine-parsed, which is a distinct but equally consequential concern in automated pipelines.

What the numbers reveal is striking. Wrapper choice alone was sufficient to flip leaderboard rankings — meaning that a model appearing superior under one formatting convention appeared inferior under another. This is not a marginal effect at the decimal place. These are rank-order reversals of the kind that determine which models get deployed in production systems, which get cited in methods sections, and which get recommended in systematic reviews of AI research tools.

Why This Problem Is Especially Acute in Scientific AI Applications

General-purpose benchmarking errors are a nuisance. In scientific contexts, they carry a higher cost. When researchers and institutions select AI systems for tasks like automated manuscript analysis, literature synthesis, statistical result extraction, or AI paper review, they typically rely on published benchmark comparisons. A researcher selecting an NLP tool to assist with screening submissions for a journal, or a graduate student choosing an AI research assistant to support their systematic literature review, is implicitly trusting that those benchmarks reflect something real about model capability.

The FSI/PSI framework demonstrates that this trust may be misplaced — not because the benchmarks are fraudulent, but because they are format-contingent in ways that are rarely disclosed. Most benchmark leaderboards report a single score per model per task. They do not report the variance across prompt wrappers. They do not disclose whether the winning model was tested with a schema that happened to align with its training distribution, while competitors were tested under less favorable formatting conditions.

This matters with particular acuteness for AI in academia because scientific workflows are deeply heterogeneous. A platform designed for automated peer review may interact with submitted manuscripts in PDF, LaTeX, DOCX, or plain text formats. The structured fields of a chemistry paper differ fundamentally from those of a clinical trial report or a computational linguistics dissertation. If the underlying model's apparent competence is sensitive to schema formatting — and the new research confirms that it is — then the performance a research institution observes in deployment may diverge substantially from what they saw in pre-deployment benchmarking.

Implications for AI-Powered Peer Review Systems

The peer review process is, at its core, a structured evaluation task. It requires a system to parse a document, identify claims, assess the adequacy of evidence, flag methodological concerns, and produce structured feedback. Every one of these sub-tasks involves a formatted prompt and an expected output schema. This means AI peer review tools are directly exposed to the vulnerabilities the FSI/PSI research has now quantified.

Consider a concrete scenario: an automated peer review platform uses a prompt wrapper that asks a model to return structured JSON containing fields for "methodological_soundness," "novelty_assessment," and "statistical_rigor." The model's behavior under this schema may differ substantially from its behavior when asked the same evaluative questions in free-text form. If the platform was benchmarked using one formatting convention and deployed using another — even a subtly different one — the PSI variance means that parseability could degrade, and the FSI variance means that the model's apparent analytical depth could shift.

Platforms like PeerReviewerAI (https://aipeerreviewer.com), which apply automated analysis to research papers, theses, and dissertations, operate precisely at this intersection of structured prompting and scientific judgment. The responsible response to research like the FSI/PSI study is to treat format sensitivity as a first-class engineering concern: to test models across multiple wrapper configurations before deployment, to monitor output parseability in production, and to report confidence intervals rather than point estimates when presenting AI-generated evaluations to researchers.

More broadly, the field of AI scholarly publishing needs to adopt a norm of wrapper transparency. When a paper reports that "GPT-X achieved 87% accuracy on our review quality benchmark," that number should be accompanied by the FSI — the range of accuracies observed across the prompt wrapper variants tested. Without this, the number is not merely incomplete; it is potentially misleading in ways that compound downstream.

Practical Takeaways for Researchers Using AI Tools

For researchers who rely on AI research assistants, automated manuscript analysis tools, or machine learning systems for scientific literature work, the FSI/PSI findings suggest several concrete adjustments to practice.

Test Your Prompts, Not Just Your Models

When evaluating an AI research tool for your workflow, do not evaluate it under a single prompt configuration. If you are using an AI paper review system to assess submissions, run the same documents through at least three structurally distinct prompt wrappers and compare outputs. Substantial divergence — in the structured fields returned, in the tone and depth of qualitative feedback, in the flagging of statistical concerns — is evidence of high FSI, and should make you cautious about over-interpreting any single output.

Treat Schema Design as a Scientific Decision

In automated research validation pipelines, the schema you impose on model outputs is not a neutral formatting choice. It is an intervention that may systematically favor or disfavor certain model behaviors. Schema fields that are tightly constrained (e.g., a numeric score from 1 to 5) produce different error profiles than open-ended fields. Researchers building NLP pipelines for scientific paper analysis should document their schema choices as carefully as they document their statistical analysis choices — including the rationale and any sensitivity analyses performed.

Be Skeptical of Single-Configuration Benchmarks

When a vendor or published study claims that their AI research assistant achieves a particular performance level on a peer review or manuscript analysis benchmark, ask whether that benchmark was conducted under a single prompt configuration. If yes, request the variance data or conduct your own multi-wrapper evaluation. The FSI framework provides a principled vocabulary for making this request: ask for the Format Sensitivity Index across the task in question.

Interpret AI Outputs Probabilistically

One of the underappreciated implications of high FSI is that model outputs should be treated as samples from a distribution, not as deterministic assessments. When an AI tool tells you that a manuscript's statistical methods are "adequate" or "insufficient," that judgment is partly a function of how the question was posed. Maintaining human oversight in AI-assisted peer review is not merely an ethical safeguard; it is a statistical necessity given what we now know about format sensitivity.

The Broader Problem of Construct Validity in AI Research Benchmarks

The FSI/PSI study fits into a wider pattern of research exposing construct validity problems in how AI systems are evaluated. Benchmark contamination, task leakage into pretraining data, sensitivity to few-shot example ordering, and now wrapper-induced score variance — each of these findings individually seems like a technical caveat. Taken together, they amount to a structural critique of how the field has been measuring model capability.

For AI in academia specifically, this has a sobering implication. Many of the claims about AI's capacity to assist with scientific tasks — literature review, methodology critique, statistical analysis, originality detection — rest on benchmark results that may carry undisclosed format sensitivity. The researcher who deploys an AI research assistant based on published performance figures is making a decision under greater uncertainty than those figures suggest.

This does not mean that AI tools for scientific research are unreliable. It means that the evidence base for their reliability needs to be strengthened, and that the research community — including those building tools like automated peer review platforms — bears responsibility for generating more rigorous, format-transparent evaluations.

Looking Forward: Toward Robust AI Peer Review

The introduction of the Format Sensitivity Index is a methodological contribution that the scientific AI community should adopt with some urgency. As AI peer review systems become more embedded in editorial workflows — from preprint screening to journal submission management to dissertation assessment — the standards for evaluating those systems must keep pace with their deployment.

The path forward involves several parallel efforts. Benchmark developers should begin reporting FSI and PSI alongside accuracy metrics as a matter of standard practice. Platform developers building AI-powered peer review systems should conduct internal wrapper robustness audits before deployment and communicate their results to institutional clients. Researchers using AI research tools should develop the prompting literacy to understand that their outputs are schema-contingent, and calibrate their interpretive confidence accordingly.

Ultimately, what the FSI/PSI research demonstrates is that rigor in AI evaluation requires the same attention to confounding variables that good experimental science has always required. Formatting is a confound. It has always been present in AI benchmarking. We now have a metric to quantify it. The question for the scientific community is whether we will integrate that metric into our standards, or continue to report point estimates and call them performance.

For AI peer review to earn genuine epistemic trust in the scientific process — not as a replacement for expert judgment, but as a reliable analytical layer — it must be held to the same standards of reproducibility and transparency that we demand of the research it evaluates. The FSI/PSI framework gives us one more precise tool for doing exactly that.