Reproducibility audits and journal sanctions both claim to police the scientific record, yet they almost never agree on which papers should disappear. The two tracks ask different questions, run on different clocks, and answer to different institutions. This piece explains how each is actually produced, using a psychology replication block as the worked example, and why the gap between them is structural rather than a sign of bad faith on either side.

Two Quarries, One Reproducibility Question

An audit tests a claim: does this effect reappear when a different team runs the same procedure? A sanction tests conduct: did a named author fabricate, plagiarise, or duplicate an image? The distinction sounds academic until you watch the two processes collide over the same paper. An audit can fail a claim without alleging that anyone lied.

That difference sets the evidence threshold. Audits need a protocol, materials, and an analysis plan agreed in advance. Sanctions need an allegation specific enough to investigate and a finding specific enough to defend in writing. A replication that lands at half the original effect size is informative to an auditor. It is close to useless to a committee deciding whether a researcher should keep a job.

Retraction reach, meaning how far a notice travels and how many citing papers it disturbs, depends on which track moves first. When an audit fires, journals often wait. When a committee fires, journals act quickly because the finding arrives pre-packaged. The gap is structural, not moral.

How a Reproducibility Audit Actually Runs

A large audit starts with preregistration. The coordinating team writes down the sampling frame, the target effects, the analysis code, and the criterion for success before any data arrive. That document is the audit's spine. Without it, a failed replication is just a disagreement between two labs with different equipment and different graduate students.

Many labs then run the same materials on independent samples, often with the original authors consulted on procedure. Success is usually defined conservatively: an effect in the same direction, sometimes with a significance test, sometimes with a confidence interval that excludes zero. The Open Science Collaboration's 2015 project in Science is the canonical example. It coordinated 270 authors across 100 published psychology studies and reported that 36 percent produced significant results on replication, against 97 percent in the originals.

That headline number hides the machinery. Each replication was its own small study with its own preregistered plan, its own sample, and its own write-up. The aggregate figure is a summary of a hundred separate judgement calls. This site has covered how the same logic plays out elsewhere, including priming studies that survive contact only where labs recorded their negative results.

Sanctions Move on a Slower Clock

Journal sanctions begin with a report to an editor, not a table of effect sizes. An editor reads the allegation, checks the figures, and may ask the authors for raw data. If the answer is unsatisfactory, the case can go to the authors' institution, where a research integrity committee takes over. Committee inquiries routinely take years, and their findings are often confidential.

Retraction notices reflect what can be stated publicly and defended legally. Image duplication is the most common technical ground because it is visible and hard to explain away. Fabrication findings are rarer in notices because they require access to notebooks and data that committees hold. Retraction Watch has logged tens of thousands of notices since it began tracking them, and the annual count has climbed steadily, though the totals mix honest error with misconduct.

The clock matters more than the volume. A committee report can surface four or five years after the allegation. By then, the paper may have been cited hundreds of times, and the citation network has its own momentum.

Case Study: A Psychology Replication Block

The 2015 Open Science Collaboration block is the cleanest worked example available. The team drew 100 studies from three psychology journals, covering social and cognitive psychology, and required each replication to follow the original procedure as closely as the original reporting allowed. Where the original paper omitted a detail, the replication team made a judgement call and documented it.

Effect sizes shrank. Across the set, the mean replicated effect was roughly half the original magnitude. Significance told a similar story: about 39 percent of the replications reached the conventional threshold, and the 36 percent figure counts only those that also matched the original direction and criterion. Subjective ratings by the replication teams suggested that 39 percent of the original results had been reproduced in the broad sense.

No retractions followed from the audit alone. That absence is the point. The audit produced a distribution, not a verdict on any single paper. Some original authors disputed the replication protocols, which is a legitimate move when the original reporting was thin.

Why Reach Diverges at the Journal Desk

Audits describe distributions. Sanctions require a named responsible party, and a failed replication does not identify one. An effect can fail to replicate because the original was a fluke, because the replication was underpowered, because the sample differed, or because the original analysis was selective. Only the last of those is a conduct question, and an audit cannot distinguish it from the others.

Editors weigh defamation risk and legal exposure before issuing a notice. A retraction is a public statement that a published result cannot be relied upon, and authors can and do contest it. When the underlying evidence is a statistical disagreement rather than a document trail, the editor's position is weak. The related problem shows up in lab-based work, where two pipette calibrations can decide whether a finding reproduces at all.

Authors can dispute replication methods, and sometimes they are right. A replication that changes the population, the instructions, or the timing has tested something adjacent to the original claim. That objection is a real cost of the audit model, and it is why audit results are best read as evidence about a literature rather than about a person.

The Cost of Running Both Tracks

Money shapes which claims get audited. A multi-lab replication with dozens of sites can cost hundreds of thousands of dollars in labour and coordination, and funders rarely foot that bill without a reason. The result is a selection effect: high-profile claims in crowded fields attract replication attempts, while quieter findings in smaller literatures go untested.

Sanctions carry their own costs. A committee inquiry consumes staff time, legal review, and institutional reputation, and the process can stretch over years. Universities sometimes settle quietly to avoid publicity, which leaves the published record untouched. The two tracks therefore operate under different budget constraints, and those constraints determine which papers get scrutinised at all.

There is also a trade-off in transparency. Audits that publish their protocols and data allow outsiders to check the checkers, but they expose the replication team to criticism. Sanction proceedings that stay confidential protect the accused until a finding is reached, but they also keep the evidence out of public view. Neither arrangement is obviously superior; each buys something at a cost.

What a Careful Reader Can Do

Check whether a claim has an independent replication before treating it as settled. A single study, however large, is one draw from a noisy process, and the citation record will not tell you whether anyone tried again. The Open Science Collaboration dataset is public and searchable.

Look for preregistration and shared data when you read a paper you intend to rely on. A registered analysis plan posted before data collection is the strongest available signal that the reported test was the test the authors intended. Absence of a plan is not proof of anything, but it lowers how much weight the result can carry.

• Distinguish a failed replication from misconduct; they are different findings with different evidentiary standards.
• Read the retraction notice for its stated reason rather than assuming fraud.
• Track whether a correction propagates into reviews and textbooks, since the original citation often outlives the notice.
• Note the funding and institutional home of any audit, as this site has argued in work on rater expectations, because who pays for the replication shapes which claims get tested.

The two tracks will keep diverging as long as audits produce distributions and sanctions require named fault. A reader who wants to know what to trust has to read both, and read them for what each one can actually establish.