0 · Disclaimer: How AI Assistance Was Used

In order to avoid accuastions of this "just being Claude defending Claude," "AI slop," "probably all hallucinations," etc., I've decided it's probably worth explaining a few key points about how this report was created:

• All metrics, methodology, and data sources were exclusively chosen by me, in consultation with my wife, who has a Master's Degree in Statistics from Penn State University.
• The methodology is directly based on my wife's input: she was the one that pointed out that trying to just compare bugs per ten lines of code before and after would likely be too effected by noise because of the low number of post-Claude samples, and that, for similar reasons, trying to build some kind of linear regression model to ascertain the relative effects of different variables would probably also not work. She specifically told me that looking at where the post-Claude releases fall into the historical distribution, and how likely from the historical distribution we would be to get releases as "bad" or worse than the post-Claude releases, was probably the best that could be done.
• I spent several days on this, two before even creating the GitHub repo and had at least one major total rewrite of the report to use a better methodology (given the feedback from my wife mentioned above). This was a lot of manual, cognitive effort on my end.
• The scripts used to fetch the data, collate it into a DuckDB database file, construct the views on that DB, and then do the statistical analysis on that data, were indeed written by GLM 5.1, as was the HTML and much of the original prose for the final report webpage you're looking at right now.
• Crucially, however, all numbers, statistics, cards, and graphs in this report are automatically templated in directly by the Python script that ran the statistical analysis, thus avoiding any possibility of hallucinations or inconsistencies in the numbers.
• After posting this on Hacker News and recieving almost no substantive input, discussion, or response on the actual content of the article, I decided to rewrite all of the prose in my own voice. If anyone complains about my verbosity or sentence structure — as they usually do, which is the reason I originally let the AI write the prose, among other reasons obsoleted by templating — they can go fuck themselves.
• If you want to replicate the data and results here, and inspect exactly how they were calculated, you can find the repository here. I have purposefully made it so that the pipeline can be run end to end completely from scratch, so you can see the entire pipeline end-to end, with no mysterious DB blobs forcing you to trust that I didn't doctor or screw up the data. If you want to be mad about the numbers, look there first.

1 · Background: The rsync Outrage

In late May 2026, rsync blew up. First, an evidence-free Mastodon post was made pointing to a spurious correlation between a regression that particular user experienced upon upgrading to a release, and that release having Claude commits in it. It was viewed an unknown number of times, but even likes and boosts passed the thousands mark handily, and it gained significant traction — as all spurious anti-AI hate does —, seeing 58 replies from 32 unique users. Someone rages about "cognitive surrender" with no evidence; another suggests adding rsync to the famous open-slopware blacklist. From there, it spread to Hacker News, with 81 comments, full of mixed dread, anger, and crowing about how this finally proves once and for all no one can use LLMs safely. Among all that was one particular comment which spurred further the view that the regressions and bugs were caused by Claude.

This On May 30, 2026, this burgeoning outrage emergently coalesced into a single focal point: a GitHub issue titled "Please Do Not Vibe Fuck Up This Software", opened against the rsync repository. It attached a screenshot of the Mastodon post criticizing the project's use of Claude. That's it. No bug report, no technical content, no attempt to actually ascertain if the concern was real or justified; just 350+ comments ranging from thoughtful concern to outright harassment (most of the most egregious, unreasonable, and outright violent comments have since been deleted; few thought to preserve them).

The thread did not stop at words. It eventually escalated to, at one point, visual depictions of fantasies of violence, when one user posted a now deleted comment including My Little Pony drawings of themselves strangling the "project janitor that pushed vibecoded commits":

Completing the internet outrage cycle, this issue in turn spread to Hacker News, generating hundreds more comments. Some attempted to point at the number of regressions after the introduction of Claude — "The Linux Mint Timeshift tool has an issue open documenting a number of regressions that are currently open on the rsync issues page, that were only introduced post-vibecoding" — as evidence that it was worse. Others pointed out that those regressions were not caused by Claude, and in response, the goalposts were moved again. Over and over, the core theme was one central claim, repeated everywhere: Claude-assisted development introduced bugs into a previously stable tool. AI is cognitive surrender, is cocaine, is loss of craft, and the users are right to be angry as a result:

People are very justifiably angry that a very stable, well trusted tool, has started to immediately go downhill… all because the main dev is vibecoding that software.

— fao_ on Hacker News

However, this isn't doesn't have to be a question solved only on the basis of — ironically — vibes. This is something that could be, at least to a degree, empirically tested. Some even pointed that out:

On Lobste.rs, in response to the Medium essay Tridge himself posted in response, finally some users like boramalper begin to actually ask for evidence one way or another:

It'd be interesting if someone actually did a timechart of regressions after each release (if at all possible) to see if the number actually went up recently or not.

— boramalper on Lobsters

User bitshift replied: "I would also love to see such a chart. It wouldn't be completely informative… But at least it would be something objective we could measure."

This analysis is that chart. Or, well, as best as it can be made, given the limitations of the data (see the previous section).

2 · Executive Summary

• 36 releases with bug data, spanning v2.4.6 to v3.4.3
• 2 releases have Claude commits: v3.4.2 (9 Claude, 0.00 sev/10c) and v3.4.3 (28 Claude, 3.29 sev/10c)
• The Claude releases bracket the IQR in opposite directions: v3.4.2 is below the IQR, v3.4.3 is above it. Neither is an outlier.
• Exact permutation test p-value = 46%: pick any 2 releases at random, you'd score as bad or worse 46% of the time. This is the strongest available test and it finds nothing.
• Fisher's exact test p-value = 74%: Claude releases are no more likely to fall above the historical median than any other releases (odds ratio 1.06).
• The historical mean is 1.8× the Claude mean (2.95 vs 1.65 sev/10c)
• v3.4.1 (59 bugs / 9 commits, no Claude) is an outlier but belongs in the baseline — it is a release, and the distribution already captures it

3 · The Metric

The analysis uses a single metric: severity-weighted bugs per 10 commits (sev/10c). Each bug is normalized to a 0–1 severity score (its LLM-assigned severity divided by 100), and those scores are summed per release instead of simply counting bugs. The raw bug count is also shown in the table for reference, but sev/10c drives all statistical tests.

How commits are assigned to releases

Every commit on the default branch was ordered by committer date to produce a sequential timeline. Each git tag points to a specific commit in this timeline. A release's range is all commits between the previous tag and its own tag. Pre-release tags ("pre", "rc") are skipped as boundaries and absorbed into their final release. Every commit belongs to exactly one release.

How bugs are found and assigned to releases

Bug reports come from three sources:

1. GitHub issues in the rsync repository (collated via the GitHub REST API),
2. the rsync Bugzilla instance (collected via the API),
3. and the rsync mailing list.

GitHub issues and mailing-list bugs are attributed to the most recent release that shipped before the bug was reported. For Bugzilla, each entry has a "Version" field that explicitly states which release the bug was reported against, and bugs are attributed to that release.

Severity scoring

To control for bug severity — so that, as someone on HN said, a typo in a button and a CVE aren't rated equally — every bug report was scored for severity on a 0–100 scale. The scorer is Qwen 3 35B, a small open-weight language model, prompted as a senior reliability engineer assessing real-world impact. Each bug report was given to the model as its title and body text (truncated to 3,000 characters), along with the following rubric:

All three bug sources — GitHub issues, Bugzilla, and the rsync mailing list — were scored. Bugzilla and mailing list reports had only a title (no body), so the model scored those from the title alone. The model was instructed to fall back on the title and lean toward the middle of the range (40–60) when the body didn't provide enough information. The model was also told to output only a severity integer via structured output (JSON schema), so there were no free-text responses to parse. Scoring was done at temperature 0 for determinism — the same input always produces the same score.

Issues scored severity 0 — feature requests, spam, off-topic rants about AI, empty submissions — are excluded from bug counts by default. This matters because some releases attracted a lot of noise on GitHub. v3.4.2 had four issues filed; the model scored all four at severity 0 (a feature-request option, a missing tarball question, and two more feature requests).

Example scores from the database, one per tier:

Why the release is the unit of analysis

Why group commits by release, bugs by release, and then ascertain the correlation — or lack thereof — between Claude commits and bugs through the intermediary of releases? This is for two reasons.

First, because the claim that the critics are making is also, itself, made in terms of releases: that having any Claude commits in a release makes the whole release more buggy as a whole in a noticeable way, not just that Claude-authored commits may introduce more bugs; the latter is a different metric, because later Claude- or human-authored commits could correct for those bugs within the same release, and nobody would then notice as part of the release, and overall it wouldn't matter to users; additionally, it's simply important, as stated elsewhere, to meet the claim of the critics where it's at. If this forces them to make their claims more nuanced — or otherwise move the goalposts — then mission accomplished.

Second, it's a problem of attribution: the vast, vast majority of bugs do not state exactly which commit caused them, because doing so would require extensive research and analysis that is often not worth it in favor of simply fixing-forward, and even if that analysis was done — via something like git bisect — it wouldn't necessarily result in anything useful, or anything at all. Many bugs can result from a combination of multiple commits, often separated significantly over time, where it's unclear whether one commit or the other really introduced the bug. Or, one commit can reveal several latent bugs introduced by other commits at once, and so on.

Why bugs and commits?

The critics' claim is simplistic, absolute, and universalistic: the rate of bugs in the Claude-exposed releases went up. Therefore, the simplest honest response is to analyize precisely what is being claimed: bugs, commits, releases, and Claude-exposed commits. If the Claude releases sit in the middle of the historical distribution, the burden shifts to the critics to explain why this particular middle is somehow worse than all the other middles that came before it. Even by weighting by severity, I feel that I am giving extensive generosity to the anti-AI point in all this, but enough of the more intelligent critics brought it up that I found it worth it.

Even if that results in is shifting the conversation toward a more nuanced discussion of the quality and type and user impact of the bugs in the releases, it will already have been a major win for the pro-AI crowd, and a shifting of the goalposts for the anti-AI crowd, and then we can do further analysis based on that. And the ball's in the anti-AI court for that game.

What this approach does not do

I'm aware that this metric does not control for commit complexity or security intensity. It is a blunt instrument. But the critics' accusation is also blunt: "Claude is making things worse." A blunt instrument is what is required in response. Blood begets blood.

4 · Results

Claude Releases

Before we jump into deeper analysis, let's just look at the two Claude releases themselves, to get a sense for them:

If that doesn't look like a red flag to you, you'd be right.

Exact Permutation Test

So the question is: are the Claude releases unusually buggy, or could you easily pull a group just as bad out of the historical distribution by dumb luck? The way you answer that question statistically is an exact permutation test, which just enumerates all pairs of two releases and asks: what fraction have a mean bug rate as bad or worse than the one we actually observed? That fraction is the p-value of the hypothesis under test.

What this p-value tells us is that the hypothesis that Claude makes releases worse has, at least so far, about as much predictive power as a coin flip: if you closed your eyes and picked 2 releases at random, you'd do as bad or worse nearly half the time. There's nothing unusual about the Claude group.

Fisher's Exact Test

The permutation test asks: how likely is it that a random group of releases scores as badly as the Claude group? But there's another way to pose the question: are Claude releases more likely than non-Claude releases to fall above the historical median? That's a textbook 2×2 contingency table, and the standard test for it is Fisher's exact test.

To emphasize, this does not mean that all Claude releases in the future will not be more buggy. We don't have nearly enough data to build a model and extrapolate out like that, and that's not what a Fisher's exact test is for. The point that's being made here is that these specific releases are not at all notable; if no one had known they were AI, no one would have cared or noticed anything out of the ordinary, and there is no evidence with which to conclude that Claude made anything worse yet, unlike the objective, absolutist, universal claims made by critics.

The Distribution

In case you're not convinced, here's a visual aid, showing where these releases fall in the distribution of all prior releases:

How to read this graph: Each dot is a release. The shaded green band is the interquartile range — the middle 50% of historical releases, from 0.29 to 2.59 sev/10c. The darker regions on either side are the lower and upper quarters.

This is another way of saying the same thing the previous two tests said, but more intuitively: the Claude releases (green dots) bracket the IQR in opposite directions. v3.4.2, with zero real bugs, sits just below the IQR; v3.4.3 sits just above it. They bracket the middle of the distribution in opposite directions. Neither is a negative outlier, and since they're on either side of the IQR, there's no evidence Claude usage bends releases in either direction.

Commit Rate

One possible objection I've seen is that while perhaps the defect rate of Claude-authored commits is not worse than human-authored ones, Claude sped up developmend so much — due to "vibe coding" perhaps — that the total number of bugs in each release got too high for comfort anyway, in which case it doesn't matter that the defect rate per commit isn't so bad, because that's not what downstream users experience. We can check this, though:

So it seems that Claude releases did not have meaningfully more commits. If anything, they had a lot fewer, but of course, the data is underpowered to say that for sure; just like the rest of this article, I'm only trying to show a startling lack of evidence for the supposed massive harms of using Claude, not trying to prove there must have been none, or that it was even "beneficial" in terms of focused releases.

But commit count is a blunt measure. A commit can be a one-line typo fix or a rewrite of the entire protocol handler. What about lines of code changed? GitHub's compare API gives us additions and deletions between each pair of consecutive tags:

Lines changed went up by a lot. But did absolute bug numbers?

So: the Claude releases changed way more lines of code than historical ones, but didn't have more bugs. More code, same bugs. That's not what you'd expect if Claude were making things worse.

Regime Check

The obvious counterargument is that maybe earlier rsync releases were less in maintenence-mode, and so had more bugs, but recent rsync releases have been more stable, so comparing the two Claude-exposed releases to the full historical distribution is masking the fact that they're actually outliers for their regime. Luckily, there's a way to test this statistically.

Yes, the historical mean (2.95 sev/10c) was driven by a bimodal distribution: v2.x releases average 1.11 sev/10c; v3.x releases average 4.23. But even within the v3.x regime, the Claude releases sit in the middle of the pack or better:

So the regime-shift argument doesn't just fail — it fails backwards. The v3.x era has a much higher mean sev/10c than v2.x. If you restrict the comparison to v3.x only, Claude releases don't stand out at all — and one of them is better than most. The only way to make Claude look like an outlier is to compare it against a quieter era and then blame the shift on Claude, when the data says the shift predates Claude entirely.

We can further test whether there are meaningfully different regimes in the version history, and thus whether using the full historical data is valid, by doing a runs test. If such regimes existed, the runs test would detect non-random clustering:

The Pre-Claude Outlier

Here's my favorite part, though. Digging into the data, one of the first things that jumped out at me with blinding clarity was that the worst release, by far, in rsync history was entirely prior to the introduction of Claude:

And yet nobody noticed. There was no AI to blame so there was no GitHub issue with 300 comments, no death threats, no threats to fork or move to openrsync. A maintainer shipped a broken release and fixed it, just like normal. The only thing that made v3.4.3 special was the availability of an enemy everyone had already decided to hate.

All Releases (chronological)

5 · What the Data Is Consistent And Inconsistent With