The corpus: self-described, not curated
We collected public GitHub repos that describe themselves as AI- or vibe-coded, with the term appearing in their name, description, or README. The first pass in June 2026 produced 107 candidates; we expanded it in July 2026 to 577 and didn't curate beyond that. 28 were skipped per engine (archive over 40MB compressed, or repo not reachable), leaving n = 549 analyzed.
Self-description is both the framing and the limitation: we took repos at their word. Some are substantial applications. Some are workshop demos and single-page toys, and 85 of the 549 have 15 or fewer scannable files, with 29 at 5 or fewer (the median repo has 69). We counted them all, because they are self-described vibe-coded output, and we report means and distributions alongside medians for the same reason: tiny repos pull the medians up.
What ran: the rules engine, with the AI pass off
Each repo's default-branch archive was fetched in memory by quality·vibes's own fetcher, the same path a normal scan uses, with nothing cloned to disk and a 90-second per-repo timeout. The quality·vibes rules engine then ran its full check set across the six categories (secrets, injection, auth, data exposure, dependencies, transport) with production thresholds, unmodified for the study.
The Claude review that's part of every real quality·vibes scan was switched off, so every number in the study is deterministic and reproducible: same repo in, same findings out. That cuts both ways. Anything needing code-reading judgment went uncounted, so the study's findings are a floor on what a full scan reports, and scores would likely shift down with the AI pass on. Scores use the production model: six weighted category subscores, severity deductions, weighted average, letter grades.
Caps, floors, and the caveats that matter
The engine caps reporting at 10 findings per rule and 300 per repo. For aggregate stats this matters: per-repo finding counts (mean 5.4, median 2) are floors rather than totals for the messiest repos. Repo-level hit rates, meaning "x% of repos had at least one finding of type y", are unaffected by the caps, which is one reason we lead with them.
The rest of the list. This is heuristic static analysis rather than a pentest: no code execution, no vulnerability database, so the study measures pattern-detectable issues only. "Vibe-coded" is self-description; we didn't verify provenance. And it's one snapshot, collected July 2026. Repos change, and a different month would produce somewhat different numbers. We'd rather you know all of that and trust what survives it: the hit rates are clean, the grade distribution is real, and every figure traces to a per-repo result row.
how it works
- 01
Collect candidates
Search public GitHub for repos that self-describe as AI- or vibe-coded in name, description, or README. 577 candidates (107 from June 2026 + 470 added July 2026), no further curation.
- 02
Fetch each repo the way a scan does
Default-branch archive, fetched in memory by quality·vibes's own fetcher with nothing written to disk, and a 90-second per-repo timeout. 28 skipped per engine (too large or not reachable), leaving n = 549.
- 03
Run the rules engine, unmodified
The production quality·vibes check set across all six categories, production thresholds and severity model, AI pass off, so every result is deterministic and reproducible.
- 04
Score with the production model
Six weighted category subscores, severity deductions (critical −40, high −22, medium −10, low −4), weighted average, letter grades, identical to what a real scan reports.
- 05
Aggregate per repo, report distributions
Hit rates as share-of-repos-with-≥1-finding (immune to the per-rule caps), plus means, medians, and the full grade distribution, with the tiny-repo inflation of the medians stated rather than hidden.
frequently asked
Why publish the methodology at all?
Because the numbers are only worth citing if you can see how they were made. Corpus selection, what ran, what was off, and where the floors are: it's all here, and the headline stats survive the caveats.
Why was the AI pass switched off?
Reproducibility. Rules-only results are deterministic, same repo in and same findings out, which makes the study auditable. It also keeps the numbers conservative, since a full quality·vibes scan adds a Claude review on top and that finds more.
Did you filter out the tiny demo repos?
No. They self-describe as vibe-coded, so they're legitimately in the population. We do flag that 85 of 549 have ≤15 files and trivially score A, which inflates the medians, and that's why the report leads with means, grade distribution, and per-category hit rates instead.
Will you re-run the study?
The setup makes re-runs cheap, same engine against a fresh set, so a periodic re-run is the plan. The July 2026 snapshot is version one; if the numbers move, the comparison gets its own write-up.
Last updated June 11, 2026