The CI that was testing nothing

A while back I opened the Actions tab on a vibe coded project and found a column of red going back further than I wanted to admit. Not the scary kind of red, the ignorable kind: it failed at "Install dependencies," the very first step, before a single test ran. I had been reading that as noise. A dependency thing. I would get to it. What I had actually built, without meaning to, was a pipeline that checked nothing on every push and reported back like it was doing its job.

If your CI passes locally but fails in CI, the fix starts with a colder question than "how do I make it green." Ask what has actually been running. In my case the honest answer was nothing: the install step was exiting before typecheck, lint, and the test suite, so for that whole stretch the gates on the main branch were decorative. The build was not lying to me so much as answering a smaller question than I thought I was asking. It was telling me install failed. I heard "some noise in CI." Those are very different sentences.

why does my build pass locally but fail in CI with no code change?

Because "my machine" and "the runner" are two different environments, and the thing that differs is almost never your code. It is the stuff around your code: the OS, the Node version, the environment variables, and the one that bit me here, the package manager itself. My CI job pinned the package manager to a floating major version. Floating means "give me the newest release in that major line," so the runner quietly upgraded to a newer minor than the one on my laptop. No commit. No changelog I read. The tool under my build changed while I was looking at my own code for the bug.

That newer line had changed how it handled native build scripts. It read a new approval block and ignored the older key my project was still relying on, and the approval block was sitting there full of placeholder values that a past me had never filled in. So the newer tool looked at my config, saw zero packages actually approved to run their install scripts, refused to build the two native packages that needed it, and exited with an error. The older tool on my laptop had never heard of the new block, fell back to the old key, and installed fine. Same repo, same lockfile, opposite result, entirely because of a version I had told the system it was allowed to drift.

what does it mean when the install step fails before the tests?

It means your test results are not just failing, they are absent, and absent is worse than failing because it is quiet. A failing test is a signal. A skipped test suite is a blank where the signal should be. When install dies first, the runner never reaches typecheck, never reaches lint, never reaches a single test, and the job goes red for a reason that has nothing to do with whether your code is correct. If you have trained yourself to skim past that particular red, which is easy to do when it looks like plumbing, you have effectively turned your whole gate off and left the light on.

This is the part I want any vibe coder to sit with, because it is the real lesson and it has nothing to do with any specific tool. A pipeline that fails before its checks run is not a weaker version of a working pipeline. It is a no-op wearing a pipeline's clothes. The presence of a CI file, green-ish history, and a bunch of yaml does not mean anything is being verified. The only thing that proves a gate works is watching it actually catch something, or at minimum watching it actually run.

how I actually fixed it

I did not trust my own guess about the cause, because the whole problem was that my guess ("it passes for me") was built in the wrong environment. So I reproduced it in an isolated tree and measured it against both versions of the package manager, the one from CI and the one from my laptop, with the placeholder config, with only the old key, and with real approvals. Four combinations, run, not reasoned about. Placeholder config on the new version reproduced the CI failure exactly. Only the old key on the new version still failed, which told me pinning the version alone would not have saved me. Real approvals passed on both versions, which told me what the actual fix was.

So the fix had two halves. First, fill in real approvals for the two native packages that genuinely need to run install scripts, and keep the older key too so anyone still on the older version resolves cleanly, with a comment in the file saying exactly why both are there. Second, and this is the half that matters beyond this one bug, pin the package manager to an exact version. Not the major line. The exact release. A floating version let a tool change my build's behavior with zero changes in my repo, and that is precisely the kind of invisible input a vibe coded project cannot afford, because you are already trusting a lot of generated glue you did not hand-write. Pin the toolchain like a dependency, the same way you would pin the model like a dependency, and bump it on purpose instead of by drift.

the shape of the lesson

There is a pattern I keep relearning: the surface that says "fine" and the reality that says "nothing ran" can look identical from the outside, and the only way through is to verify in the environment that actually does the work, not the one that is convenient to check. It is the same reason a thing can work in preview and break on deploy. Your dev box is a very convincing picture of production. It is not production. Your green-ish CI history is a very convincing picture of a working gate. It is not the gate. Go make the gate catch something, in the place it will actually run, and only then believe it.

questions that keep coming up

My CI is green. Doesn't that mean my tests pass? Not by itself. Green means the job exited zero. Confirm the test step actually ran and reported a count you recognize. A job that dies at install, or one where the test step was never wired in, can be green or red for reasons that never touched your assertions. Open the logs once and read down to where your tests report their numbers. If you cannot find that line, your tests did not run.

Should I pin exact versions or floating ones in CI? For the tools that shape your build, the language runtime, the package manager, the action versions, pin exact and bump on purpose. Floating is how a release you never read changes your build with no commit to blame. The convenience of automatic upgrades is not worth an input you cannot see. Your app dependencies live in a lockfile for the same reason; your toolchain deserves the same treatment.

Why did my install start refusing to run a package's build scripts? Modern package managers increasingly block install-time scripts by default and require you to approve the specific packages that need them, because a package that runs arbitrary code on install is a real supply-chain surface. When that approval config is missing or stale, a native package silently fails to build. That is the guardrail working, not a bug. Approve the few you actually trust, by name, rather than switching the protection off wholesale.

If you are wiring up automation or CI for a vibe coded project and want a second set of eyes on whether your gates are actually gating, /work-with-us. Send me the pipeline and the thing it is supposed to protect, and I will help you make the green mean something. Work with VibeKoded.

The red column was embarrassing to find. The blank one underneath it, the tests that had not run in longer than I would like, was the actual problem. Green is a claim. Watch the gate catch something before you believe it.