Do Smoke Tests Catch npm Package Errors? What They Miss and What to Run Instead

A common belief among engineering teams is that comprehensive smoke tests are sufficient to catch dependency errors caused by npm package usage. They are not. Smoke tests catch the case where a known code path existed and broke. They cannot catch the case where a new code path was added between releases and ships untested. The short version: smoke tests cover existing paths; deterministic static analysis of your npm dependency call sites covers new paths added between releases, which is where most package-related incidents originate.

Quick Answer: Smoke tests cover code paths that already exist. A deterministic static analysis tool covers code paths added between releases, where new misuse of an npm dependency most often ships. To check your TypeScript codebase against 280+ npm package Profiles in 30 seconds: npx nark

What smoke tests are good at

Smoke tests are not the problem. A well-maintained smoke test suite is a real asset. It:

• Catches obvious regressions on existing user flows
• Confirms the deploy pipeline works end-to-end
• Surfaces broken happy paths before customers do
• Costs little to maintain once written

If your team runs comprehensive end-to-end and smoke tests, you are already ahead of most shops. This article is not arguing against smoke tests. It is arguing that smoke tests cover one class of failure, and a different class is what ships package-related incidents to production.

The class of bug smoke tests will not catch

Smoke tests exercise the paths they were written to exercise. They cannot exercise paths the test author has not yet seen. Three concrete cases.

1. Code paths added between releases

A developer opens a PR that adds a new feature:

// src/api/recommendations.ts (NEW)
import axios from 'axios'

export async function fetchRecommendations(userId: string) {
  const response = await axios.get(`/api/recommendations/${userId}`)
  return response.data
}

The smoke test suite was written against the previous feature set. It exercises checkout, signup, dashboard load. It does not exercise fetchRecommendations, because that endpoint did not exist when the smoke test was last updated. The new code ships. In production, the upstream service returns a 503 during a partial outage. axios.get throws an AxiosError. The caller did not wrap the call in try/catch. The user sees a white screen.

Nothing about the smoke test was wrong. The test simply did not know the new path existed.

2. Package upgrades that change error contracts

A major version upgrade of a package can change which methods throw versus return null, often without changing the happy path. Smoke tests pass:

// Looks identical before and after the upgrade
const customer = await stripe.customers.retrieve(customerId)

If the upgrade changes a method that previously returned null for a deleted customer into one that throws StripeInvalidRequestError, code that handled the null case with if (!customer) now crashes when the deleted-customer path triggers in production. The smoke test never exercised the deleted-customer case because it only knows the happy path.

This is the deterministic class of regression a Profile scanner catches at PR time: a method's throwing behavior changed, and the call site has not been updated to match.

3. AI-generated code that compiles but misuses the API

Cursor, Claude, and Copilot all routinely suggest code like this:

// AI-suggested, compiles cleanly, no warning
const user = await prisma.user.findUnique({ where: { id } })
return user.email

TypeScript compiles it. The smoke test passes because the happy path returns a user. The bug is that findUnique returns User | null. If the row does not exist, user.email throws TypeError: Cannot read properties of null. The error reaches your error boundary, the user sees a generic failure page, and your on-call gets paged.

AI suggestions multiply this class of bug. Every PR ships a few more calls that compile but skip error handling for edge cases the model did not surface. Smoke tests do not catch them because the tests exercise the success branch.

Where smoke tests, Snyk, and CodeRabbit still leave gaps

If you run smoke tests, Snyk, and an AI reviewer like CodeRabbit or Cursor Bug Bot, you cover a lot. You still leave a quadrant uncovered. The 2x2:

Read the table column by column. The deterministic static analysis column either catches CVEs (Snyk) or catches API misuse (Nark). The probabilistic column catches everything AI can reason about: logic, naming, intent, architecture.

Snyk genuinely catches CVEs that you would never spot reading a PR diff. Cursor Bug Bot genuinely catches logic bugs in the moment a developer is writing code. CodeRabbit genuinely catches naming and structural issues a busy reviewer would skim past. They each occupy a real slot. None of them deterministically check whether your call sites match the throwing contract of the package version you have installed.

An advisor described the difference as a "softer pillow vs prophylactic" model. Snyk is a softer pillow for the supply chain quadrant. AI reviewers are a softer pillow for code-quality judgment. A Profile scanner is the prophylactic for API correctness. They are not the same purchase and they do not compete.

The same picture as a coverage map by failure class:

Type errors             ->  TypeScript strict mode
CVE / supply chain      ->  Snyk, Socket.dev
General code review     ->  CodeRabbit, Copilot, Cursor Bug Bot
Runtime regressions     ->  Smoke tests, end-to-end tests
API usage correctness   ->  Nark (deterministic Profile scanner)

Each row is a different class of bug. Each column is a different tool. Smoke tests own the runtime-regression row. They do not own the API-usage-correctness row, and no tool with that row owned was sitting in your stack before Nark.

The teams that abandoned CodeRabbit for noise did not abandon the problem CodeRabbit tried to solve. They abandoned the probabilistic approach to it. The deterministic approach stays quiet because it only fires when the failure mode is encoded.

What a deterministic Profile scanner does

A Nark Profile is a YAML specification of how a specific npm package version is meant to be used. This is static analysis in the classic sense (read the source, check it against a rule set, emit findings) applied specifically to the contract between your code and its npm dependencies. The Profile encodes:

• Which methods throw, which return null, which return discriminated unions
• Which error codes the package documents (P2002, card_declined, ECONNREFUSED)
• Which methods need cleanup (open connections, file handles, timers)
• Which methods change throwing behavior between major versions

The scanner reads your TypeScript AST and checks each call site against the Profile for the package version you have installed. It produces the same output on the same input every time. There is no model, no temperature, no token budget. It runs locally, in milliseconds, without uploading your code.

Profiles for 280+ packages ship with the open-source Nark corpus: axios, prisma, stripe, openai, anthropic, redis, pg, zod, twilio, and many more.

How to install one

One line. No signup. No code uploaded.

npx nark

Real output against a TypeScript project:

$ npx nark
Scanning 142 files...
3 violations found:

src/api/checkout.ts:42  prisma.user.create called without try/catch
                        Throws PrismaClientKnownRequestError on
                        unique constraint violation (P2002).

src/api/webhook.ts:18   axios.post called without try/catch
                        Throws AxiosError on 4xx, 5xx, and network
                        failures.

src/jobs/notify.ts:67   stripe.charges.create called without try/catch
                        Throws StripeCardError when the card is declined.

Run `npx nark explain <id>` for fix recommendations.

Each violation cites a specific Profile postcondition. The citation is what makes the result deterministic: the same call site flagged today will be flagged tomorrow against the same Profile version. No drift, no model variance, no flaky CI.

When to add it to CI

The decision is about ceremony, not whether to run it. By team shape:

• Solo developer: Run locally before pushing. No CI step required. The 30-second scan is your last pre-push check.
• Team: Add the GitHub Action to every pull request. Comment on changed files only. Block merge on new violations, allow merge on the existing baseline. This catches the upgrade-changed-error-contracts class of bug at PR review time, before the deploy that would have shipped the regression to production.
• Enterprise: Required check, self-hosted runner, blocks merge. Pair with Snyk for the CVE quadrant. Pair with AI review for the logic quadrant. All three quadrants now covered.

The reason the CI gate is the conversion is that smoke tests run after the build. Profile scans run on every pull request, before the merge. The class of bug catalogued above (new paths, upgraded packages, AI-suggested misuse) never reaches the smoke test if the scan catches it at PR time.

Try it now: a 3-step adoption path

npx nark

The shortest path from "I read the article" to "the gap is closed in CI":

1. Run the scan locally. 30 seconds. No signup, no code uploaded. A first scan on a healthy mid-sized project typically surfaces a handful of deterministic violations, each citing a real Profile postcondition.
2. Triage the output. Fix the high-risk ones now. Baseline the rest (commit them as a known-state file so they do not block the team) and work them down over the next few sprints.
3. Add it to CI. GitHub Action on every pull request. Block merge on new violations, allow the baseline. The upgrade-changed-error-contracts class and the new-paths class both get caught at PR review, before they reach the smoke test.

For self-hosted runners or compliance-grade audit logs, the enterprise tier runs entirely inside your VPC with no code upload.