Why Your GitHub Actions Security Scanner Shows Green When It's Doing Nothing

Three things together make your GitHub Actions security scanner report green when it's actually doing nothing: continue-on-error: true on the scan step (which you intentionally added during rollout), actions/upload-artifact@v4's default behavior on a missing file (a warning, not a failure), and the lack of an explicit "did the scan produce output?" gate. We dogfooded Nark + Semgrep + gitleaks on our own SaaS this week and the combination silently broke our pipeline 18 times in a row before we noticed. The fix is a three-line verification step.

Quick Answer: Add an explicit test -f your-scanner-output.json step between the scan and the upload-artifact step. Fail loud if the file is missing. Without it, an OOM or config error in the scanner leaves no output, upload-artifact@v4 emits a No files were found with the provided path warning, and continue-on-error: true reports the whole job as success. Your dashboard goes green, no findings are reviewed, no PRs are gated — and no one notices for weeks.

The Failure Mode in One Sentence

continue-on-error: true + actions/upload-artifact@v4 + a scanner that crashes = a green job that produced nothing.

The combination is so common that the failure is almost invisible. Every recommended scanner-integration tutorial includes continue-on-error: true so that "shadow-mode" rollouts don't block PRs. Almost every one then pipes the scanner's output file into actions/upload-artifact@v4 for retention. And almost every one assumes the scanner step will at least produce the output file even if it exits non-zero.

When that assumption breaks — because the scanner OOMs, fails to find its config, can't reach a registry, or hits any number of environment-specific issues — the whole pipeline silently produces nothing.

What We Saw This Week

We added three security scanners to our SaaS CI in a single sprint: Nark (for missing error-handling), Semgrep (OWASP Top 10 + secret detection), and gitleaks (committed-secret detection). All three followed the standard pattern: shadow-mode continue-on-error: true, output to a JSON or SARIF file, upload as a workflow artifact, summarize in the step summary.

The first post-merge CI run on main reported:

Nark scan (self) — ✅ success
Semgrep — ✅ success
Supply chain (audit + secret scan) — ✅ success

The dashboard was green. Twelve commits and 18 workflow runs later, it was still green. The team felt good. The launch narrative looked solid.

When we finally pulled the artifacts to capture the baseline for our comparison articles, this is what we found:

Job	Artifact size	Findings
Nark scan (self)	(no artifact uploaded)	(no scan ran to completion)
Semgrep	(no artifact uploaded)	(no scan ran to completion)
Supply chain (gitleaks part)	10.7 KB	17 findings

Two of three scanners had been silently emitting nothing for the entire week.

What Went Wrong

The smoking gun was buried in the workflow logs as a ##[warning]:

Nark scan (self) | ##[warning]No files were found with the provided path: nark-audit.json. No artifacts will be uploaded.
Semgrep         | ##[warning]No files were found with the provided path: semgrep.sarif. No artifacts will be uploaded.

Tracing back through the same logs to the scan step itself revealed the underlying causes:

Nark scan hit a Node out-of-memory error:

Nark scan (self) | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Nark scan (self) | ##[error]Process completed with exit code 134.

GitHub-hosted runners default Node's old-generation heap to roughly 2 GB. The saas web app's TypeScript program load needed more. The scanner aborted before writing nark-audit.json.

Semgrep hit a configuration conflict:

Semgrep | [ERROR]: Cannot create auto config when metrics are off. Please allow metrics or run with a specific config.
Semgrep | ##[error]Process completed with exit code 2.

Semgrep's --config auto mode derives its ruleset from a metadata upload to their registry — which requires telemetry to be on. We had --metrics=off because we don't want to upload anything. The two flags are mutually exclusive, but the error message was logged inside a continue-on-error: true step and never propagated.

In both cases, the scan step exited non-zero, continue-on-error: true ate the failure, the output file was never produced, actions/upload-artifact@v4 emitted a warning instead of failing, and the job's overall conclusion became success. The GitHub Actions UI showed the green checkmark across the board.

The Three-Line Fix

Add an explicit verification step between the scan and the upload. Fail loud if the output file is missing:

- name: Run nark
  id: nark
  continue-on-error: true
  run: |
    npx -y nark@latest \
      --tsconfig apps/web/tsconfig.json \
      --output nark-audit.json
  env:
    NODE_OPTIONS: '--max-old-space-size=8192'

- name: Verify nark produced an audit
  if: steps.nark.outcome == 'success' || steps.nark.outcome == 'failure'
  run: |
    test -f nark-audit.json || {
      echo "::error::Nark did not produce nark-audit.json — scan crashed before writing output. See the Run nark step log."
      exit 1
    }

- name: Upload nark audit
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: nark-audit
    path: nark-audit.json

The if: on the verify step is important — without it, the step would skip when the scan failed, defeating the purpose. The outcome (not conclusion) lets us check the underlying exit code even when continue-on-error: true masked it into a success conclusion.

The same pattern applies to every other scanner. For Semgrep:

- name: Verify semgrep produced SARIF
  if: steps.semgrep.outcome == 'success' || steps.semgrep.outcome == 'failure'
  run: |
    test -f semgrep.sarif || {
      echo "::error::Semgrep did not produce semgrep.sarif"
      exit 1
    }

For gitleaks:

- name: Verify gitleaks produced SARIF
  if: steps.gitleaks.outcome == 'success' || steps.gitleaks.outcome == 'failure'
  run: |
    test -f gitleaks.sarif || {
      echo "::error::Gitleaks did not produce gitleaks.sarif"
      exit 1
    }

The pattern is identical because the failure mode is identical. The job-level conclusion is independent of what the scan actually produced; the verify step is what ties the two back together.

Why `continue-on-error: true` Stays

You might wonder: if continue-on-error: true is what hides the failure, why not just drop it?

Because it's the right pattern for shadow-mode rollout. The first time a scanner runs on an existing codebase, it surfaces 50 to 250 findings. Blocking PRs from day one is how scanners get ripped out. The shadow-mode pattern is:

Phase 1: continue-on-error: true. The scanner runs, posts a baseline, doesn't block.
Phase 2: triage the baseline. Add .nark/suppressions.json (or .semgrepignore) entries for the framework-level patterns, fix the real targets.
Phase 3: drop continue-on-error: true. New violations now block merges.

The verify-artifact step is what keeps Phase 1 honest. Without it, the scanner can silently break and you wouldn't find out for a week.

Why It Took 18 Runs to Notice

Two things together. First, the dashboard was green — there was no visible signal to investigate. Second, the data we were checking was the step summary we wired into the scanner output. Both scanners had a "Summarize" step after the upload:

- name: Summarize nark result
  if: always()
  run: |
    if [ -f nark-audit.json ]; then
      echo "### Nark scan result" >> "$GITHUB_STEP_SUMMARY"
      # ...
    else
      echo "### Nark scan result" >> "$GITHUB_STEP_SUMMARY"
      echo "nark-audit.json was not produced — the scanner likely crashed." >> "$GITHUB_STEP_SUMMARY"
    fi

The summary step did print "nark-audit.json was not produced" — into the workflow summary page that no one was looking at. The dashboard view showed green. The step summary view, which is two clicks deeper, showed the truth.

This is the deeper lesson: a failure signal that lives behind two clicks isn't a failure signal. Where the signal needs to surface is the dashboard, where the dashboard's signal is the job's overall conclusion. The verify-artifact step puts the failure into that channel.

What To Check Right Now

If you're running any security scanner in CI with continue-on-error: true and actions/upload-artifact@v4, do this:

Open the latest run of your scanner job on main.
Click into the upload-artifact step.
Look for ##[warning]No files were found with the provided path.

If you see it, your scanner has been silently failing. Add the verify step. Re-run.

For Nark specifically, the recommended GitHub Actions workflow has the verify step pre-wired, along with NODE_OPTIONS=--max-old-space-size=8192 to prevent the OOM that started this whole investigation. The same patterns work for Semgrep, gitleaks, and any other scanner that writes to a file.