Skip to content

Fail on existing alerts under --strict-blocking, streamline exit code behavior - #453

Open
lelia wants to merge 11 commits into
mainfrom
lelia/ce-498-fix-strict-blocking-baseline-and-duplicate
Open

lelia wants to merge 11 commits into
mainfrom
lelia/ce-498-fix-strict-blocking-baseline-and-duplicate

Conversation

@lelia

@lelia lelia commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Makes --strict-blocking fail on existing blocking alerts, stops alerts from being dropped or duplicated, and decides every exit code in one place. Ships as 2.13.0 because exit codes change; see the CHANGELOG upgrade notes.

Root Cause

  • Existing alerts missing: the cached diff-scans endpoint returns unchanged packages with empty alerts, so --strict-blocking exited 0 with blocking alerts present. The GitLab security report and FOSSA output missed the same alerts.
  • Duplicates: create_new_diff compared type/purl keys against artifact-ID keys, so every added package also counted as unchanged.
  • Dropped CVEs: alerts collapsed on (purl, alert type), and every CVE of one severity shares a type, so a package's second advisory was dropped, even a blocking one behind a warning.
  • Exit codes: setup/API failures exited at separate call sites, bypassing the handler that applies --disable-blocking and --exit-code-on-api-error. Some config-file values bypassed argparse validation, and exception handling reloaded the config file instead of retaining the startup policy. Config errors exited 1, the findings code and Ctrl+C shared 2.
  • The diff API's incomplete flag was never read, and a 403 on the repository lookup was treated as a missing repository.

Fix

  • socketsecurity/exit_codes.py: failures that aren't findings raise InfrastructureError, and cli() maps them through both flags. Invalid flags exit 2, which --disable-blocking doesn't override; Ctrl+C exits 130. The CLI parses configuration once and retains the original exit policy through runtime failures.
  • Configuration is validated before credentials or network setup, including typed booleans/integers/choices, positive timeouts, readable config files and file-list shape. CLI overrides apply before validation. --help describes the same exit contract as the docs.
  • Core.add_existing_alerts_from_full_scan reads alerts on unchanged packages from the new full scan when --strict-blocking, --enable-gitlab-security or --legal-format fossa needs them. It's skipped when the comparison already has them; a failed fetch or unmatched package fails the run with the infrastructure-error code.
  • PR/MR comment: under --strict-blocking, existing alerts get their own table, blocking first, with license violations grouped per package, ignores applied, and a cap below GitHub's comment size limit. Ignoring all visible rows preserves the warning about undisplayed findings; ignore-all clears those too.
  • Issue.consolidation_key() adds the ecosystem and GHSA/CVE ID, and the comment shows the ID next to the title. GitLab finding IDs include the ecosystem, full package coordinate and advisory ID, with PURL prefixes normalized across full scans and comparisons. Existing finding IDs change once on upgrade.
  • Unchanged packages are keyed by artifact ID, and select_diff_alerts skips alerts already listed as new.
  • An incomplete comparison fails without falling back to streaming, including when its artifacts are null or absent. Repository creation happens only after a 404; permissions, rate limits and server failures remain infrastructure errors. A 401/403 lookup names the repo:list scope.

Testing

  • uv run pytest -q -o addopts= --cov=socketsecurity --cov-report=term:skip-covered: 985 passed. Default collection now includes both unit and core suites, so make test covers these regressions. ruff check, uv lock --locked --offline, import smoke and git diff --check pass.
  • Regression coverage includes missing/malformed/empty full-scan results, GitLab advisory and package-coordinate identity, truncated-comment ignore round trips, and CLI suppression of existing blockers with authorization and --disable-ignore. The added bug reproductions fail before their fixes. An additional 141 CLI audit cases cover setup failures, findings/warnings, all five exit policies (including API-error code 0), real SIGINT delivery, invalid configuration, config-file removal, temporary findings overrides, full-scan behavior and skipped SCM events. Repository lookup status/response variants and incomplete-comparison artifact shapes are covered too.
  • README migration table and CI outcome matrix document exit-code changes and flag precedence; troubleshooting and the CLI reference cover incomplete data, comment ignores and GitLab finding-ID migration.
  • Not verified live: that full-scan stream alerts carry the policy action after Alert Resolutions. One run against a scan with a known existing Block alert should confirm it before release.
  • Known gap, not addressed: when --enable-diff falls back to a full scan because no manifest changed, blocking isn't evaluated, so --strict-blocking has no effect on those runs.

Public Changelog

--strict-blocking now fails on existing blocking alerts and lists them in PR/MR comments, and no longer lists findings twice. Every CVE on a package is reported. Exit codes changed: invalid flags exit 2, Ctrl+C exits 130, and --disable-blocking and --exit-code-on-api-error now apply to missing tokens and other setup failures. GitLab finding IDs change once to distinguish ecosystems, namespaces and advisories.

Fixes CE-460
Fixes CE-498
Fixes CE-499
Refs ASK-631

@lelia
lelia requested a review from a team as a code owner November 14, 2026 01:57
@cursor

cursor Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Summary

High Risk
Breaking exit-code semantics for CI and a one-time GitLab finding ID migration, plus stricter failure when comparisons or existing-alert data are incomplete—directly affects whether pipelines pass or block.

Overview
Release 2.13.0 centralizes how socketcli exits and tightens blocking, reporting, and diff semantics—CI pipelines that match on exit codes need updating (see README upgrade table).

Exit codes are unified in socketsecurity/exit_codes.py: usage/config errors exit 2 (not suppressed by --disable-blocking), infrastructure failures go through InfrastructureError and honor --exit-code-on-api-error / --disable-blocking, and Ctrl+C exits 130. Config is validated (types, timeout, files) before tokens or network setup, and startup exit policy is frozen for the run.

--strict-blocking now blocks on existing alerts on unchanged packages by hydrating them from the new full scan when the diff omits them; incomplete comparisons and missing alert data fail as infrastructure errors. PR/MR comments add a separate existing alerts table (with truncation and ignore behavior). Alert deduplication uses Issue.consolidation_key() (ecosystem, package, type, advisory ID) so multiple CVEs on one package are not collapsed. GitLab finding IDs incorporate ecosystem, coordinate, and advisory (one-time ID change on upgrade).

Repository lookup treats 401/403 as missing repo:list scope instead of attempting repo:create. Docs, changelog, and broad core test suites document and lock the exit contract and these behaviors.

Reviewed by Cursor Bugbot for commit 766e50b. Configure here.

@lelia
lelia temporarily deployed to socket-firewall November 14, 2026 01:58 — with GitHub Actions Inactive
@lelia lelia changed the title List each finding once under --strict-blocking List each finding once under --strict-blocking Nov 14, 2026
@lelia

lelia commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@lelia
lelia temporarily deployed to socket-firewall November 14, 2026 03:45 — with GitHub Actions Inactive
create_new_diff built the unchanged package set by filtering packages
keyed by type/purl against added and removed packages keyed by artifact
ID. The keys never matched, so every added package was also treated as
unchanged, and under --strict-blocking its alerts were reported twice:
once as new and once as existing. Against an empty baseline that doubled
every finding in JSON, SARIF, Slack and GitLab output, and inflated the
"EXISTING blocking issues" count in the summary and GitLab commit status.
The exit code was not affected.

Unchanged packages are now selected and keyed by artifact ID, which also
lets transitive unchanged alerts resolve their introducing package.
select_diff_alerts additionally skips an unchanged alert already listed
as new, matching on purl and alert type like the rest of the diff.

Bumps the version to 2.12.2.
Issue.purl holds only name and version, so an npm and a PyPI package can
share it. Keying the dedupe on purl and alert type alone would hide an
existing PyPI alert behind a new npm alert with the same name, version
and type.

The regression tests now run Core.create_new_diff with the scan upload
mocked out, instead of repeating its package assembly, so they fail if
create_new_diff goes back to filtering unchanged packages by purl key.
@lelia
lelia force-pushed the lelia/ce-498-fix-strict-blocking-baseline-and-duplicate branch from fa1fd7b to 4f19040 Compare November 17, 2026 21:23
@lelia
lelia deployed to socket-firewall November 17, 2026 21:23 — with GitHub Actions Active
Failures that are not security findings now raise InfrastructureError
instead of calling sys.exit, and cli() maps them through --disable-blocking
and --exit-code-on-api-error. Before, a missing API or SCM token, missing
reachability dependencies, an ineligible plan, a failed repository create
and a failed reachability run each exited with a hard-coded 2 or 3 that
ignored both flags.

- Invalid flags and config values exit 2, the code argparse already uses
  for its own usage errors, instead of 1, which means blocking findings.
  --disable-blocking does not apply to them.
- Ctrl+C exits 130 instead of 2, so it no longer shares a code with usage
  errors.
- A diff scan the API marks incomplete fails as an infrastructure error
  instead of reporting a result that may be missing alerts. It does not
  fall back to the streaming comparison, which reads the same scans.
- A repository lookup that fails with 401 or 403 is reported as a missing
  repo:list scope instead of being treated as a missing repository and
  ending in a repo:create failure.
The diff collapsed alerts on (purl, alert type) so a package pulled in by
several manifests reports each finding once. Every CVE of one severity on a
package shares an alert type, so a package's second advisory was dropped
from every output, and a blocking advisory could be dropped behind a
warning on the same package, letting the run pass.

Alerts now collapse on Issue.consolidation_key, which adds the ecosystem
and the GHSA or CVE ID. The PR comment shows that ID next to the title, so
two advisories on one package no longer render as identical rows.
@lelia
lelia deployed to socket-firewall November 17, 2026 22:02 — with GitHub Actions Active
…mment

The cached diff-scans endpoint returns unchanged artifacts with no alerts,
so --strict-blocking had nothing to block on and passed runs with blocking
alerts on packages the change didn't touch. The GitLab security report and
FOSSA-format output, which list every current finding, missed the same
alerts.

When one of those outputs is enabled, the CLI now reads the alerts on
unchanged packages from the new full scan. It skips that request when the
comparison already carries them, as the streaming fallback does, and fails
the run if the request fails rather than reporting no existing alerts.

Under --strict-blocking the PR comment lists existing alerts in their own
table after the new ones, blocking first, and stops adding rows before the
comment reaches GitHub's size limit. Ignore comments apply to existing
alerts too, since they now fail the run and each row offers the command.
Bumps the minor version because exit codes change: invalid flags exit 2
instead of 1, Ctrl+C exits 130 instead of 2, more failures honor
--disable-blocking and --exit-code-on-api-error, and --strict-blocking now
fails on existing blocking alerts. The CHANGELOG lists each change a CI
step matching on exit codes needs to know about, and the README exit-code
table and the --disable-blocking and --strict-blocking docs describe the
new behavior.
@lelia
lelia deployed to socket-firewall November 17, 2026 22:07 — with GitHub Actions Active
@lelia lelia changed the title List each finding once under --strict-blocking Fail on existing alerts under --strict-blocking, keep every CVE, and decide exit codes in one place Nov 17, 2026
@lelia

lelia commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@lelia lelia changed the title Fail on existing alerts under --strict-blocking, keep every CVE, and decide exit codes in one place Fail on existing alerts with --strict-blocking, keep all CVEs, decide exit codes in one place Nov 17, 2026
@lelia lelia changed the title Fail on existing alerts with --strict-blocking, keep all CVEs, decide exit codes in one place Fail on existing alerts under --strict-blocking, streamline exit code behavior Nov 17, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread socketsecurity/core/messages.py Resolved
Comment thread CHANGELOG.md Resolved
Comment thread socketsecurity/core/messages.py Resolved
@lelia
lelia deployed to socket-firewall November 17, 2026 22:13 — with GitHub Actions Active
GitLab finding IDs were built from package, alert type and severity, so
two CVEs of one severity on a package shared an ID and the report kept
only one. The ID now includes the advisory ID when the alert has one;
other findings keep the IDs GitLab already tracks them by. A repeated ID
is skipped, since GitLab expects each finding once per report.

In the PR comment's existing-alerts table, license policy violations are
grouped per package with the license row, as in the new-alerts table,
instead of rendering as security findings with a malware link.

The 2.13.0 changelog entry is cut to short bullets of observable
behavior.
@lelia
lelia deployed to socket-firewall November 17, 2026 22:17 — with GitHub Actions Active
@lelia

lelia commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@lelia

lelia commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@lelia
lelia deployed to socket-firewall November 17, 2026 22:32 — with GitHub Actions Active

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@lelia

lelia commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@lelia
lelia deployed to socket-firewall November 17, 2026 22:43 — with GitHub Actions Active

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 766e50b. Configure here.

This branch was successfully deployed

1 active deployment
socket-firewall — 766e50bd Deployed Nov 17, 2026 by lelia via python-sfw-smoke-enterprise #618
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant