Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,46 @@
# Changelog

## 2.13.0

### Upgrade notes

Exit codes change in this release. Check any CI step that matches on a specific
code; see [Exit codes](README.md#exit-codes).

- Invalid flags exit `2` instead of `1`. `--disable-blocking` doesn't apply to
them.
- Ctrl+C exits `130` instead of `2`.
- Missing tokens and other setup failures now honor `--disable-blocking` and
`--exit-code-on-api-error`.
- `--strict-blocking` now fails on existing blocking alerts.
- Config-file booleans, integers and choices are validated before scan setup.

### Added

- Under `--strict-blocking`, PR/MR comments list existing alerts in their own
table.

### Changed

- Incomplete scan comparisons or existing-alert results fail the run with the
infrastructure-error code.
- GitLab security report IDs include the ecosystem, full package coordinate and
advisory ID, so existing findings get new IDs once.

### Fixed

- `--strict-blocking`, the GitLab security report and FOSSA output include
alerts on unchanged packages.
- `--strict-blocking` no longer lists findings on added packages twice.
- Every CVE on a package is reported, not one per severity.
- GitLab reports keep findings from different ecosystems and namespaces separate.
- PR/MR comments group existing license violations and retain warnings about
omitted findings after individual ignores.
- Invalid file lists and unreadable config files exit with the usage-error code.
- Exit-code flags retain their startup values if the config file changes during a run.
- A token without `repo:list` is reported as a missing scope, not a missing
repository. Only a missing repository triggers a create request.

## 2.12.1

### Changed: bump pinned Coana CLI to 15.15.1
Expand Down
45 changes: 34 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,12 +217,13 @@ Minimal pattern:

## Exit codes

| Code | Meaning |
|------|---------|
| `0` | Clean scan — no blocking issues (or `--disable-blocking` set) |
| `1` | Blocking security finding(s) detected |
| `2` | Scan interrupted (SIGINT / Ctrl+C) |
| `3` | Infrastructure or API error (timeout, network failure, unexpected error) |
| Code | Meaning |
|-------|---------|
| `0` | Successful report, skipped SCM event, or `--disable-blocking` set; findings gate only evaluated diffs |
| `1` | Blocking security finding(s) detected |
| `2` | Invalid flags or configuration (usage error), including argparse's own errors |
| `3` | Infrastructure or API error: missing API or SCM token, permission or plan problem, timeout, network failure, incomplete scan comparison, unexpected error |
| `130` | Scan interrupted (SIGINT / Ctrl+C) |

`--exit-code-on-api-error <N>` remaps the infrastructure-error code (`3`) to any
value — e.g. a Buildkite
Expand All @@ -239,12 +240,12 @@ error, so the supervisor's exit status (commonly 124 or 137) takes precedence.
The two flags that affect exit codes can cancel each other out, so the order of
precedence matters:

- **`--disable-blocking` wins over everything.** It forces exit `0` for *all*
outcomes — security findings *and* infrastructure errors. If you set it,
`--exit-code-on-api-error` has no effect (you'll always get `0`).
- **`--disable-blocking` forces exit `0` for findings, infrastructure errors and
interrupts.** It takes precedence over `--exit-code-on-api-error` and
`--strict-blocking`. Usage errors still exit `2`.
- **`--exit-code-on-api-error` only applies when `--disable-blocking` is *not*
set.** It changes the infra-error code (and the generic-error code); it never
touches the security-finding code (`1`).
set.** It changes the infrastructure-error code, including unexpected runtime
failures. Findings (`1`), usage errors (`2`) and Ctrl+C (`130`) keep their codes.

So for the common "don't let Socket outages block my pipeline, but still fail on
real findings" goal, use `--exit-code-on-api-error` **without** `--disable-blocking`:
Expand All @@ -263,6 +264,28 @@ scan exit `0` on *both* findings and outages — the `soft_fail: 100` rule would
never match, and real findings would stop blocking. That's usually not what you
want.

### Upgrading to 2.13.0

Update pipeline rules and wrapper scripts that match specific exit codes:

| Outcome | Before 2.13.0 | From 2.13.0 |
|---------|---------------|-------------|
| Invalid flags or configuration | CLI validation: `1`; argparse: `2` | `2` for both; never suppressed by `--disable-blocking` |
| Ctrl+C | `2` | `130`, or `0` with `--disable-blocking` |
| Missing token or other setup failure | Fixed `2` or `3`, depending on the failure | Infrastructure-error code (`3` by default); both exit-code flags now apply |

Blocking findings still exit `1`. With `--strict-blocking`, existing blocking
alerts can now fail jobs that previously passed. GitLab report findings also get
new IDs once; see the [release notes](CHANGELOG.md#2130).

Configuration is validated before credentials or network setup. Use native
JSON/TOML booleans such as `false`, a positive integer `timeout`, and an array of
file paths for `files`. Invalid or unreadable configuration exits `2` even when
credentials are missing or blocking is disabled.

For a compact comparison of failure-handling policies, see
[Exit codes in CI](docs/ci-cd.md#exit-codes-in-ci).

## Common gotchas

See [`docs/troubleshooting.md`](https://github.com/SocketDev/socket-python-cli/blob/main/docs/troubleshooting.md#common-gotchas).
Expand Down
40 changes: 39 additions & 1 deletion docs/ci-cd.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,44 @@ steps:
- "results.sarif"
```

## Exit codes in CI

Choose which outcomes may leave your scan step green. The values below are the
CLI's exit status; CI cancellation and external timeouts can have their own status.

| Outcome | Default | `--exit-code-on-api-error 100` | `--disable-blocking` |
|---------|---------|--------------------------------|----------------------|
| No blocking findings | `0` | `0` | `0` |
| Blocking findings | `1` | `1` | `0` |
| Invalid flags or configuration | `2` | `2` | `2` |
| Setup, API or runtime failure | `3` | `100` | `0` |
| Ctrl+C handled by the CLI | `130` | `130` | `0` |

Use a custom infrastructure code with your CI provider's exit-code allowlist; the
[Buildkite example](../README.md#how-these-options-interact) shows this pattern.
For providers without that feature, remap infrastructure errors to `0`:

```bash
# Keep findings blocking; allow setup/API failures to pass.
socketcli --config .socketcli.toml --exit-code-on-api-error 0

# Report-only mode: findings, infrastructure failures and Ctrl+C exit 0.
# Invalid flags/configuration still exit 2.
socketcli --config .socketcli.toml --disable-blocking
```

Combining the flags gives `--disable-blocking` precedence. Avoid blanket shell
`|| true` or job-wide `continue-on-error` when only API failures should be allowed:
they also hide findings and usage errors. Even with exit `0`, a setup/API failure
does not mean a clean scan; check the logs for a completed report.

Findings block only when a [diff comparison runs](#scan-type-by-pipeline).
SCM event filters can also skip unsupported events with exit `0`; confirm a report
was created when validating a new workflow.
`--strict-blocking` also checks existing blocking alerts; an incomplete comparison
or incomplete existing-alert data is an infrastructure failure. See the
[2.13.0 migration table](../README.md#upgrading-to-2130) when updating existing CI rules.

## Config file usage in CI

Use `--config .socketcli.toml` or `--config .socketcli.json` to keep pipeline commands small.
Expand Down Expand Up @@ -465,7 +503,7 @@ With `--scm github` or `--scm gitlab`, the detected event decides the scan type:

| Event | Scan | Blocks the build |
|:------|:-----|:-----------------|
| Pull request / merge request | Diff scan against the repository's baseline | Yes, on newly introduced alerts |
| Pull request / merge request | Diff scan against the repository's baseline | Yes, on new blocking alerts; also existing blockers with `--strict-blocking` |
| Any other pipeline, including default-branch pushes | Full scan | No |

A full scan has no baseline, so it cannot tell a newly introduced alert from one
Expand Down
37 changes: 30 additions & 7 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ If you don't want to provide the Socket API Token every time then you can use th

**Reachability Analysis Requirements:**

The Python CLI verifies the following **up front** (before invoking the analysis engine) and exits with code **3** if any are unmet:
The Python CLI verifies the following **up front** (before invoking the analysis engine). Missing requirements use the infrastructure-error code (`3` by default), subject to `--exit-code-on-api-error` and `--disable-blocking`:
- `node` - Required to run the analysis engine (the Coana CLI). The engine needs Node.js 22.13 or later.
- `uv` - Required by the analysis engine
- An **Enterprise** Socket organization plan (any `enterprise*` plan, including Enterprise trials)
Expand All @@ -372,7 +372,7 @@ Separately, the analysis engine (coana) needs the **per-ecosystem build toolchai
| Host | Used for | If blocked |
|---|---|---|
| `api.socket.dev` | Socket API calls, and a signed URL for the engine download | The CLI fails |
| `storage.googleapis.com` | Downloading the engine and its analyzers (about 120 MB) | The CLI exits with code **3**: `Failed to run reachability analysis: HTTPSConnectionPool(host='storage.googleapis.com', ...)` |
| `storage.googleapis.com` | Downloading the engine and its analyzers (about 120 MB) | The CLI reports an infrastructure error (`3` by default, subject to the exit-code flags): `Failed to run reachability analysis: HTTPSConnectionPool(host='storage.googleapis.com', ...)` |
| The package registry of each scanned ecosystem, for example `registry.npmjs.org` | Installing your project's dependencies for analysis | The scan still completes, but the affected vulnerabilities are reported as `INSTALL ERROR` with no reachability result |

The engine comes from Socket, not npm, so an npm mirror or registry proxy does not cover the engine download. The registry row does not apply when your dependencies are already installed in the target path, for example a `node_modules` directory from an earlier `npm ci`. The engine is cached as described below; with a warm cache, `storage.googleapis.com` is not contacted.
Expand All @@ -389,6 +389,13 @@ Socket's reachability analysis comes in three forms, referred to by their full n

## Config file support

Values are validated after CLI overrides and before scan setup. Boolean options
use native JSON/TOML booleans; integer options use integers, with `timeout` greater
than zero. `files` accepts an array of nonempty file paths, or its JSON string
representation. Invalid values and unreadable config files exit `2`, including
with `--disable-blocking`. Runtime exit policy uses the configuration loaded at
startup; changing or removing the file during a run does not change its exit code.

Use `--config <path>` to load defaults from a `.toml` or `.json` file.
CLI arguments always take precedence over config file values.

Expand Down Expand Up @@ -430,20 +437,23 @@ Sample config files:
### CI/CD usage tips

For CI-specific examples and guidance, see [`ci-cd.md`](ci-cd.md).
Exit statuses and flag precedence are listed in [Exit codes](../README.md#exit-codes),
with pipeline examples in [Exit codes in CI](ci-cd.md#exit-codes-in-ci).

The CLI runs a pinned version of the Coana CLI, the analysis engine; it does **not** auto-update the engine. It downloads the engine from the Socket API on first use and keeps it in `$XDG_CACHE_HOME/socket/coana/v1` (`~/.cache/socket/coana/v1` when `XDG_CACHE_HOME` is unset). Pass `--reach-version latest` to opt into the newest published version. Use `--reach` to enable reachability analysis during a full scan, or add `--only-facts-file` (with `--reach`) to submit only the reachability facts file (`.socket.facts.json`) when creating the full scan.

#### Advanced Configuration
| Parameter | Required | Default | Description |
|:-------------------------|:---------|:--------|:----------------------------------------------------------------------|
| `--ignore-commit-files` | False | False | Compare regardless of which files changed, scanning every manifest |
| `--disable-blocking` | False | False | Non-blocking CI mode: the CLI always exits **0**, even when blocking alerts are present (including with `--strict-blocking`). Also exits 0 on uncaught runtime errors and Socket API failures, so the job is treated as successful while findings and errors are still logged. Takes precedence over `--strict-blocking`. |
| `--exit-code-on-api-error` | False | 3 | Remap infrastructure failures, including missing tokens, setup failures, incomplete scan data and unexpected runtime errors. Findings (`1`), usage errors (`2`) and Ctrl+C (`130`) are unaffected. `--disable-blocking` takes precedence. |
| `--disable-blocking` | False | False | Non-blocking CI mode: the CLI exits **0** even when blocking alerts are present (including with `--strict-blocking`). Also exits 0 on infrastructure errors (missing API or SCM token, Socket API failures, uncaught runtime errors) and interrupts, so the job is treated as successful while findings and errors are still logged. Invalid flags still exit **2**. Takes precedence over `--strict-blocking`. |
| `--disable-ignore` | False | False | Disable support for `@SocketSecurity ignore` commands in PR comments. When set, alerts cannot be suppressed via comments and ignore instructions are hidden from comment output. See [Who can ignore an alert](#who-can-ignore-an-alert). |
| `--ignore-authorization` | False | enforce | Who may suppress alerts with `@SocketSecurity ignore`. `enforce` requires write access and honors the command with a warning when the provider cannot report it; `strict` rejects it in that case; `off` honors any commenter. See [Who can ignore an alert](#who-can-ignore-an-alert). |
| `--strict-blocking` | False | False | Fail on ANY security policy violations (blocking severity), not just new ones. Only works in diff mode. See [Strict Blocking Mode](#strict-blocking-mode) for details. |
| `--enable-diff` | False | False | Enable diff mode even when using `--integration api` (forces diff mode without SCM integration) |
| `--scm` | False | api | Source control management type |
| `--timeout` | False | 1200 | Timeout in seconds for each API request. This is not a total CLI runtime limit and does not limit local discovery, Git, or reachability analysis. |
| `--timeout` | False | 1200 | Positive integer timeout in seconds for each API request. This is not a total CLI runtime limit and does not limit local discovery, Git, or reachability analysis. |

#### Plugins

Expand Down Expand Up @@ -641,8 +651,10 @@ Security issues detected by Socket Security:

### Important Notes

- **Diff Mode Only**: The flag only works in diff mode (with SCM integration). In API mode, a warning is logged.
- **Diff Mode Only**: The flag applies when a scan comparison runs, with or without SCM integration. A full scan logs a warning and does not evaluate existing violations; see [Scan type by pipeline](ci-cd.md#scan-type-by-pipeline).
- **Error-Level Only**: Only fails on `error=True` alerts (blocking severity), not warnings.
- **Existing alerts come from the full scan**: The scan comparison doesn't carry alerts on unchanged packages, so the CLI reads them from the new full scan, at the cost of one more API request per run. If that request fails or omits an unchanged package, the run fails with the infrastructure-error code rather than passing without them.
- **PR/MR comments**: Existing alerts are listed in their own table after the new ones, blocking first, with license violations grouped per package. `@SocketSecurity ignore` applies to them too. Very large sets are cut off before the comment reaches GitHub's size limit, with a link to the full report. Ignoring every displayed alert preserves the warning about omitted findings; `ignore-all` also clears omitted findings.
- **Priority**: `--disable-blocking` takes precedence - if both flags are set, the build will always pass.
- **First Scan**: On the very first scan of a repository, there are no "existing" violations, so behavior is identical to standard mode.

Expand Down Expand Up @@ -881,9 +893,16 @@ The GitLab Security Dashboard report and the JSON/SARIF diff outputs use differe

### Alert Ignoring via PR/MR Comments

When using the CLI with SCM integration (`--scm github` or `--scm gitlab`), users can ignore specific alerts by reacting to Socket's PR/MR comments. Ignored alerts are removed from `--enable-json`, `--enable-sarif`, and console output.
In SCM diff mode (`--scm github` or `--scm gitlab`), an authorized
`@SocketSecurity ignore <ecosystem>/<package>@<version>` or
`@SocketSecurity ignore-all` comment removes matching new alerts from the CLI's
reports, including the GitLab report. With `--strict-blocking`, it also removes
matching existing alerts from blocking checks and reports. Without strict blocking,
the GitLab report still includes existing findings.

However, the GitLab Security Dashboard report includes **all** alerts matching your security policy (new and existing), regardless of comment-based ignores. This ensures the Security Dashboard always reflects the full set of known issues. To suppress a vulnerability from the GitLab report, adjust the alert's policy in Socket's dashboard rather than ignoring it via a PR comment.
`--disable-ignore` disables these commands. See [Who can ignore an alert](#who-can-ignore-an-alert)
for authorization rules. To suppress a finding across future PRs, update its policy
or triage state in Socket's dashboard.

### Report Schema

Expand All @@ -900,6 +919,10 @@ Socket CLI generates reports compliant with [GitLab Dependency Scanning schema v

**Schema compatibility:** The v15.0.0 schema is supported across all GitLab versions 12.0+ (both self-hosted and cloud). The report includes the `dependency_files` field, which is required by v15.0.0 and accepted as an optional extra by newer schema versions, ensuring maximum compatibility across GitLab instances.

Finding IDs distinguish the ecosystem, full package coordinate and advisory, so
different namespaces or CVEs remain separate findings. Upgrading to 2.13.0 changes
existing finding IDs once; GitLab may show them as newly detected on the first report.

### Performance Notes

When `--enable-gitlab-security` (or `--enable-json` / `--enable-sarif`) is used with a full scan (non-diff mode), the CLI fetches package and alert data from the scan results to populate the report. This adds time proportional to the number of packages in the scan. Without these output flags, no additional data is fetched and scan performance is unchanged.
Expand Down
6 changes: 6 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ Run tests:
make test
```

This runs both the unit and core suites. To focus on the CLI exit-code contract:

```bash
uv run pytest tests/core/test_cli_exit_contract.py tests/core/test_exit_codes.py
```

Run lint/format checks:

```bash
Expand Down
12 changes: 12 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Troubleshooting

## Unexpected CI exit status

| Symptom | What to check |
|---------|---------------|
| Exit `2`, even with `--disable-blocking` | Fix the flags or configuration reported in the logs. Check config-file readability, native boolean/integer values, a positive timeout and the `files` array. Usage errors are never suppressed. |
| Exit `3` with no reported findings | Check tokens, scopes, baseline lookup and scan-completeness errors. This is a failed evaluation; `--exit-code-on-api-error` can remap its code. |
| Ctrl+C is treated as a configuration error | Update wrapper/CI rules: Ctrl+C changed from `2` to `130` in 2.13.0. |
| Exit `0` despite expected blockers | Check `--disable-blocking`, remapping to `0`, and whether the run produced a full scan instead of a diff. Full scans do not gate on findings. |

See [Exit codes in CI](ci-cd.md#exit-codes-in-ci) for the flag combinations and
[upgrade guidance](../README.md#upgrading-to-2130) for older pipeline rules.

## API token scopes for scan comparisons

PR/MR runs compare the new scan against the repository's head scan. That comparison
Expand Down
Loading
Loading