# Brimyr Every documentation page, concatenated. Canonical site: https://docs.magmamoose.com/brimyr/ --- # Action reference Source: docs/action.md URL: https://docs.magmamoose.com/brimyr/action/ Every input and output of the `magmamoose/brimyr` composite action, with the default the code actually uses. Inputs are all optional. For the CLI behind it, see [CLI reference](cli.md). The action has two gates, **patch coverage** (always) and **net-new quality findings** (opt-in, `quality: 'true'`). They share one job summary, one PR comment, and one exit code, which is the worse of the two. ## Checkout | Input | Type | Default | Description | | --- | --- | --- | --- | | `checkout` | bool | `true` | Run `actions/checkout` first. Set to `false` if you already checked out, which you must do if any earlier step touches the workspace. | | `fetch_depth` | int | `0` | Checkout depth. Must stay `0`: patch coverage needs the merge base, and a shallow clone fails with `ShallowCloneError`. | ## Gate | Input | Type | Default | Description | | --- | --- | --- | --- | | `mode` | enum | `auto` | `auto`, `pr` (gate) or `baseline` (report only, never blocks). `auto` picks `pr` on a pull request and `baseline` on a push. It governs **both** halves: baseline has no diff, so neither patch coverage nor the net-new finding set is gated there. | | `threshold` | float | `80` | Patch coverage below this percentage fails the gate with exit `1`. | | `min_lines` | int | `20` | Diffs with fewer changed executable lines than this are not gated at all. Set `0` to gate every diff. See [Patch coverage](patch-coverage.md#the-sample-size-floor). | | `base_ref` | string | *(PR base SHA)* | Override the base ref or SHA to diff against. | | `head_ref` | string | *(PR head SHA, else `github.sha`)* | Override the head ref or SHA. | | `exclude` | string | *(empty)* | Comma-separated globs, whitespace around the commas is trimmed. Matching changed files leave the denominator entirely, so they are not counted as covered, they just don't count. `*` crosses `/`. | | `strip_prefix` | string | *(empty)* | Comma-separated path prefixes to strip from coverage paths before matching them to diff paths. | ## Test run | Input | Type | Default | Description | | --- | --- | --- | --- | | `ecosystem` | string | *(auto-detect)* | Force one or more of `python`, `javascript` (jest), `vitest`, `node-test`, `dotnet`, `java`, `go`, `shell`, comma-separated. Each named ecosystem runs its own test command and the reports are merged, so `dotnet,shell` gates both halves of a repo whose scanners are bash. A forced ecosystem runs at the repo root; auto-detection also finds [projects in subdirectories](index.md#projects-in-subdirectories) when the root has none. | | `test_command` | string | *(detected)* | Replace the detected test command with a shell command string. | | `provision` | bool | `true` | Install the repo's test dependencies first, with the repo's own dependency manager (`uv run` / `poetry run` / `npm ci`), so a detected suite can actually be launched. See [Dependency provisioning](#dependency-provisioning). Ignored when `test_command` is set. | | `coverage_file` | string | *(empty)* | Ingest pre-made reports as `path[:format]`, comma-separated, and skip the test run. **Globs are expanded**, which is how you name `dotnet test`'s per-project `TestResults/*/coverage.cobertura.xml`. A pattern matching nothing is an error, not an empty result. Format is sniffed when you leave it off. | | `test_timeout` | int | `3600` | Kill the test run after N seconds. A timeout is a **broken run** (exit 2), never 0% coverage. `0` waits indefinitely. | ### Dependency provisioning Brimyr runs your tests on the runner, so something has to install them. By default Brimyr does it, using whatever the repo already declares: | Repo shape | What runs | | --- | --- | | `uv.lock`, or a `[project]` / `[tool.uv]` table | `uv run --with pytest-cov `, plus `--extra` for any `test`, `tests`, `testing` or `dev` extra and `--group` for any `test`, `tests` or `testing` group the project defines | | a `[project]` table with no `uv.lock`, and a `requirements*.txt` that pins **every** requirement with `==` and names every dependency the project declares | the requirements file is the lock: `uv run --no-project --with pytest-cov --with pip --with-requirements python -m `, from the project directory, as a `pip install -r` job would run it (`--with-editable .` too for a `src/` layout) | | `requirements*.txt` and nothing else | the same requirements environment | | Poetry 1.x (`[tool.poetry]`, no `[project]`) | `poetry install`, then `poetry run ` | | `package.json` with no `node_modules` | `npm ci` (falling back to `npm install`), then the detected command | | a vitest suite with no coverage provider declared | `@vitest/coverage-v8` installed with `--no-save` at vitest's own installed version, since `vitest run --coverage` writes nothing without one | | No `node` on PATH, and a `package.json` (at the root or in a subdirectory) or a `.bats` file | `actions/setup-node` installs Node first: the version in `.nvmrc` or `.node-version`, otherwise the current LTS. A runner that already has Node keeps its own. | | No `go` on PATH, and a `go.mod` (at the root or in a subdirectory) | `actions/setup-go` installs the version the shallowest `go.mod` asks for. A runner that already has Go keeps its own. | | Maven, .NET, Go | nothing else: `mvn`, `dotnet test` and `go test` restore their own | | A bats suite with no `bats` on PATH | `npx --yes bats ...` | | A bats suite with `kcov` on PATH | the run is wrapped in `kcov`, writing Cobertura under `coverage/kcov/`. kcov is used when present and [never installed](shell.md). | `pytest-cov` is injected because it is Brimyr's requirement, not yours: a repo can have a complete pytest setup and still fail `pytest --cov` on an unrecognised argument. It is added to the run, never to your lockfile. A plugin your pytest configuration names but nothing declares is injected the same way: `asyncio_mode` without `pytest-asyncio` otherwise fails every `async def` test. A plugin you do declare keeps your version. Nothing is installed into the job's own interpreter. Provisioning declines, and says why on stderr, whenever the repo shows no shape it can act on or the tool that owns it is missing. The run then proceeds exactly as before. Set `provision: 'false'` when the job installs dependencies itself. Setting `test_command` disables it too: an explicit command is your contract, so its setup is yours as well. ## PR comment | Input | Type | Default | Description | | --- | --- | --- | --- | | `pr_comment` | bool | `false` | Post one consolidated comment and update it in place on later pushes. Never blocks. | | `github_token` | string | `${{ github.token }}` | Token used to comment. Needs `pull-requests: write`. The job token comments as `github-actions[bot]`. | | `token_broker_url` | string | *(empty)* | Broker base URL. When set, the comment is authored by `Brimyr[bot]` instead. Fails soft: any problem falls back to the job token. See [PR comment](pr-comment.md). | ## SonarQube Set `sonar_url` and `sonar_token` and the action installs a scanner and runs an analysis. Everything here is non-blocking. See [SonarQube](sonarqube.md). | Input | Type | Default | Description | | --- | --- | --- | --- | | `sonar_url` | string | *(empty)* | SonarQube host URL. Empty disables the whole Sonar leg. | | `sonar_token` | string | *(empty)* | SonarQube token. Pass a secret. Used only when `sonar_url` is set. | | `sonar_project_key` | string | *(repo slug)* | Project key. Defaults to `owner_repo`, because Sonar keys can't contain `/`. | | `sonar_organization` | string | *(empty)* | Sonar organization. SonarCloud only. | | `sonar_sources` | string | *(repo root)* | Value for `sonar.sources`. | | `sonar_args` | string | *(empty)* | Extra properties, comma-separated, each as `-Dkey=value`. Java needs `-Dsonar.java.binaries=...` here or the analysis is skipped. | | `sonar_scanner_version` | string | `8.1.0.6389` | SonarScanner CLI version installed for non-.NET repos. Digits and dots only. .NET repos get `dotnet-sonarscanner` instead. | ## Artifacts | Input | Type | Default | Description | | --- | --- | --- | --- | | `html_report` | bool | `false` | Render a browsable HTML coverage report and upload it. Needs the .NET runtime for ReportGenerator. See [HTML report](html-report.md). | | `html_artifact_name` | string | `brimyr-coverage-html` | Artifact name for that report. | | `emit_json_artifact` | bool | `true` | Upload the patch-coverage JSON summary. | | `json_artifact_name` | string | `brimyr-coverage` | Artifact name for the JSON summary. | ## Quality findings Coverage is half of quality assurance. Set `quality: true` and the action runs [Chargate](https://github.com/MagmaMoose/chargate) as a nested step, MegaLinter's quality linters → SARIF → net-new classification against this PR's diff, then decides pass/fail on `quality_fail_on` and folds the verdict into the same job summary and the same PR comment as coverage. Chargate reports, Brimyr gates. Needs Docker on the runner. | Input | Type | Default | Description | | --- | --- | --- | --- | | `quality` | bool | `false` | Also gate on net-new quality findings. Report-only even when on, until you set `quality_fail_on`. | | `quality_fail_on` | enum | `none` | SARIF level at or above which a net-new finding blocks: `none` (report only), `note`, `warning`, `error`, `any`. `error` is the equivalent of Chargate's `fail_on: high`; `any` blocks on every net-new finding, including ones its linter left unlevelled. | | `quality_linters` | string | *(empty)* | Comma-separated MegaLinter linter keys to run instead of Chargate's curated `quality` set (`GO_GOLANGCI_LINT`, `JAVASCRIPT_ES`, `JAVA_PMD`, `PYTHON_RUFF`, `TYPESCRIPT_ES`). A linter that emits no SARIF is invisible to the gate, and Chargate skips it by name with a reason rather than running it for nothing. | | `quality_json_artifact_name` | string | `brimyr-quality` | Artifact name for the quality JSON summary. Uploaded when `quality` is on and `emit_json_artifact` is `true`. | The threshold speaks SARIF **levels** and not Chargate's severity bands. Chargate gates on per-result verdicts, where a missing `security-severity` falls back to the level, so its bands work. Brimyr reads only the counts document, whose `per_severity_*` maps are populated solely from a real `security-severity`, a property quality linters essentially never emit, so a band-valued threshold read off it would match nothing on every PR. It defaults to `none` for the same reason a first PR should not go red with hundreds of findings: measure a release cycle, then pick a level. The nested step is `magmamoose/chargate`, pinned by SHA and bumped by Dependabot like every other action here. It runs with `fail_on: none` and `continue-on-error: true`, so Chargate never sets this job's exit code and a Docker or MegaLinter failure can't take the coverage gate down with it. That is not a silent pass: the action branches on that step's `outcome`, and on anything other than `success` it passes `--quality-scan-broken`, which reads no file at all and reports a scan that did not complete, exit `2`, and `quality_gate_result` `error`. It goes by the outcome rather than by the file because the file settles nothing. `chargate ci` writes its counts JSON *before* it decides whether the scan produced any runs, so a failed scan can leave a well-formed row of zeros behind, which is precisely what a clean pull request looks like. The verdict is folded into the coverage summary and the coverage comment, and the job's exit code becomes the worse of the two halves on the `0` < `1` < `2` scale, clean coverage does not launder a blocking quality finding. In `baseline` mode neither half gates. !!! note "An exit-`0` scan is not proof of a complete one" Chargate can decline to start a linter, no image for the runner's architecture, no SARIF output, the linter disabled, and still exit `0`, so anything that linter would have reported is missing from the count, and missing findings are what a clean repo looks like too. The action forwards Chargate's `linters_skipped` output as `--quality-scan-note`, and Brimyr states the shortfall next to the count in the summary and the comment. It never blocks. See [Quality findings](quality-findings.md#a-scan-that-completed-is-not-necessarily-a-full-one). !!! note "The pin has to carry the `quality` flavor" `action.yml` pins `magmamoose/chargate@753df5673aa274e87596c0b88159dc4b0cce33d1` (v2.11.71), which does, so `quality: 'true'` is usable. v1.9.0 of this action pinned v2.11.25, which predates the flavor; on that pin the nested step fails and Brimyr reports a broken scan (exit `2`, `quality_gate_result` `error`) rather than a clean quality half. If you pin Brimyr below v1.9.1, leave `quality` off. ## Runtime | Input | Type | Default | Description | | --- | --- | --- | --- | | `python_version` | string | `3.12` | Python used to run the CLI. Brimyr installs itself into its own venv and calls it by absolute path, so it doesn't depend on this. | !!! warning "`python_version` has a job-wide side effect" The action runs `actions/setup-python`, which changes the Python for every step **after** it, not just for Brimyr. Leave it alone unless the runner's default Python is older than 3.11. ## Outputs | Output | Description | | --- | --- | | `mode` | Resolved run mode: `pr` or `baseline`. | | `gate_result` | `pass`, `fail`, `error` or `skipped`. `error` means a broken run, never 0% coverage; `skipped` means nothing was gated, either because no test suite was detected or because the suites that ran measure no coverage at all ([bats without kcov](shell.md)). Neither is a measurement. | | `patch_coverage` | Patch coverage percentage, two decimal places. | | `covered_lines` | Covered changed executable lines. | | `total_lines` | Total changed executable lines, the patch-coverage denominator. | | `total_coverage` | Overall coverage across the files the run measured. **Empty string** when nothing was measured, never `0.00`, so an unmeasured run and a genuinely zero-covered one don't look the same to a downstream `if`. | | `quality_gate_result` | `pass`, `fail` or `error` for the quality half. `error` means the scan did not complete, which is the state to notice first: it is a tool error, not zero findings. Empty when `quality` is off. | | `quality_net_new_count` | Net-new (PR-introduced) quality findings. Empty when `quality` is off. | | `quality_blocking_count` | Net-new quality findings at or above `quality_fail_on` : the ones that actually block. | | `quality_fail_on` | The threshold that was in force. It ships alongside the verdict because at `none` a report-only run and a genuinely clean one both say `pass`, and they are not the same thing. Empty when `quality` is off. | ## Permissions ```yaml permissions: contents: read pull-requests: write # only if pr_comment is true id-token: write # only if token_broker_url is set ``` ## Environment variables The action sets these for you. They matter if you run the CLI directly. | Variable | Read by | Purpose | | --- | --- | --- | | `SONAR_TOKEN` | `sonar.py`, `sonar_dotnet.py` | Sonar credential. Passed in the environment, never on argv, so it can't leak into a process listing. Rename with `--sonar-token-env`. | | `GITHUB_TOKEN` | `github_comment.py` | Comment credential. Rename with `--github-token-env`. | | `GITHUB_API_URL` | `cli.py` | API base. Set this for GitHub Enterprise. Defaults to `https://api.github.com`. | | `GITHUB_REPOSITORY` | `cli.py` | `owner/repo`. Used for the comment target and the default Sonar project key. | | `GITHUB_EVENT_NAME`, `GITHUB_EVENT_PATH` | `cli.py`, `modes.py` | Mode resolution and PR number discovery. | | `GITHUB_OUTPUT`, `GITHUB_STEP_SUMMARY` | `report.py` | Where step outputs and the job summary are written. | | `GITHUB_ACTIONS` | `cli.py` | Switches warnings to `::warning::` annotations. | | `ACTIONS_ID_TOKEN_REQUEST_URL`, `ACTIONS_ID_TOKEN_REQUEST_TOKEN` | `broker_client.py` | Injected by Actions when `id-token: write` is set. Used to mint the OIDC token the broker exchanges. | --- # Architecture Source: docs/architecture.md URL: https://docs.magmamoose.com/brimyr/architecture/ Brimyr is one `brimyr` Python CLI (`src/brimyr/cli.py:main`) behind three GitHub surfaces, running two gates: **patch coverage** and **net-new quality findings**. The design splits cleanly into a **pure core** and a thin set of **side-effecting edges**. ## Module map ```text src/brimyr/ cli.py # argparse dispatch: coverage | ci | local | lint | version coverage/ # ★ THE PURE CORE: deterministic, no I/O, heavily tested diff.py # unified-diff text -> DiffIndex (changed files + added line ranges) model.py # CoverageReport / FileCoverage + CoverageBuilder (covered-wins merge) lcov.py # lcov .info -> CoverageReport cobertura.py # Cobertura XML -> CoverageReport jacoco.py # JaCoCo XML -> CoverageReport (its own format, not Cobertura) gocover.py # Go profile -> CoverageReport (blocks trimmed to statement lines) patch.py # DiffIndex ∩ CoverageReport -> PatchCoverage (the gate's heart) # + compute_total_coverage: the reported, never-gated total git.py # the ONLY git/subprocess boundary (merge-base, diff, shallow detect) detect.py # ecosystem markers -> Ecosystem (test command + coverage format) provision.py # repo shape -> how to install its deps, so the suite can be LAUNCHED runner.py # run tests with coverage, locate + ingest the file (broken-run rule) sonar.py # sonar-scanner runner (failure-isolated, never raises) sonar_dotnet.py # dotnet sonarscanner begin/end WRAPPING the build (.NET only) html_report.py # ReportGenerator wrapper -> browsable HTML artifact (optional) gate.py # patch % + threshold -> pass/fail + exit code quality.py # pure: chargate's net-new counts + fail_on level -> a verdict modes.py # PR (gate) vs baseline (no gate) resolution report.py # GitHub job summary + step outputs (also renders the PR comment) github_comment.py # marker-based upsert of the ONE PR comment (never raises) broker_client.py # Actions OIDC -> Brimyr[bot] token (fails soft, falls back) local.py # local base resolution for the pre-push check ``` ## The design rule `coverage/` is **pure**: it takes already-parsed data (unified-diff text + a coverage report) and returns numbers. `git.py`, `runner.py`, `sonar.py`, `sonar_dotnet.py` and `html_report.py` are the only modules that shell out, and each injects its runner, so the core is unit-tested with synthetic diff text and coverage strings, no real repository or toolchain required. `quality.py` is pure in the same way without living under `coverage/`: it is handed already-parsed JSON and returns a verdict, and `cli.py` does the reading. The two network edges, `github_comment.py` and `broker_client.py`, follow the same shape: stdlib `urllib` only, with the opener injected so the tests need no network, and neither ever raises out into the gate. !!! warning "Keep the boundary" Do **not** import `subprocess`, `os`, network code, or GitHub Actions into `coverage/`. That separation is what makes the crown-jewel `patch.py` trivially testable and deterministic. ## Data flow (PR / gate mode) 1. **`modes.resolve_mode`** decides PR (gate) vs baseline (no gate) from `GITHUB_EVENT_NAME` or an explicit flag. 2. **`detect.detect_ecosystems`** sniffs marker files → the ecosystem(s) and their test commands (or the escape hatch / forced ecosystem is used instead). When the root matches nothing, it searches a few directories down for projects of their own, each carrying the `project_dir` it was found in. 3. **`provision.plan`** reads the same tree for a dependency manager it can use, and returns a setup command (`npm ci`), a wrapped test command (`uv run …`), or an empty plan and a reason. Detecting a suite that then cannot be launched is worth nothing, and that is a red gate on a repo whose tests are fine. 4. **`runner.run_tests`** runs the plan's setup, then each ecosystem's command with coverage on, from its project directory, locates the emitted file, and parses it (`coverage.lcov` / `coverage.cobertura` / `coverage.gocover`) into a `CoverageReport` whose paths are repo-relative. A failed setup, a failed run or an empty one sets `RunResult.broken`. 5. **`git.compute_changed_lines`** resolves `merge-base(base, head)`, runs `git diff --unified=0`, and hands the text to `coverage.diff.parse_unified_diff` → a `DiffIndex`. 6. **`coverage.patch.compute_patch_coverage`** intersects the diff with the report → a `PatchCoverage` (covered / total changed-executable lines, per-file misses). 7. **`gate.decide_gate`** applies the threshold (and the broken-run rule) → a `GateDecision` and exit code. 8. **`sonar.run_scanner`** (optional) ships quality + coverage to SonarQube. It is failure-isolated: it never raises, so a Sonar outage can't fail the gate. 9. **`quality.decide_quality_gate`** (optional) decides the net-new half on `fail_on`, over the counts `cli` read and `quality.parse_counts` validated out of the JSON the nested Chargate step left behind, or, told the scan never completed, `quality.broken_decision` reads nothing and reports a tool error. Same `Mode.gates` flag as coverage, so baseline gates neither. 10. **`report`** writes the GitHub job summary and step outputs, `render_summary`'s coverage block, with `render_quality_summary`'s net-new block appended when that half ran. 11. **`github_comment.post_pr_comment`** (optional) puts that same rendered summary on the PR as a single marker-owned comment, creating it once, then `PATCH`ing it on every later push. When `token_broker_url` is set, **`broker_client.mint_bot_token`** first exchanges the job's Actions OIDC token for a `Brimyr[bot]` installation token; on any failure it returns `None` and the job's `GITHUB_TOKEN` is used instead. Both are failure-isolated, like Sonar. Baseline mode skips the gating, both halves. It computes coverage against an empty `DiffIndex`, ships to Sonar, and never blocks; a baseline quality run is report-only for its own reason (no diff, so no net-new set worth gating), which the summary names rather than blaming the threshold. The process exits with the worse of the two halves' codes. ## The quality half calls Chargate instead of importing it Brimyr is quality assurance, and coverage is only half of that. The other half judges the **net-new** findings a MegaLinter quality run produced. Chargate already owns a finished net-new engine (`chargate filter-sarif`), so brimyr does not vendor it or re-implement it: `action.yml` runs `magmamoose/chargate` as a nested step, and `quality.py` reads the two files that step leaves behind. The `--counts-json` document is the verdict's only input; the filtered SARIF is display-only, skimmed for the `path:line [rule]` strings the summary lists. Anything unreadable, unrecognised, or self-contradicting is exit `2`, because a half that cannot evaluate must not report a pass, and so is a scan that never completed, which `action.yml` detects from the nested step's `outcome` and passes on as `--quality-scan-broken`, reading no file at all. `brimyr lint` runs that half on its own, under its own PR-comment marker. `brimyr ci --quality-counts` instead folds the same verdict into the one job summary and the one PR comment coverage already writes, which is the consolidated view and the reason to prefer it. What the threshold means, and how to turn the half on, are on [Quality findings](quality-findings.md). ## The token broker is not part of the CLI `broker/` in the repository is a **separate deployable**: the AWS Lambda service that mints `Brimyr[bot]` tokens, with its own `pyproject.toml`, lockfile, Ruff config and CI job. Nothing under `src/brimyr` imports it; `broker_client.py` talks to it over HTTPS like any other remote service, which is what keeps the CLI stdlib-only and dependency-free. Its architecture, local LocalStack loop and go-live runbook live with the code, in [`broker/README.md`](https://github.com/MagmaMoose/brimyr/blob/main/broker/README.md). ## Exit-code contract | Code | Meaning | | --- | --- | | `0` | pass | | `1` | patch coverage below threshold, or blocking net-new quality findings | | `2` | broken test run / setup / usage error, or unreadable quality input | A *broken* test run is a tool error (`2`), never "0% patch coverage". When both halves run in one `brimyr ci`, the process exits with the worse of the two codes on that same `0 < 1 < 2` scale: clean coverage never launders a blocking quality finding. ## Testing Tests mirror modules 1:1 under `tests/` (e.g. `test_patch.py`, `test_gate.py`, `test_cobertura.py`). The pure core is tested with synthetic inputs; the subprocess boundaries inject their runner so they are exercised without a real toolchain, git, or a live SonarQube. --- # CLI reference Source: docs/cli.md URL: https://docs.magmamoose.com/brimyr/cli/ All three GitHub surfaces drive the same `brimyr` CLI. Exit codes: `0` pass · `1` patch coverage below threshold, or blocking net-new quality findings · `2` broken test run / setup / usage error, a quality input the gate could not evaluate, or a quality scan that did not complete. A run that does both halves exits with the worse of the two. ```sh brimyr [options] ``` ## `brimyr coverage` The pure patch-coverage computation: coverage file(s) + a base/head → patch coverage % + a gate exit code. Runs **no** tests, so it's decoupled from any toolchain and unit-tested in isolation. ```sh brimyr coverage --coverage-file coverage.xml --base "$BASE" \ --threshold 80 --json-out coverage.json ``` | Flag | Default | Purpose | | --- | --- | --- | | `--coverage-file` | (required) | Coverage file `path[:format]` (`lcov`\|`cobertura`\|`jacoco`\|`gocover`; inferred from the extension, `.xml` is resolved by its root element, and Go profiles are detected by their `mode:` first line). Repeatable. | | `--base` | (required) | Base ref/SHA (PR target). | | `--head` | `HEAD` | Head ref/SHA. | | `--repo` | `.` | Path to the git repository. | | `--threshold` | `80` | Patch-coverage % that blocks below it. | | `--strip-prefix` | none | Path prefix to strip from coverage paths before matching (repeatable). | | `--min-lines` | `20` | Skip the threshold when the diff has fewer changed executable lines than this. `0` gates every diff. | | `--exclude` | none | Glob whose matching changed files leave the denominator entirely (repeatable). `*` crosses `/`. | | `--no-merge-base` | off | Diff `base..head` directly instead of `merge-base(base, head)..head`. | | `--json-out` | none | Write the patch-coverage summary as JSON. | | `--no-gate` | off | Always exit `0` (report only). | | `--quiet` | off | Suppress the human summary. | ## `brimyr ci` The full CI flow: detect the ecosystem(s), run the right test command with coverage, gate on patch coverage (PR events), and run `sonar-scanner` (non-blocking). Given the [quality flags](#quality-flags) it decides the net-new quality half as well, and renders both verdicts into one summary and one comment. ```sh brimyr ci --mode auto --sonar-url https://sonar.example.com --sonar-project-key my-svc ``` Key flags beyond the shared options: | Flag | Default | Purpose | | --- | --- | --- | | `--mode` | `auto` | `auto` (from `GITHUB_EVENT_NAME`), `pr` (gate), or `baseline` (no gate). | | `--coverage-file` | none | Escape hatch: ingest a pre-made report instead of running tests. Repeatable. | | `--ecosystem` | none | Force `python`\|`javascript`\|`vitest`\|`node-test`\|`dotnet`\|`java`\|`go`\|`shell` instead of auto-detect. `javascript` is jest. Repeatable. A forced ecosystem runs at the repo root. | | `--test-command` | none | Override the detected test command (a shell command string). | | `--sonar-url` | none | SonarQube host URL (enables the non-blocking scanner run). | | `--sonar-token-env` | `SONAR_TOKEN` | Env var holding the Sonar token. | | `--sonar-project-key` / `--sonar-organization` | none | Sonar targeting. | | `--sonar-sources` | `.` | `sonar.sources` value. | | `--sonar-arg` | none | Extra raw `sonar-scanner` arg, e.g. `-Dsonar.foo=bar` (repeatable). | | `--html-report` | none | Directory for a browsable HTML coverage report. Needs ReportGenerator; never affects the gate. | | `--test-timeout` | `3600` | Kill the test run after N seconds. A timeout is a broken run (exit 2). `0` waits forever. | ### PR comment flags Also accepted by `brimyr local` and `brimyr lint`, though they only do anything when there is a PR to comment on and a token to comment with. All of them are non-blocking: nothing here can change the exit code. See [PR comment](pr-comment.md) for the full behaviour. | Flag | Default | Purpose | | --- | --- | --- | | `--pr-comment` | off | Post/update the single comment on the PR, consolidated (coverage plus the quality block, when that half ran) on `ci` and `local`; quality only on `lint`, under its own marker. | | `--pr-number` | from the event | PR number (read from `$GITHUB_EVENT_PATH` otherwise). | | `--repo-slug` | `$GITHUB_REPOSITORY` | `owner/repo` being commented on. | | `--github-token-env` | `GITHUB_TOKEN` | Env var holding the comment token; needs `pull-requests: write`. | | `--token-broker-url` | none | Broker base URL. Authors as `Brimyr[bot]`; falls back **silently** to the token above on any failure. Needs `id-token: write`. | | `--github-api-url` | `$GITHUB_API_URL`, else `api.github.com` | GitHub API base (GitHub Enterprise). | ### Quality flags Also accepted by `brimyr local`. Supplying **either** `--quality-counts` or `--quality-scan-broken` turns the quality half on: Brimyr reads what `chargate filter-sarif` left behind, or, for a scan that did not complete, reads nothing and says so, decides pass/fail on its own threshold, and renders that verdict into the **same** summary and the **same** PR comment as coverage. One consolidated view is the reason to prefer this over the standalone [`brimyr lint`](#brimyr-lint). The process exit code becomes the worse of the two halves. | Flag | Default | Purpose | | --- | --- | --- | | `--quality-counts` | none | Chargate's `filter-sarif --counts-json` output. The verdict's only input; supplying it turns the quality half on. | | `--quality-findings` | none | Chargate's `filter-sarif --out` net-new SARIF. Read only to list findings in the summary; a result count that contradicts the counts JSON is a hard error (exit `2`). | | `--quality-fail-on` | `none` | SARIF level at or above which a net-new finding blocks: `none` (report only), `note`, `warning`, `error`, `any`. | | `--quality-scan-broken` | off | The scan did not complete. Turns the quality half on, reads no file at all, and reports a tool error (exit `2`, `quality_gate_result` `error`) : the counts file a failed scan leaves behind is a row of zeros, which is what a clean PR looks like. | | `--quality-scan-note` | none | Linters the scan could not run, stated in the summary next to the count. A completed scan is not necessarily a full one. Never blocks. | | `--quality-json-out` | none | Write the quality summary as JSON here. | ## `brimyr local` Run the gate against a **locally inferred** base (the repo's default branch) to check a branch before pushing. Same flags as `ci`, including the quality and PR-comment ones, plus an optional `--base` to override the inferred base. In practice it is the coverage half you run locally, since the quality half needs Chargate's output. ```sh brimyr local # detect, run tests, gate vs the default branch brimyr local --base main ``` ## `brimyr lint` Gate on the net-new **quality** findings Chargate already classified. Brimyr does not import, vendor or re-implement that engine, it reads the two files `chargate filter-sarif` writes and decides on them. Chargate reports, Brimyr gates. Runs no linter and parses no diff. ```sh brimyr lint --counts chargate-reports/counts.json \ --findings chargate-reports/net-new.sarif --fail-on error ``` | Flag | Default | Purpose | | --- | --- | --- | | `--counts` | none | Chargate's `filter-sarif --counts-json` output. The gate's only input. Required unless `--scan-broken` says there is nothing worth reading. | | `--findings` | none | Chargate's `filter-sarif --out` net-new SARIF. Read only to list findings in the summary; a result count that contradicts `--counts` is a hard error (exit `2`). | | `--fail-on` | `none` | SARIF level at or above which a net-new finding blocks: `none` (report only), `note`, `warning`, `error`, `any`. | | `--no-gate` | off | Always exit `0` (report only). | | `--scan-note` | none | Linters the scan could not run, stated in the summary next to the count. A completed scan is not necessarily a full one. Never blocks. | | `--scan-broken` | off | The quality scan did not complete. Skips every read and reports a tool error (exit `2`) : the counts file a failed scan leaves behind is a row of zeros, which is what a clean PR looks like. | | `--json-out` | none | Write the quality summary as JSON here. | | `--quiet` | off | Suppress the human summary. | The threshold speaks SARIF **levels** and not Chargate's severity bands. Chargate gates on per-result verdicts, where a missing `security-severity` falls back to the level, so its bands work. Brimyr reads only the counts document, whose `per_severity_*` maps are populated solely from a real `security-severity`, a property quality linters essentially never emit, so a band-valued threshold read off it would match nothing on every PR. `error` is the equivalent of Chargate's `fail_on: high`, and `any` blocks on every net-new finding including the ones its linter left unlevelled. It also takes the [PR comment flags](#pr-comment-flags), and comments under its **own** marker: the two subcommands can run in either order, or only one of them, so sharing a marker would mean whichever ran last erased the other's verdict. Two comments is the cost of running the halves separately. Exit codes are the coverage gate's: `0` pass · `1` blocking net-new findings · `2` an input it could not read, a `schema_version` it does not recognise, counts that contradict themselves, or `--scan-broken`. A gate that cannot evaluate its input never goes green. ## `brimyr version` Prints the brimyr version (also `brimyr --version`). ## Exit codes | Code | Meaning | | --- | --- | | `0` | Pass. Includes a vacuous pass (nothing coverable changed) and a skip (no test suite detected). | | `1` | Patch coverage below the threshold, or blocking net-new quality findings. **Only ever a verdict on the pull request.** | | `2` | Broken run, setup error, or an input the gate could not evaluate. Never a verdict on the code. | A run that does both halves exits with the worse of the two. The line between `1` and `2` is load bearing, so environment problems are held to it: - A `--repo` that is not a directory is `2`, not a green skip. Detection finds no marker files in a path that does not exist, so before this was checked a mistyped `--repo` reported "no test suite detected" and passed. - Git failing to start at all (not installed, or an unreadable working directory) is `2`, with a message naming the path. - A `--json-out` / `--quality-json-out` that cannot be written is **not** fatal. The gate has already been decided by then, so the write failure is a warning and the run keeps its own exit code: losing the summary, the PR comment and the verdict because a receipt could not be filed is the worse trade. Parent directories are created for you. --- # .NET solutions Source: docs/dotnet.md URL: https://docs.magmamoose.com/brimyr/dotnet/ Brimyr detects a .NET repo from `*.sln`, `*.slnx`, `*.csproj`, `*.fsproj` or `*.vbproj`, runs ```bash dotnet test --collect:"XPlat Code Coverage" --results-directory ./TestResults ``` and reads every `TestResults/**/coverage.cobertura.xml` it produced. ## Multi-project solutions `dotnet test` on a solution writes **one Cobertura report per test project**, each in its own `TestResults//` directory. Brimyr ingests **all** of them and merges them covered-wins, so a source file exercised by two different test projects comes out covered rather than half-covered. This matters more than it sounds. Patch coverage treats a file the report never mentions as contributing nothing to the denominator. That is deliberate, and it is what stops an unrelated language's files from dragging the number around. But it also means that a *missing* report is indistinguishable from *nothing coverable changed*: every changed file in the missing project silently leaves the denominator and the gate reports a comfortable pass. Merging every report is what keeps that from happening. If you see a suspiciously round 100% on a solution with several test projects, check that each one actually emitted a report: ```bash find TestResults -name coverage.cobertura.xml | wc -l # expect one per test project ``` ## Excluding generated code Scaffolded code is the usual reason a real patch-coverage number is unusable. Nobody writes tests for an EF Core migration, it can be thousands of lines, and one scaffolded migration in a pull request can sink an otherwise well-tested change below the threshold. `exclude` drops matching **changed files** from the denominator entirely. They are not counted as covered, they simply do not count: ```yaml - uses: magmamoose/brimyr@v1 with: exclude: "*Migrations*,*ModelSnapshot*,*.Designer.cs,*AspNetCoreGeneratedDocument*,**/obj/**" ``` Globs are matched against the repo-relative, forward-slash path, and `*` crosses `/`, so `*Migrations*` catches the folder at any depth without you having to know how deep it sits. This is the same idea as coverlet's `Exclude` and ReportGenerator's `-classfilters`, and you can use either instead: anything already absent from the coverage report is absent from the denominator too. `exclude` is for the case where the file *is* in the report and you still do not want it gated. !!! note "Excluding everything is a pass, not a zero" A pull request that only touches excluded files has no coverable changed lines, so it is a vacuous 100% pass: the same rule that applies to a docs-only change. It is not scored 0%. ## Gating, not just reporting A coverage report published to Pages tells you the number after the fact. It does not stop a change that lowers it, and on a large codebase the total barely moves anyway, which is why total-coverage targets get ignored. Brimyr gates on the **lines the pull request changed**, so the number responds to the work in front of you, and pre-existing uncovered code never blocks. The two compose: keep publishing the HTML report if it is useful, and let Brimyr answer "is *this change* tested". ```yaml name: Coverage on: [pull_request] permissions: contents: read pull-requests: write # for the PR comment jobs: coverage: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 with: { fetch-depth: 0 } # patch coverage needs the merge-base - uses: actions/setup-dotnet@v4 with: { dotnet-version: '8.0.x' } - run: dotnet build --configuration Release - uses: magmamoose/brimyr@v1 with: checkout: 'false' threshold: '80' pr_comment: 'true' exclude: "*Migrations*,*ModelSnapshot*,*.Designer.cs" ``` Start with `mode: baseline`, which computes and reports the number without gating. Once the numbers look right on real pull requests, switch to the default and pick a threshold from what you actually saw. ## Mixed .NET and JavaScript A repo with both `*.sln` and `package.json` is detected as both ecosystems. Brimyr runs each one's test command, reads Cobertura from one and lcov from the other, and merges them into a single patch-coverage number, no separate merge step, and no ReportGenerator pass, required. ### Reuse the reports you already have If your pipeline already runs the tests, point Brimyr at the reports instead of letting it run them again. On a large solution that halves the PR's CI time. `dotnet test` writes each project's report to a per-run GUID directory, so use a glob: ```yaml with: coverage_file: "**/TestResults/*/coverage.cobertura.xml" ``` A pattern that matches nothing is an error, not an empty result: contributing no reports would pass at a vacuous 100%. If you already produce merged reports another way, feed those in directly: ```yaml with: coverage_file: "TestResults/merged.cobertura.xml,web/coverage/lcov.info" ``` ## SonarQube Brimyr runs the **SonarScanner for .NET** automatically when it detects a .NET repo. Set `sonar_url` and `SONAR_TOKEN`; there is nothing else to configure. It has to work differently from every other ecosystem, and the difference is not cosmetic. SonarSource documents that the SonarScanner CLI *"doesn't support C# or VB.NET analysis"*, C# issues are produced by Roslyn analyzers that `dotnet sonarscanner begin` injects into the compilation, and `end` *"collects the analysis data generated by the build"*. No compile between the two steps means no analysis at all. So the scanner **wraps** the run: ```bash dotnet sonarscanner begin /k:… /d:sonar.cs.cobertura.reportsPaths=**/TestResults/**/coverage.cobertura.xml dotnet build --no-incremental --disable-build-servers dotnet test --collect:"XPlat Code Coverage" ← Brimyr's normal run dotnet sonarscanner end ``` Three consequences worth knowing: * **The coverage path is declared at `begin`**, before any report exists, hence the wildcard. `end` accepts only three flags (`sonar.token`, `sonar.clientcert.password`, `sonar.scanner.truststorePassword`); everything else must be given up front. The wildcard is also what makes a multi-test-project solution work without enumerating GUID directories. * **`--no-incremental` is not optional.** A cached build compiles nothing, so the analyzers `begin` injected never run and `end` finds no analysis data, a green run that uploaded an empty analysis. * **`coverage_file` disables it.** If you hand Brimyr a ready-made report there is no build, so there is nothing for `end` to collect. Brimyr skips the wrap rather than uploading an empty analysis. !!! note "The property is `reportsPaths`, plural" `sonar.cs.cobertura.reportsPaths`, unlike `sonar.python.coverage.reportPaths` and `sonar.javascript.lcov.reportPaths`. Sonar is not consistent here, and the singular form is silently ignored. If `end` cannot run or fails, Brimyr removes `.sonarqube/`. Nothing reached the server either way (upload happens at `end`), but that directory holds the injected MSBuild hooks and leaving it behind makes the *next* build in the workspace fail for reasons that look unrelated. --- # Go Source: docs/go.md URL: https://docs.magmamoose.com/brimyr/go/ Brimyr detects a Go module from `go.mod` plus at least one `_test.go` file that `go test ./...` would actually run, and runs ```bash go test -coverprofile=coverage.out ./... ``` Coverage is built into the Go toolchain, so there is nothing to install beyond Go itself: no plugin, no converter. `go test` fetches the module's dependencies on its own. When the runner has no `go` on PATH, the action installs one with `actions/setup-go`, at the version the repo's own `go.mod` asks for. A runner that already has Go keeps its own. ## What counts as a test signal `go.mod` alone says there is a module, not that anything tests it, so detection also wants a `_test.go` file belonging to the root module. These do not count: - anything under `vendor/`, `testdata/`, or a directory whose name starts with `.` or `_` (the go tool skips all of them when it expands `./...`); - a test file inside a **nested module**, a subdirectory with its own `go.mod`. `./...` stops at a nested module, so its tests are not the root module's. A repo whose only module sits in a subdirectory is still found: see [projects in subdirectories](index.md#projects-in-subdirectories). ## How a coverage profile becomes line coverage Go does not report lines. It reports **blocks**: ```text mode: set github.com/org/repo/internal/names/names.go:85.29,86.13 1 1 github.com/org/repo/internal/names/names.go:86.13,88.3 1 0 ``` Each record is a span of source (`startLine.startCol,endLine.endCol`), how many statements it holds, and how often it ran. Two things have to happen before that can be compared with a pull request's diff. **The import path comes off.** The file is named by its module path, which the diff never carries. Brimyr reads the module path from `go.mod` (every `go.mod` in the repo, longest first, so a nested module claims its own files), and maps `github.com/org/repo/internal/names/names.go` back to `internal/names/names.go`. **The span is trimmed to its statements.** A function's first block starts *at* its opening brace, on the `func` line, and an `if` body's block ends just past its closing brace. Counting every line of every span would put the `func` signature, every lone `}`, and every blank or comment line inside a body into the patch-coverage denominator. Measured on a real operator codebase, 16% of the lines inside blocks were a lone closing brace. Brimyr reads the source and trims each block past braces and whitespace at both ends, then drops blank and `//` lines. What is left is a statement metric again. On this function ```go func Add(x, y int) int { // a comment inside the body if x > 100 { return 0 } return x + y } ``` tested with a small `x`, Brimyr counts three lines (the `if`, the early `return`, the final `return`), two of them covered: 66.7%, the same figure `go test -cover` prints. Counting the whole spans would have said 71%. A block with no statements (`func Empty() {}`) is skipped, and a line two blocks share (`} else {`) is covered if either block ran. ## Untested packages are in the denominator Since Go 1.22, `go test -coverprofile ./...` includes packages that have no test files, every block at count 0. A brand-new package nobody tests therefore counts as **uncovered** rather than vanishing from the denominator, which is the honest answer and not the default for every ecosystem. ## Supplying a profile yourself A profile produced elsewhere in your pipeline can be handed over directly. Brimyr recognises one by its first line (`mode: set`, `count` or `atomic`), whatever the file is called: ```yaml with: coverage_file: 'coverage.out' ``` `coverage.out:gocover` names the format explicitly. SonarQube receives the same profile under `sonar.go.coverage.reportPaths`. ## What Brimyr does not do - **No `-race`.** It needs cgo and a C toolchain the runner may not have, and it measures nothing extra. Keep it in your own CI. - **No `-coverpkg`.** Each package is measured by its own tests, which is what `go test -cover` reports and what SonarQube's documentation runs. Coverage one package gets from another package's tests is not counted. If you want it, run your own command and pass the profile with `coverage_file`. --- # HTML coverage report Source: docs/html-report.md URL: https://docs.magmamoose.com/brimyr/html-report/ Brimyr can render the coverage it produced as a **browsable HTML report** and upload it as a build artifact, so a repo with no SonarQube still has something to look at. ```yaml - uses: magmamoose/brimyr@v1 with: html_report: 'true' ``` Download `brimyr-coverage-html` from the run's artifacts and open `index.html`. ## It wraps ReportGenerator [ReportGenerator](https://github.com/danielpalme/ReportGenerator) (Apache-2.0) does the rendering. That is not a shortcut. It is the only sane choice: it **sniffs each input file independently**: by XML root element, or a `TN:`/`SF:` prefix for lcov, so the whole list Brimyr produced goes over in one `-reports:` argument with no branching on format. A polyglot repo gets **one** report covering its Cobertura and its lcov together. Every alternative is single-format. `genhtml` is lcov-only, `coverage html` needs the binary `.coverage` file, and JaCoCo's own CLI wants `.exec` plus compiled classes rather than the XML. Any of them would mean a converter per ecosystem and three moving parts instead of one. ## Why an artifact and not a URL A short-lived hosted page sounds better and is worse. It means storage, auth, expiry and a leak surface, coverage percentages, repo names and the directory tree of a private repo, in exchange for saving a download. The artifact gives you almost all of the value and none of that. ## Cost ReportGenerator is framework-dependent .NET; there is no self-contained build. * **GitHub-hosted runners**: free. The .NET SDK is preinstalled on all three images, so `dotnet tool install --global` needs no project and no setup step. * **Container or self-hosted runners**: add `actions/setup-dotnet` before Brimyr. Without it Brimyr emits a warning and skips the report; the gate is unaffected. This is why `html_report` is **off by default**. It is not a price to charge a Python repo that never asked for it. ## Failure isolation Same contract as the SonarQube leg: a missing renderer, a missing runtime or a malformed report is a `::warning::` and the run continues. **The gate never depends on it.** !!! note "Brimyr renders its own PR comment" ReportGenerator has a `MarkdownSummaryGithub` report type, and Brimyr deliberately does not use it: it is whole-project: the opposite of what this tool gates on, and the free build writes a `reportgenerator.io/pro` upsell row into the table. The PR comment stays Brimyr's own. --- # Brimyr Source: docs/index.md URL: https://docs.magmamoose.com/brimyr/ **Brimyr is quality assurance for a pull request: it gates on the coverage of the lines that PR changed, and on the net-new quality findings that PR introduced.** Point it at any repo and it figures out the rest, it detects the ecosystem, runs the test suite with coverage instrumentation on, and gates on the coverage of the changed lines (diff-cover semantics), blocking below a threshold (default **80%**). Pre-existing uncovered code never blocks. Turn `quality: 'true'` on and the same run also classifies the linters' findings against that same diff. One job, one summary, one PR comment. ```yaml - uses: magmamoose/brimyr@v1 # that is the whole configuration ``` That is the part nothing else does. Patch-coverage *maths* is well-trodden; what every other tool has in common is that **you bring your own coverage report**: you work out the right test command for each language and wire it per repo. Across an estate in several languages, that wiring *is* the project. Brimyr is **quality assurance**; [Chargate](https://github.com/MagmaMoose/chargate) is **security assurance**, and [Diatreme](https://github.com/MagmaMoose/diatreme) builds and releases. Brimyr and Chargate are twins, not competitors, Chargate gates net-new *security* findings, Brimyr gates coverage **and** net-new *quality* findings. The line between them is the **subject**, not the tool: Brimyr's quality half runs Chargate's net-new engine rather than growing its own ([see below](#coverage-is-half-of-quality-assurance)). ## What it detects | Ecosystem | Markers | Test command | Coverage format | | --- | --- | --- | --- | | Python | `pyproject.toml`, `setup.py`, `tox.ini` | `pytest --cov` | Cobertura | | JavaScript / TypeScript | `package.json` | `jest`; `vitest run` when the repo uses vitest; `c8 npm test` when its `test` script runs `node --test` | lcov | | .NET | `*.sln`, `*.slnx`, `*.csproj`, `*.fsproj`, `*.vbproj` | `dotnet test --collect` | Cobertura | | Java / JVM | `pom.xml` | `mvn ... jacoco:prepare-agent test jacoco:report` | JaCoCo | | Go | `go.mod` plus a `_test.go` file | `go test -coverprofile=coverage.out ./...` | [Go coverage profile](go.md) | | Shell | a real `*.bats` file | `bats --recursive` over the directories holding them | Cobertura, [only if `kcov` is installed](shell.md) | A polyglot repo matches more than one, produces one report per language, and Brimyr merges them into a single number. Override any of it with `ecosystem`, `test_command`, or `coverage_file`. See [Action reference](action.md). ### Projects in subdirectories Some repos keep nothing at the root: a `backend/` with its own `pyproject.toml`, a `frontend/` with its own `package.json`, an `agent/` beside them. When the root matches no ecosystem, Brimyr looks up to three directories down for Python, JavaScript / TypeScript and Go projects, and treats each one it finds as a project of its own: - its dependencies are installed and its tests run **from its own directory**, exactly as a CI job with `working-directory: backend` would run them; - its report names files relative to that directory, and Brimyr puts the directory back (`app/main.py` becomes `backend/app/main.py`), so two projects that both have a `src/index.ts` stay two files; - the summary and every log line name it: `Python in backend/`, `JavaScript / TypeScript (vitest) in frontend/`. The search is a fallback, not a second pass. A repo whose root is detected today is run exactly as before, sub-projects and all, so no existing verdict changes underneath it. It stops at a project it has found (a workspace whose root script runs every package is not run twice), and never enters `node_modules`, `vendor`, hidden directories, fixtures (`fixtures`, `testdata`) or build output (`dist`, `build`, `target`). Java and .NET are not searched for: their markers prove a build exists, not that anything tests it, and running `mvn test` in every library module would turn a green repo red. A forced `ecosystem` always runs at the root. The same run can also ship coverage to SonarQube for the long-run trend, non-blocking. It does that only when `sonar_url` and `sonar_token` are both set and a `sonar-scanner` is on PATH; otherwise the step is skipped with a reason and nothing fails. See [SonarQube](sonarqube.md). ## Three faces, kept separate - **Blocking: the patch-coverage gate.** The percentage of *changed executable lines* the tests covered, diffed against `merge-base(base, head)`. Blocks below the threshold. Computed **locally**; no SonarQube involvement. - **Report-only by default: the net-new quality gate.** The quality findings *this PR introduced*, classified against that same diff. Off unless `quality: 'true'`, and even then `quality_fail_on` defaults to `none`, it counts and reports, and blocks only at a SARIF level you choose. - **Non-blocking, and opt-in: one `sonar-scanner` run.** Off unless `sonar_url` and `sonar_token` are set. Sonar's native quality analysis plus ingesting the coverage file → SonarQube, for history and the coverage/quality trend. Sonar derives new-vs-old code itself (its New Code Period); you never feed it "net-new". The two gates share one job, one summary and one PR comment, and the job exits on the worse of the two. !!! danger "Broken test run ≠ 0% coverage" If the test run failed, timed out, or produced no coverage, that is an **error (build red)**: never reported as "0% patch coverage" that hard-fails the gate. ## Coverage is a byproduct of the test run You run the tests *with instrumentation on* (`pytest --cov`, `jest --coverage`, `dotnet test --collect`) and that single run emits the coverage file. There is no separate "measure coverage" pass. Brimyr detects the ecosystem and runs the right command; polyglot repos produce **one coverage file per language**, merged. ## Coverage is half of quality assurance The other half is what the linters say. Set `quality: 'true'` and Brimyr also gates the **net-new quality findings** this pull request introduced, classified against the same diff the coverage gate uses, and folds that verdict into the same job summary and the same PR comment as coverage. The job exits on the worse of the two. Brimyr implements none of that classification. Chargate already owns a finished net-new engine, so Brimyr does not import it, vendor it or re-implement it, it **calls it as a nested step**: a shared package would buy version skew, lockfile drift and a diamond dependency inside one job, and a subprocess in its own environment has none of those properties. That is [ADR 0002](https://github.com/MagmaMoose/brimyr/blob/main/.claude/decisions/0002-quality-gate-calls-chargate.md). Chargate's own `fail_on` is pinned to `none` there, so it can never set the job's exit code, it reports, Brimyr decides. `quality_fail_on` defaults to `none`, which makes it **report-only** until you say otherwise. MegaLinter's quality half over a mature repo is far denser than its security half, and a first PR that goes red with hundreds of findings is how a gate becomes decoration nobody reads. Ship it reporting, measure a release cycle, then pick a SARIF level. See [Quality findings](quality-findings.md). A scan that never ran is not a clean one. The nested step is `continue-on-error`, so it cannot take the coverage gate down with it, but Brimyr goes by that step's `outcome` rather than by the counts file it may have left behind, a failed scan can leave a well-formed row of zeros on disk, which is what a clean PR looks like, and reports a tool error instead: exit **2**, `quality_gate_result` `error`. Nor is a scan that *completed* necessarily a full one: when Chargate could not start a linter it says which, and Brimyr states that shortfall beside the count rather than passing a smaller scan off as the whole answer. ## Two surfaces, one CLI | Surface | What it is | When to use | | --- | --- | --- | | **Composite action** | `action.yml` | When you compose your own steps. | | **pre-push hook** | `.pre-commit-hooks.yaml` (`brimyr` hook) | Catch a shortfall locally before pushing. | See [Setup & usage](setup.md) to wire one up, [Action reference](action.md) for every input and output, [Architecture](architecture.md) for how it fits together, [Patch coverage](patch-coverage.md) for the precise classification rules, and [Quality findings](quality-findings.md) for the other half. When something goes wrong, [Troubleshooting](troubleshooting.md). ## The PR comment Opt in with `pr_comment` and the verdict lands on the pull request as **one** comment, updated in place on every push rather than stacked, and carrying **both halves**: the coverage number, the threshold and the changed lines the tests never executed, then the net-new findings count, its per-level breakdown and what it blocks on. Set `token_broker_url` as well and it is authored by **Brimyr[bot]** rather than the shared `github-actions[bot]`. Neither can fail the gate: a comment is a convenience, and a convenience must never turn a green PR red. See [PR comment](pr-comment.md). ## Modes - **PR events** → run tests → patch-coverage gate, plus the quality gate when `quality: 'true'` → ship to SonarQube. - **Push to default branch / scheduled** → run tests → ship to SonarQube as the trend baseline → **no** gate. Quality findings are still counted and shown; a baseline run has no diff to gate against, and says so. `mode: auto` (default) picks this from the event; force it with `mode: pr|baseline`. ## Why patch coverage? Requiring 80% coverage on a whole legacy codebase is a non-starter; ignoring coverage on new code lets it rot. Patch coverage splits the difference: hold *new and changed* lines to a bar, leave the back-catalogue alone. - **Gate** on what *this PR* changed → actionable, no legacy-debt noise. - **Ship** the full coverage to SonarQube → the long-run trend, and Sonar's own quality gate (not to be confused with Brimyr's net-new quality gate above). ## Rollout cost is the point The comparison that matters is not "which tool computes coverage", it is what it takes to switch this on across an estate. For eight repos in four languages: | | SonarQube | Brimyr | | --- | --- | --- | | Server to host / licence | a server, or a per-committer plan | none | | Per repo | a project, a token, a quality gate, workflow YAML | one `uses:` line | | Per ecosystem | a *different scanner*, `dotnet sonarscanner` for .NET, `mvn sonar:sonar` for Java, the CLI for the rest | detected | | Producing the coverage report | yours to work out, per language | it runs your tests | | Per-PR diff coverage | Developer Edition and above | included | ## License Apache-2.0. --- # Java / JVM Source: docs/java.md URL: https://docs.magmamoose.com/brimyr/java/ Brimyr detects a Maven repo from `pom.xml`, runs ```bash mvn -B org.jacoco:jacoco-maven-plugin:prepare-agent test org.jacoco:jacoco-maven-plugin:report ``` and reads every `**/target/site/jacoco/jacoco.xml` it produced. The plugin goals are invoked **by coordinate**, so this works on a `pom.xml` that has no JaCoCo plugin configured at all, nothing has to be added to the build to start gating. If your `pom.xml` already binds `prepare-agent` and `report` to the build lifecycle, a plain `mvn -B verify` is enough and you can say so: ```yaml with: test_command: 'mvn -B verify' ``` ## JaCoCo XML is not Cobertura This is the one thing to get right. JaCoCo emits **its own XML format**, and it shares the `.xml` extension with Cobertura: ```xml ``` Brimyr tells the two apart by the **root element** (`` vs ``), not by the file name, so `coverage_file: target/site/jacoco/jacoco.xml` just works. !!! danger "Why the extension is not enough" Parsing a JaCoCo file as Cobertura does not fail. It finds no ``, returns an **empty report**, and every changed Java file is then a file the report never mentions. Those files leave the denominator, and the gate gladly reports **100% over completely untested code**. If a tool ever hands you a coverage number you cannot reconcile, this is the first thing to check. You can always be explicit: ```yaml with: coverage_file: 'build/reports/jacoco/test/jacocoTestReport.xml:jacoco' ``` ### Partially covered lines count as covered A line with `ci>0` *and* `mi>0` is usually a short-circuited boolean. JaCoCo's own LINE counter calls it covered and so does diff-cover, so Brimyr does too, otherwise every `a && b` in a pull request would count against you. ## Multi-module reactors A multi-module build writes **one report per module**, each under that module's own `target/`. Brimyr ingests all of them and merges them covered-wins, so a class exercised by a sibling module's tests comes out covered. This matters for the same reason it matters on a multi-project .NET solution: a *missing* report is indistinguishable from *nothing coverable changed*, so dropping one silently removes that module's changed lines from the denominator and inflates the number. If a reactor build reports a suspiciously round 100%, count the reports: ```bash find . -path '*/target/site/jacoco/jacoco.xml' | wc -l # expect one per module with tests ``` Aggregating into a single report first (the `report-aggregate` goal) also works, point `coverage_file` at it and Brimyr will not run any tests. ## Source paths JaCoCo names files as `/`, `nl/example/isam/case/CaseService.java`, which is **source-root-relative** and so is missing the `backend/src/main/java/` prefix that `git diff` reports. Brimyr reconciles that by suffix matching; there is nothing to configure. ## Gradle `build.gradle` and `build.gradle.kts` are recognised as Java markers but are **not** auto-detected, because the built-in command is `mvn` and running it in a Gradle repo would fail the run and turn the build red. The JaCoCo parser is shared, only the invocation differs, so name the command: ```yaml with: ecosystem: 'java' test_command: './gradlew test jacocoTestReport' coverage_file: 'build/reports/jacoco/test/jacocoTestReport.xml' ``` Gradle's `jacocoTestReport` task writes HTML by default; make sure XML is on (`reports { xml.required = true }`). ## Excluding generated code The JVM equivalent of the `.NET` problem: generated sources, MapStruct/Lombok output, JAXB and OpenAPI stubs. `exclude` drops matching **changed files** from the denominator entirely. They are not counted as covered, they simply do not count: ```yaml with: exclude: "**/generated-sources/**,**/target/**,*_MapperImpl.java,**/generated/**" ``` Globs match the repo-relative, forward-slash path and `*` crosses `/`, so a pattern catches a folder at any depth without you having to know how deep it sits. JaCoCo's own `` configuration works too, and is preferable when you want the class gone from *all* reporting rather than just from the gate: anything absent from the report is absent from the denominator already. ## Full example ```yaml name: Coverage on: [pull_request] permissions: contents: read pull-requests: write # for the PR comment jobs: coverage: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 with: { fetch-depth: 0 } # patch coverage needs the merge-base - uses: actions/setup-java@v4 with: { distribution: 'temurin', java-version: '21', cache: 'maven' } - uses: magmamoose/brimyr@v1 with: checkout: 'false' threshold: '80' pr_comment: 'true' exclude: "**/generated-sources/**,**/generated/**" ``` Start with `mode: baseline`, which computes and reports the number without gating. Once the numbers look right on real pull requests, switch to the default and pick a threshold from what you actually saw. ## SonarQube `sonar.coverage.jacoco.xmlReportPaths` is set from the reports Brimyr found, so the same run that gates the pull request can also feed the SonarQube trend, non-blocking, as always. !!! warning "Java needs `sonar.java.binaries`" `sonar-scanner -Dsonar.sources=.` over a Java repo fails outright with *"please provide compiled classes with sonar.java.binaries"*. Brimyr cannot infer it, so rather than run a scan that cannot succeed it **skips with a warning** until you supply it: ```yaml with: sonar_url: https://sonar.example.com sonar_args: '-Dsonar.java.binaries=**/target/classes' ``` Sonar also documents that the CLI scanner should not be used for Maven or Gradle projects at all, `mvn sonar:sonar` is the supported path and will give a better analysis. Brimyr's patch-coverage gate is unaffected either way: it reads the JaCoCo reports directly and never talks to SonarQube. --- # Patch coverage Source: docs/patch-coverage.md URL: https://docs.magmamoose.com/brimyr/patch-coverage/ **Patch coverage** is the fraction of *changed executable lines* the test run covered. A line counts iff it is **changed by the PR**: in an added/modified hunk on the new side, diffed against `merge-base(base, head)`, **and** the coverage tool considers it **executable**. The merge-base survives base-branch rebases and force-pushes. ```text covered changed-executable lines patch % = ───────────────────────────────────── × 100 all changed-executable lines ``` ## Classification rules | Case | Behaviour | Configurable | | --- | --- | --- | | Brand-new file | every executable line counts | none | | Modified hunk | only the changed executable lines count | none | | Pre-existing uncovered line in a changed file | excluded: never penalised | none | | Blank line / comment / brace | excluded (not in the coverage report) | none | | Changed file the report never mentions (a doc, an untested new file) | contributes nothing (diff-cover behaviour) | none | | Renamed / copied file | matched by head path; changed lines line-matched | none | | Deleted file | dropped | none | | Nothing coverable changed (docs-only PR) | **vacuous pass** (100%) | none | | Broken / empty test run | **tool error (exit 2)**, not 0% | none | | Missing merge-base / shallow clone | **fails loudly**: needs `fetch-depth: 0` | none | The denominator is deliberately *changed-and-executable*: coverage tools only report executable lines, so blank lines and comments fall out naturally, and a genuinely untested new file the suite never imported isn't in the report, it contributes nothing rather than tanking the score. This matches `diff-cover`. ## Why a broken run is not 0% If the test command exits non-zero or emits no parseable coverage, reporting that as "0% patch coverage" would be actively misleading: it conflates *no signal* with *bad signal*. Brimyr treats it as a **tool error (build red, exit 2)** instead, the same philosophy as a broken security scanner being a tool error, not a finding. ## Path matching Coverage-report paths and `git diff` paths rarely match byte-for-byte: coverage tools emit absolute paths, ``-rooted paths, or monorepo-prefixed paths, while the diff is repo-relative. Matching therefore falls back: 1. **exact** normalized-path match; 2. a coverage path that **ends with** the diff path (absolute coverage path); 3. a diff path that ends with a (shorter) coverage path. Pass `strip_prefix` (action) / `--strip-prefix` (CLI) to peel a known root (a monorepo subdir, a Cobertura ``) before matching. ## The threshold `threshold` (default **80**) is the patch-coverage percentage below which the gate blocks. `--no-gate` (or baseline mode) makes the run report-only: coverage is still computed and shipped to SonarQube, nothing blocks. ### The sample-size floor `min_lines` (default **20**) is the number of changed executable lines below which the threshold is **not applied**. Below that the percentage is too coarse to act on: one uncovered line out of three is 67%, which fails an 80% gate while telling you nothing useful about the change. SonarQube applies exactly this rule at exactly this number: *"the conditions on coverage are ignored until the number of new lines to cover is at least 20"*, so matching it stops a Brimyr verdict and a SonarQube verdict from disagreeing on small pull requests. !!! warning "It is a hole, deliberately" A 19-line change with no tests at all passes. That is the same *shape* of hole as the one that makes a project-level coverage gate unusable, just much smaller. The difference is that Brimyr **says so**: ```text ⚪ Only 3 changed executable line(s), below the 20-line minimum, so the 80.0% threshold was not applied (patch coverage was 33.3%). ``` SonarQube applies the same exemption silently, which is how a team ends up believing small PRs are gated when they are not. Set `min_lines: '0'` to gate every diff. A diff with **no** coverable lines at all is still a separate case, a vacuous pass, the same as a docs-only change, and is reported as such rather than as "too small". !!! tip "Local gate, Sonar trend" The gate is computed **locally** from the coverage file, it never depends on SonarQube. The `sonar-scanner` run is a separate, non-blocking step that feeds Sonar the same coverage for the long-run trend. ## Total coverage, reported next to it Brimyr also reports an **overall** coverage figure alongside the patch number. It is reported and never gated on: the same split Chargate uses, where the build blocks on net-new findings but the full SARIF still ships. ```text | Patch coverage | **100.0%** | | Covered / changed executable lines | 2 / 2 | | Total coverage (measured files) | 7.3% | | Covered / executable lines across 2 file(s)| 3 / 41 | ``` The gate is unchanged: `gate_result`, the exit code and `failed` all still derive from patch coverage alone. A codebase at 7% total does not fail a well-tested PR. ### What the number actually means **"Total coverage (measured files)", not "project coverage".** A coverage report only mentions files the test run loaded, so a module no test imports is absent from the report and therefore absent from this denominator, which means **adding a brand-new untested file can raise it**. Say "coverage of what we measured". It also will not match SonarQube's figure, which applies its own `sonar.coverage.exclusions` and source scoping. That is fine and expected: this number is the at-a-glance trend inside a PR comment, and **SonarQube owns the authoritative long-run total**. If you only want one number, use Sonar's. !!! note "Nothing measured is not 100%" When a run measures no executable lines, total coverage is reported as *absent*, not as 100%. Patch coverage's vacuous 100% exists because a docs-only PR genuinely has nothing to cover; a total of 100% over zero lines would just be a lie that looks like good news. The `total_coverage` action output is an **empty string** in that case, never `0.00`. ### It obeys `exclude` The same globs that drop generated code from the patch denominator drop it here. They have to: a comment showing a patch figure that ignored the EF migrations next to a total that didn't would disagree by twenty points and read as a bug in the tool. ### Baseline mode In baseline (report-only) mode the patch rows are **omitted** entirely rather than printed as a vacuous `100.0% (0 / 0)`, because baseline computes patch coverage against an empty diff. Baseline plus total coverage is the useful combination for a repo with no PR CI to attach a gate to: run it on pushes to the default branch, block nothing, and start producing a trend. --- # PR comment Source: docs/pr-comment.md URL: https://docs.magmamoose.com/brimyr/pr-comment/ Brimyr can post its verdict as a comment on the pull request, so a reviewer sees the numbers without opening the job log. It is **opt-in** (`pr_comment: 'true'`), there is exactly **one comment per marker**: and the action's path uses one, and it is **never** able to change the gate verdict. The comment body is the same Markdown Brimyr writes to the GitHub job summary: one renderer, so the two can never disagree about the run they are describing. ## One comment, two blocks On the normal PR path: the composite action, which runs `brimyr ci` : the single comment carries **both halves of the run**, each under its own heading: | Heading | What it reports | Rendered by | | --- | --- | --- | | `## Brimyr: Quality Assurance` | patch coverage: the percentage, the threshold, and the changed lines the tests never executed | `report.render_summary` | | `## Brimyr: Net-new findings` | the quality findings this PR introduced, and whether any of them block | `report.render_quality_summary` | The second block appears only when the quality half ran (`quality: 'true'`, or `--quality-counts` / `--quality-scan-broken` on the CLI). Coverage alone is the default, and then the comment is just the first block. Consolidating them is the point rather than a convenience: two blocks in one comment is why `brimyr ci` takes the quality inputs at all, instead of the run posting a second comment of its own. ## What the coverage block looks like A failing gate renders the number, the threshold, and the changed lines the tests never executed: the only three things you need to know what to write a test for: ```markdown ## Brimyr: Quality Assurance **Mode:** `pr` · **Gate:** `fail` · **Ecosystem:** Python | Metric | Value | |--------|-------| | Patch coverage | **72.0%** | | Covered / changed executable lines | 18 / 25 | | Threshold | 80.0% | ❌ **Patch coverage 72.0% is below the 80.0% threshold.** Uncovered changed lines: - `src/app/billing.py`, 41, 42, 43, 57 - `src/app/invoices.py`, 12, 13 ``` A passing gate collapses to a single line under the same table (`✅ Patch coverage 92.3% meets the 80.0% threshold.`), and a PR that changed nothing coverable says so (`✅ No changed executable lines to cover, vacuous pass.`). A **broken test run** replaces the table entirely with a blockquote saying the tests failed or produced no coverage and that this is a tool error, **not** 0% patch coverage: the same distinction the [gate itself makes](patch-coverage.md#why-a-broken-run-is-not-0). Reporting a crashed suite as "0%" in a comment would be the most misleading thing Brimyr could put on a PR. Long lists are capped so the comment stays readable on a big refactor: at most 15 line numbers per file, at most 20 files, with a `… and N more file(s)` tail. If a `sonar-scanner` run happened, its outcome is appended as a final `**SonarQube:**` line. ## What the net-new findings block looks like ```markdown ## Brimyr: Net-new findings **Gate:** `report-only` · **Blocks on:** `none` | Metric | Value | |--------|-------| | Net-new findings | **16** | | Pre-existing (never blocking) | 214 | | Net-new by level | error=2, note=3, warning=11 | 📋 Report-only, `quality_fail_on` is `none`, so findings are counted and shown but nothing blocks. ``` `Gate:` is `pass`, `fail`, `report-only` or `error`, and `Blocks on:` always names the threshold that was in force. Under the default `quality_fail_on: none` the run cannot block, so `Gate:` reads `report-only` no matter how few findings there are, a clean PR and a loaded one are told apart by the count, never by the word. (The `quality_gate_result` action **output** has no `report-only` value and prints `pass` for both, which is why `quality_fail_on` is echoed as an output of its own.) A `Blocking at ` row joins the table once the threshold can actually fire, and a `Suppressed in source (never blocking)` row appears when there are any. The closing line names the verdict. A run that cannot block: the default threshold, or baseline mode whatever the threshold, gets the `📋 Report-only,` line above instead. Once the threshold can fire it is one of: ```text ❌ **2 net-new finding(s) at or above `error`.** ✅ 16 net-new finding(s), none at or above `error`. ✅ No net-new quality findings. ``` Findings themselves are listed in a collapsed `
` block as `path:line [rule]`, capped at 20 with a `… and N more` tail. Summary only, there are no per-finding inline review comments, deliberately: Chargate posts those because security findings are sparse, and a first PR carrying two hundred inline comments is how a gate gets switched off. !!! danger "A broken scan is not zero findings" When the quality scan did not complete, the table is replaced wholesale, exactly as a broken test run replaces the coverage table: > ❌ **The quality scan did not complete**: Chargate errored or produced no > report. This is a tool error (build red), **not** zero net-new findings. A run that scanned nothing and a repository with nothing to find produce the same number, so the comment must not let them look the same. If Chargate ran but could not start every linter, a `⚠️ **The scan was not complete**` line names the ones it skipped, next to the count it qualifies. It never blocks. See [Quality findings](quality-findings.md#a-scan-that-completed-is-not-necessarily-a-full-one). ## Exactly one comment, updated in place The comment carries a hidden HTML marker. There are two of them, and **one comment per marker**: | Marker | Owned by | Carries | | --- | --- | --- | | `` | `brimyr ci` : the composite action, i.e. the normal PR path | the consolidated comment: coverage, plus net-new findings when that half ran | | `` | a standalone `brimyr lint` run | net-new findings only | Invisible in rendered Markdown, but it is how a later run finds the comment it already owns. Every run lists the PR's comments, and if one carries **its** marker it is `PATCH`ed; otherwise a new one is `POST`ed. So a PR that gets pushed to forty times ends with one comment showing the current number, not forty comments and an archaeology problem. !!! warning "Two markers, because the halves can run separately" `brimyr lint` gets its own marker rather than a second body under the first because the two subcommands can run in either order, or only one of them, a shared marker would mean whichever ran last silently erased the other's verdict. Two comments is the cost of running the halves as separate steps; passing the same two Chargate files to `brimyr ci` is how you get one. !!! warning "The marker is namespaced on purpose" It is `brimyr:pr-summary`, not a generic "our comment" match. Chargate comments on the same PRs; a looser match ("any comment we can parse") would have the two tools overwriting each other's summaries. One owner per surface. The lookup is paginated and capped at 20 pages (2000 comments), so a pathological thread cannot spin the job forever, past that cap Brimyr simply posts a new comment rather than hanging. ## Wiring it up ```yaml permissions: contents: read pull-requests: write # ← required; the default `read` cannot comment jobs: brimyr: runs-on: ubuntu-latest steps: # ... checkout with fetch-depth: 0, install your test deps ... - uses: magmamoose/brimyr@v1 with: pr_comment: 'true' # github_token defaults to the job token → comments as github-actions[bot] ``` `github_token` defaults to the job's `GITHUB_TOKEN`. Point it at a different token only if you want a different identity and are not using the broker below. Outside a pull request there is nothing to comment on, on a push/baseline run the step is a no-op, no matter what `pr_comment` says. ## It never fails the gate A missing token, a `403` from a job with only `pull-requests: read`, a network blip, GitHub being down, every one of them is reported and then dropped. The run prints a line like ```text brimyr: PR comment updated as Brimyr[bot] ``` on stderr (suppressed by `--quiet`) and the exit code is whatever the gate, or, when both halves ran, the worse of the two, decided. This is the same contract the [SonarQube step](setup.md#sonarqube) follows: a comment is a convenience, and a convenience must never be able to turn a green PR red. The corollary is worth stating plainly, because it is the failure people actually hit: **if you forget `pull-requests: write`, nothing goes red.** The comment just never appears, and the reason is one line in the step log. ## Authoring as Brimyr[bot] By default the comment is authored by `github-actions[bot]`, shared with every other workflow in the repo. Set `token_broker_url` and it is authored by **`Brimyr[bot]`** instead, its own identity and avatar, so the comment is recognisable at a glance and distinct from Chargate's on the same PR. ```yaml permissions: contents: read pull-requests: write id-token: write # ← required; without it there is no OIDC token to exchange # ... - uses: magmamoose/brimyr@v1 with: pr_comment: 'true' token_broker_url: https://broker-brimyr.magmamoose.com ``` What happens inside that one job: 1. The job asks the Actions runtime for an **OIDC token** with `aud=brimyr`. The runtime only issues one when the job declares `id-token: write`, and the token is signed by GitHub, a runner cannot forge one, which is the entire basis of the design. 2. Brimyr `POST`s it to the broker along with the `owner`/`repo` being commented on. 3. The broker verifies the signature, the issuer, the audience, and that the token's own `repository` claim **matches the repo asked for**: then mints a GitHub App installation token scoped to that one repo with `pull_requests: write` and nothing else. The minted token is short-lived, is handed straight to the comment client, and is never printed, logged, written to a step output, or put in the job summary, not even truncated. !!! danger "It fails soft, and it fails silently" Every failure path, no OIDC token, broker down, App not installed, DNS gone, falls back to `GITHUB_TOKEN`. The comment still posts, the gate verdict is untouched, and **the only symptom is the byline reverting to `github-actions[bot]`**. Nothing turns red. Nobody reads a byline. That is the right trade, a broken broker must never cost anyone a merge, but it means the broker can be dead for months without anyone noticing. This is exactly why `.github/workflows/broker-smoke.yml` exists and is not optional: a weekly (Mondays, 07:23 UTC) real mint against the deployed broker, plus `workflow_dispatch` to re-arm it, because GitHub disables scheduled workflows on a repo after 60 days of inactivity and a silently disabled smoke test is exactly as useless as no smoke test. When the mint fails, the reason is appended to that same stderr line: the byline is the symptom; this is the diagnosis: | Log message | What to fix | | --- | --- | | ``no Actions OIDC token, does the job declare `permissions: id-token: write`?`` | Add `id-token: write` to the job. By far the most common cause. | | `broker returned 401 (invalid_oidc)` | The token failed signature/issuer/audience verification. | | `broker returned 403 (repo_mismatch)` | The OIDC token's `repository` claim is not the repo the mint asked for. | | `broker returned 403 (repo_not_allowed)` | The repo is not on the broker's allowlist. | | `broker returned 403 (app_not_installed)` | The Brimyr GitHub App is not installed on that repo. | | `broker returned 502 (mint_failed)` | GitHub refused the mint; check the App's private key and permissions. | | `broker unreachable (URLError)` | DNS, TLS, or the broker being down. The exception *class name* is deliberate, nothing request-derived is logged, because the OIDC token travels in that request. | Only that fixed vocabulary of broker error codes is ever echoed. A broker that decided to reflect the request back could otherwise put the OIDC token straight into your build log. ## The broker is a separate deployable `broker/` in this repository is the AWS Lambda service that does the minting. It is **not** part of the `brimyr` CLI: it has its own `pyproject.toml`, its own lockfile, its own Ruff config and its own CI job, and nothing under `src/brimyr` imports anything from it. The CLI stays stdlib-only and dependency-free; `broker_client.py` speaks to the broker over HTTPS like any other stranger. **You do not need to run one.** Leave `token_broker_url` unset and everything on this page still works, only the byline differs. If you do want to stand one up, its documentation lives with the code: - **[`broker/README.md`](https://github.com/MagmaMoose/brimyr/blob/main/broker/README.md)**: architecture, the LocalStack loop (`make -C broker all`), what a local run can and cannot prove, and the full go-live runbook: seeding the App private key into SSM by hand, installing the App, applying the Terraform, and the post-apply checks. That runbook is deliberately **not** duplicated here. It is operational detail tied to specific account IDs, key fingerprints and Terraform outputs, it belongs next to the Terraform it describes, where a change to one is a change to the other in the same diff. --- # Quality findings Source: docs/quality-findings.md URL: https://docs.magmamoose.com/brimyr/quality-findings/ Coverage is half of quality assurance. The other half is **the findings a linter reports on the lines this PR changed**: and Brimyr gates on those the same way it gates on coverage: net-new only, pre-existing never blocks. ```yaml - uses: magmamoose/brimyr@v1 with: quality: 'true' ``` That runs [Chargate](https://github.com/MagmaMoose/chargate) as a nested step, MegaLinter's quality linters, then net-new classification against the PR diff, and folds its verdict into the same job summary and the same PR comment as coverage. **Report-only by default**; see [Start report-only](#start-report-only). `action.yml` pins `magmamoose/chargate@753df5673aa274e87596c0b88159dc4b0cce33d1` (v2.11.71), which carries the curated `quality` flavor added in v2.11.26, so `quality: 'true'` is usable. !!! note "If you pin Brimyr below v1.9.1, leave `quality` off" v1.9.0 shipped this feature against `chargate@a852f9c` (v2.11.25), which predates the flavor by seventeen seconds of release timing. On that pin the nested step fails, the counts file is never written, and Brimyr exits `2` with `quality_gate_result` set to `error` : the boundary working as designed, since a gate that cannot evaluate must not go green. It is inert rather than dangerous, but it is a red X on every PR. ## The boundary is the subject, not the tool **Brimyr is quality assurance. Chargate is security assurance.** Brimyr's quality half calls Chargate's engine, so both tools now emit Chargate-derived findings, but they are not the same findings and they are not two views of one thing. A *security* finding belongs to Chargate's gate; a *quality* finding belongs to this one. Read a finding by its subject, not by which tool surfaced it. ## Brimyr calls Chargate; it does not share a library with it Chargate already owns a finished net-new engine. `chargate filter-sarif` takes a SARIF report plus a base and head, and writes two files: the net-new-only SARIF, and a counts JSON. Brimyr does not import that engine, vendor it, or republish it. It **calls** it, and gates on what comes back. Every argument against extracting a shared package, version skew, `uv.lock` drift, a diamond dependency inside one job, mutating a consumer's interpreter, is an argument about imports into a shared interpreter. A subprocess in its own environment has none of those properties, and both tools keep `dependencies = []` literally rather than with an asterisk. **Chargate reports; Brimyr gates.** The nested step runs with `fail_on: none`, so it can never set the job's exit code, `brimyr ci` decides on its own threshold. ## The threshold is a SARIF level, not a severity band `quality_fail_on` takes one of `none`, `note`, `warning`, `error`, `any`, and blocks on net-new findings at or above it. | Value | Blocks on | | --- | --- | | `none` | nothing, report-only (the default) | | `note` | `note`, `warning`, `error` | | `warning` | `warning`, `error` | | `error` | `error` only: the equivalent of Chargate's `fail_on: high` | | `any` | every net-new finding, including ones the linter left unlevelled | Chargate's own gate speaks severity bands, and copying that vocabulary here would have been the obvious move. It would also have shipped a threshold that could never fire. Chargate gates on *per-result* verdicts, where a missing `security-severity` falls back to the SARIF level (error→high, warning→medium, note→low), so its `fail_on: high` works perfectly well over a quality scan. Brimyr never sees those verdicts. It sees the counts document, and there the two breakdowns are **not** interchangeable. !!! danger "`per_severity_net_new` is empty on a quality scan" That map is populated only from a real numeric `security-severity` property, which is a security-scanner convention that quality linters essentially never emit. `per_level_net_new` covers every result. A band-valued threshold read off this document would therefore match nothing, on every PR, forever, while looking exactly like a configured gate, so the threshold speaks the vocabulary the document actually carries. ## Start report-only `quality_fail_on` ships as `none` on purpose. MegaLinter's quality half over a mature repo produces hundreds of net-new findings on the first pull request, because "changed line touched by a formatter-opinionated linter" is a far denser event than "changed line with a security finding". The security flavor is sparse. The quality flavor is not. The failure mode is **abandonment, not error**: the first real PR goes red, someone sets the threshold back to `none` to unblock it, and six weeks later the gate is decoration nobody reads. No test catches that, so the defaults are the mitigation: 1. **Report-only out of the box.** Findings are counted and shown; nothing blocks. 2. **Summary comment only**: no per-finding inline review, and the listing caps at 20 with the remainder stated. Chargate posts inline comments because security findings are rare; these are not. 3. **Five linters, not a flavor's worth**: `GO_GOLANGCI_LINT`, `JAVASCRIPT_ES`, `JAVA_PMD`, `PYTHON_RUFF`, `TYPESCRIPT_ES`. Override with `quality_linters`. 4. **`error` before `any`.** Measure a release cycle, then pick a level. !!! note "There is no .NET linter in the set" No C#/VB.NET linter sets `can_output_sarif` at MegaLinter v10.0.0, so none of them can reach a SARIF-based gate. A .NET repo turning `quality` on will see zero findings, and that is the tooling, not the code. Said here rather than discovered six weeks later. ## One run, one comment Supplying the quality inputs does not add a second gate step or a second PR comment. The verdict is rendered into the same job summary and the same comment as coverage, a second heading, `## Brimyr: Net-new findings`, below the coverage block's `## Brimyr: Quality Assurance`, and the run's exit code is the worse of the two on the usual scale: `0` pass, `1` a gate blocked, `2` a tool error. What both blocks look like is on [PR comment](pr-comment.md#one-comment-two-blocks). A broken test run still reports `2` even when the quality half is clean, and a clean coverage number does not launder a blocking quality finding. ## It reads counts, not SARIF Brimyr gates on the counts JSON alone. The net-new SARIF is read only to list `path:line [rule]` strings in the summary: Brimyr builds no SARIF model and resolves no severity, because that is Chargate's job and a second implementation here is the coupling the process boundary exists to avoid. Three things are therefore hard errors (exit `2`), never a comfortable zero: * **A `schema_version` Brimyr does not recognise**: including a missing one, which is an older Chargate and the same version skew by another route. * **A counts document that disagrees with itself.** Every net-new result contributes exactly one SARIF level, so `per_level_net_new` must sum to `net_new_count`. When it does not, one of the two numbers is wrong and nothing on this side can tell which. * **A net-new SARIF holding a different number of results than the counts claim.** Chargate writes both files from one in-memory result, so a mismatch means one is stale or truncated. Each of these would otherwise present as "0 net-new findings", which is indistinguishable from a clean PR. **Check the denominator before believing a good result.** ## A scan that completed is not necessarily a full one An exit-`0` Chargate run is not proof the scan was **complete**. Chargate can decline to start a linter, no image for the runner's architecture, no SARIF output, the linter disabled, and still exit `0`. Whatever that linter would have reported is then simply absent from the count, and absent findings are exactly what a clean repository looks like. That shortfall is neither a failure nor a finding, so it is neither gated on nor swallowed: it is **said out loud**. Chargate names the linters it could not run, the action forwards that list as `--quality-scan-note` (`--scan-note` on `brimyr lint`), and the summary states it beside the number it qualifies: > ⚠️ **The scan was not complete**: these linters did not run: … Anything they would > have reported is missing from the count above. It never blocks. A missing linter image is not the pull request's fault, and failing on one would be its own kind of noise, but a count produced by half a scan must not be presented as though it were the whole answer. ## Running it without the action `brimyr lint` is the same gate over files you already have, Chargate's output from a previous step, or a run of `chargate filter-sarif` by hand: ```sh chargate filter-sarif --sarif full.sarif --base "$BASE" --head HEAD \ --out net-new.sarif --counts-json counts.json --no-gate brimyr lint --counts counts.json --findings net-new.sarif --fail-on error ``` Run on its own it posts under its **own** PR comment marker, so it cannot overwrite the coverage comment. The consolidated single comment comes from passing the same two files to `brimyr ci` instead, which is what the action does. See [CLI reference](cli.md). ## Failure isolation, and its limit The nested Chargate step is `continue-on-error`: a MegaLinter or Docker failure must not take the coverage gate down with it. That is the same contract as the [SonarQube](sonarqube.md) and [HTML report](html-report.md) legs. It stops there, and deliberately. `action.yml` branches on the nested step's own `outcome`, and on anything other than `success` it hands Brimyr `--quality-scan-broken` instead of a counts path. That flag reads nothing at all: it reports a scan that did not complete, exit `2`, `quality_gate_result` `error`. **A quality scan that never ran must not read as a clean quality half**: it just should not also erase the half that did work. The outcome is what it trusts because the file proves nothing. `chargate ci` writes its counts JSON *before* it decides whether the scan produced any runs, so a scan that ran nothing leaves a well-formed **row of zeros** on disk and only then exits `2`. Read on its own, that document is indistinguishable from a clean pull request; the step's outcome is the only signal that tells the two apart. The other path is real, just not the one the action takes: a counts file passed explicitly to `brimyr lint --counts` that is missing or unreadable is still exit `2`, with a message saying it could not be read. --- # Setup & usage Source: docs/setup.md URL: https://docs.magmamoose.com/brimyr/setup/ Brimyr gates a pull request on two things: **patch coverage** (always) and **net-new quality findings** (opt-in). It runs your tests **on the runner**, and by default it installs their dependencies first using whatever your repo already declares: `uv run`, `poetry run`, `npm ci`. So the minimal wiring is the action on its own, with no setup step: that is what makes it provisionable fleet-wide. Install them yourself if you'd rather (a `setup` step in the reusable workflow, your own steps with the composite action) and set `provision: 'false'`, or skip the test run entirely by feeding a pre-made coverage report via `coverage_file`. The full table of what gets provisioned, and when it declines, is in the [Action reference](action.md#dependency-provisioning). ## 1. Composite action ```yaml # .github/workflows/coverage.yml name: Coverage on: pull_request: push: branches: [main] permissions: contents: read pull-requests: write # only needed for the PR comment jobs: coverage: runs-on: ubuntu-latest steps: - uses: magmamoose/brimyr@v1 with: threshold: '80' pr_comment: 'true' # sonar_url: https://sonar.example.com # sonar_token: ${{ secrets.SONAR_TOKEN }} ``` On pull requests it runs your tests with coverage, gates on patch coverage, and ships to SonarQube when `sonar_url` is set. On a push to the default branch it runs a non-gating baseline that still feeds the trend. No checkout step: the action checks out with `fetch-depth: 0` by default, which is what patch coverage needs. Add your own `actions/checkout@v7` with `fetch-depth: 0` and set `checkout: 'false'` **only** if an earlier step in the job needs the workspace, because the action's own checkout would run too late for it. Every input and output is in the [Action reference](action.md). > **Wire it on `pull_request`, not `pull_request_target`.** Brimyr runs the PR's > *own* test code on the runner. `pull_request_target` runs that code with the base > repo's write token and secrets in scope, so a malicious fork could exfiltrate > them, exactly the privilege a gate that runs the PR's tests doesn't need. Both > events gate if > Brimyr sees them, but `pull_request` is the safe default; only reach for > `pull_request_target` if you fully control who can open PRs. ## 2. pre-push hook ```yaml # .pre-commit-config.yaml repos: - repo: https://github.com/MagmaMoose/brimyr rev: v1.0.0 hooks: - id: brimyr ``` ```sh pre-commit install --hook-type pre-push git push # runs the tests + patch-coverage gate against the default branch ``` It runs at **pre-push** (not pre-commit): patch coverage means running the test suite, too heavy for every commit but fine before a push. ## Ingesting a pre-made report (escape hatch) If your CI already produces a coverage file, point Brimyr at it instead of letting it run the tests: ```yaml - uses: magmamoose/brimyr@v1 with: coverage_file: coverage.xml # or coverage/lcov.info, comma-separated checkout: 'false' ``` Format is inferred from the extension (`.xml` → Cobertura, `.info`/`.lcov` → lcov); append `:lcov` / `:cobertura` to force it. ## SonarQube Optional and non-blocking: ```yaml - uses: magmamoose/brimyr@v1 with: sonar_url: https://sonarqube.example.com sonar_token: ${{ secrets.SONAR_TOKEN }} sonar_project_key: my-service ``` The action installs a scanner for you and points it at the coverage files it just produced. **A Sonar failure never fails the gate**: it becomes a `::warning::` and the run continues. The token travels in `SONAR_TOKEN`, never on the command line. .NET is the exception, and Brimyr handles it automatically: the CLI scanner can't analyse C# at all, so `dotnet sonarscanner` wraps the build instead of following it. Java needs `sonar_args: '-Dsonar.java.binaries=...'` or the analysis is skipped with a warning. Both are covered in [SonarQube](sonarqube.md). ## Quality findings Off by default. Turn it on and the action runs [Chargate](https://github.com/MagmaMoose/chargate) as a nested step and folds the net-new findings into the same summary and the same comment as coverage: ```yaml - uses: magmamoose/brimyr@v1 with: quality: 'true' # quality_fail_on: 'error' # leave unset to stay report-only ``` It needs **Docker** on the runner. `quality_fail_on` defaults to `none`, so the half ships **report-only**: findings are counted and shown, nothing blocks. Measure a release cycle before picking a level: the failure mode here is abandonment, not error. The threshold speaks SARIF levels (`note`, `warning`, `error`, `any`), not Chargate's severity bands. Full detail, and why, in [Quality findings](quality-findings.md). ## Local development ```sh uv sync # install deps + dev tools uv run pytest -q # run the test suite uv run ruff check . # lint uv run ruff format --check . # format check (CI gate) ``` (If `uv` is not on PATH, `python -m uv ...` works after `pip install uv`.) ## Building these docs ```sh uv run --group docs mkdocs serve # live preview at http://127.0.0.1:8000 uv run --group docs mkdocs build # render to ./site (gitignored) ``` The `docs` dependency group (`mkdocs-material`) is non-default, so `uv sync` and CI are unaffected until you opt in with `--group docs`. ## PR comment Brimyr can post **one** consolidated comment on the pull request. It carries the coverage verdict: the percentage, the threshold, and the changed lines the tests never executed, and, when the quality half is on, the net-new findings beneath it, under a second heading in the same comment. It is updated in place on every push rather than stacked, so a long-running PR keeps exactly one comment. ```yaml permissions: contents: read pull-requests: write # ← required; the default `read` cannot comment # ... - uses: magmamoose/brimyr@v1 with: pr_comment: 'true' # github_token defaults to the job token (comments as github-actions[bot]) ``` Add `token_broker_url`, plus `id-token: write` on the job, and the comment is authored by **Brimyr[bot]** instead of the shared `github-actions[bot]`: ```yaml permissions: contents: read pull-requests: write id-token: write # ← required to mint the Brimyr[bot] identity # ... with: pr_comment: 'true' token_broker_url: https://broker-brimyr.magmamoose.com ``` Commenting is **non-blocking**: a missing token, a 403, or a network failure is reported on stderr and never changes the gate verdict, the same contract the SonarQube step follows. The broker is non-blocking too, and *silently* so: any problem with it costs the byline and nothing else. Both `pr_comment` and `token_broker_url` are inputs of the **composite action**; the reusable workflow does not forward them today. See [PR comment](pr-comment.md) for what both blocks look like, how the two comment markers work, and the full list of broker failure messages. --- # Shell / bats Source: docs/shell.md URL: https://docs.magmamoose.com/brimyr/shell/ Brimyr detects a shell repo from a real `.bats` file, runs ```bash bats --recursive ``` and gates on the exit status. Whether it also gates on *coverage* depends on one thing: whether `kcov` is installed on the runner. Usually it is not, and that is fine. ## What counts as a shell suite A `tests/` directory is not evidence of anything. It ships with most repositories in existence, so the marker only decides whether it is worth looking; the confirming search is what decides. Brimyr needs an actual `*.bats` file that is not vendored: - `node_modules`, `.venv`, `vendor` and the other usual directories are skipped, because `bats` is itself an npm package and installing it puts its own `.bats` files in the tree. - `test/bats`, `test/test_helper`, `bats-support`, `bats-assert`, `bats-file`, `bats-mock` and `bats-detik` are skipped for the same reason. The conventional bats layout vendors the framework as a submodule, and bats-core's own checkout carries a `test/*.bats` suite. Without that rule, Brimyr would run the framework's tests instead of yours. Whatever survives decides the command. The directories holding those files are passed to `bats --recursive`, a directory already covered by another is dropped, and a `.bats` file sitting at the repo root is passed as itself rather than as `.` (which would make `--recursive` walk every vendored checkout in the tree). ```text tests/scan.bats -> bats --recursive tests scripts/tests/lib.bats -> bats --recursive scripts/tests tests/a.bats + tests/unit/b.bats -> bats --recursive tests smoke.bats -> bats --recursive smoke.bats ``` If your layout defeats that, set `test_command:` and it is used verbatim. ## Running it: `bats` does not need to be installed If `bats` is not on PATH, Brimyr runs it through `npx --yes bats`, which fetches bats-core into the runner's npm cache. This is the same route the JavaScript ecosystem takes with jest, and it exists for the same reason: a shell `127` is a broken run and a red gate on a repository whose tests are perfectly fine. Set `provision: 'false'` to turn it off and use whatever the job installed itself. ## Coverage: kcov if you have it, and a green result if you do not `kcov` is the only realistic coverage tool for bash, and it is installed nowhere by default. Brimyr therefore: - **uses it when it is on PATH.** The run is wrapped as `kcov --include-path=. --exclude-pattern=... coverage/kcov bats ...`, and the Cobertura reports it writes under `coverage/kcov/` are ingested like any others. The `.bats` files themselves are excluded, so the denominator is the scripts under test. - **never installs it.** It is a system package (`apt-get install kcov`) or a source build, so provisioning it would mean either `sudo` into the caller's runner image for every shell repository on the estate, or minutes of build time per job. Both are the wrong trade for a gate that is meant to be free. So the common case is a bats suite with no coverage at all, and that case is **green**: ```text ⚪ Tests ran, but no coverage was measured, Shell produced no coverage report. The suite passed; there is simply no number to gate on. ``` `gate_result` is `skipped`, the exit code is `0`, and the summary replaces the coverage table rather than printing `100% of 0 lines`, which is what a well tested pull request looks like. This is not a softening of the broken-run rule. It is a different question: *can* this ecosystem produce coverage at all? Only `shell` says no. Every other ecosystem instruments as it runs, so a missing report there still means the run broke and still turns the build red, including a .NET test project that has no `coverlet.collector`. To measure your bash, install kcov in the job before the Brimyr step: ```yaml - run: sudo apt-get update && sudo apt-get install -y kcov - uses: magmamoose/brimyr@v1 ``` ## What changes when you pick up this version Detection is not opt-in, so a repository that already had a bats suite and a Brimyr workflow starts **running** that suite. If it has never run in CI, expect it to have opinions. Two ways out, both explicit: - `ecosystem: 'dotnet'` (or whichever you do want) pins the run to that list, and detection is skipped entirely. - `test_command:` replaces every ecosystem's command with yours. Nothing else about an existing run changes: a repository with no `.bats` file behaves exactly as it did. ## Alongside another language `ecosystem` is comma-separated and detection is not exclusive, so a .NET repository whose scanners are bash runs both and merges the reports: ```yaml - uses: magmamoose/brimyr@v1 with: ecosystem: 'dotnet,shell' test_command: '' # leave empty: each ecosystem runs its own command ``` Set `test_command` there and you lose the split, because one command replaces every ecosystem's. That is the trap this ecosystem exists to remove: forcing `test_command: 'bats tests'` on a .NET repository traded the .NET coverage for a bats run that then produced no report and failed the build. When one half measures and the other does not, the summary says so under the table: ```text ⚠️ Not everything was measured, Shell produced no coverage report, so changed Shell lines are not in the figures above (they are absent from the denominator, not counted as uncovered). ``` That warning matters. A file the report never mentions contributes nothing to patch coverage, so an unmeasured half does not lower the percentage, it narrows what the percentage is about. ## shellcheck Not part of this. Static analysis of shell belongs to the [quality half](quality-findings.md), which runs MegaLinter's linters through Chargate. `BASH_SHELLCHECK` is a valid MegaLinter key but is not in Chargate's curated `quality` set, so ask for it explicitly: ```yaml - uses: magmamoose/brimyr@v1 with: quality: 'true' quality_linters: 'BASH_SHELLCHECK,PYTHON_RUFF,JAVASCRIPT_ES,TYPESCRIPT_ES,JAVA_PMD,GO_GOLANGCI_LINT' ``` `quality_linters` **replaces** the curated set rather than adding to it, which is why the whole list is spelled out above. --- # SonarQube Source: docs/sonarqube.md URL: https://docs.magmamoose.com/brimyr/sonarqube/ Brimyr can ship coverage to SonarQube in the same run that gates the pull request. Set two inputs and the action installs a scanner for you: ```yaml - uses: magmamoose/brimyr@v1 with: sonar_url: ${{ vars.MM_SONAR_URL }} sonar_token: ${{ secrets.MM_SONAR_TOKEN }} ``` An empty `sonar_url` disables the whole leg, so passing an unset variable simply turns it off. The project key defaults to the repo slug with `/` replaced by `_`, because Sonar keys can't contain a slash. !!! note "This is a full analysis, not a coverage upload" Running the scanner performs SonarQube's own analysis: bugs, code smells, duplication and security hotspots arrive with the coverage whether or not you look at them. There is no coverage-only mode. ## The gate never depends on it Every failure here is a `::warning::` annotation and the run continues. A missing scanner, a bad URL, an outage, an absent token: none of them change the exit code. The patch-coverage gate is computed locally from the coverage file and never talks to SonarQube. ## One scanner per ecosystem | Ecosystem | Scanner | Coverage property | | --- | --- | --- | | Python | `sonar-scanner` (after the tests) | `sonar.python.coverage.reportPaths` | | JavaScript / TypeScript | `sonar-scanner` (after the tests) | `sonar.javascript.lcov.reportPaths` | | Java | `sonar-scanner` (after the tests) | `sonar.coverage.jacoco.xmlReportPaths` | | Go | `sonar-scanner` (after the tests) | `sonar.go.coverage.reportPaths` (the profile as `go test` wrote it) | | .NET | `dotnet sonarscanner`, **wrapping the build** | `sonar.cs.cobertura.reportsPaths` | | Shell | `sonar-scanner` (after the tests) | none: Sonar has no importer for shell coverage, so nothing is uploaded for it | .NET is the one that can't follow the same shape. SonarSource documents that the SonarScanner CLI doesn't support C# or VB.NET at all, because those issues come from Roslyn analyzers that `dotnet sonarscanner begin` injects into the compilation, and `end` collects what the build produced. So for .NET the scanner wraps the run: ```text dotnet sonarscanner begin /k:... /d:sonar.cs.cobertura.reportsPaths=**/TestResults/**/coverage.cobertura.xml dotnet build --no-incremental --disable-build-servers dotnet test --collect:"XPlat Code Coverage" dotnet sonarscanner end ``` Brimyr does this automatically when it detects a .NET repo. Three consequences: - The coverage path is declared at `begin`, before any report exists, so it's a wildcard. `end` accepts only three flags, and the wildcard is also what makes a multi-test-project solution work without naming GUID directories. - `--no-incremental` isn't optional. A cached build compiles nothing, the injected analyzers never run, and `end` uploads an empty analysis from a green job. - Setting `coverage_file` disables the wrap. With no build there's nothing for `end` to collect, so Brimyr skips rather than uploading an empty analysis. When the .NET scanner runs, the plain `sonar-scanner` pass doesn't also run. The .NET scanner analyses the whole tree, and a second pass on the same project key would overwrite it. !!! note "The .NET property is `reportsPaths`, plural" Unlike `sonar.python.coverage.reportPaths` and `sonar.javascript.lcov.reportPaths`. Sonar isn't consistent here, and the singular form is silently ignored. ## Java needs `sonar.java.binaries` `sonar-scanner` over a Java repo fails with `please provide compiled classes with sonar.java.binaries`. Brimyr can't infer the path, so rather than run a scan that can't succeed it skips with a warning until you supply it: ```yaml with: sonar_url: https://sonar.example.com sonar_args: '-Dsonar.java.binaries=**/target/classes' ``` Sonar also documents that the CLI scanner shouldn't be used for Maven or Gradle projects at all. `mvn sonar:sonar` gives a better analysis. The patch-coverage gate is unaffected either way: it reads the JaCoCo reports directly. ## What the scanner install does The install step only runs when `sonar_url` is set, and it carries `continue-on-error`, so a failure leaves no scanner on `PATH` and Brimyr reports `skipped` instead of failing your job. - **.NET repos** get `dotnet tool install --global dotnet-sonarscanner`. Already-installed isn't an error: it falls through to `update`, then to using what's there. - **Everything else** downloads the SonarScanner CLI and verifies it against SonarSource's published `.sha256` rather than a hash pinned in this repo, which would go stale the moment `sonar_scanner_version` is bumped. ## Community Build can't gate a pull request Worth knowing before you plan around it. SonarQube Community Build analyses **only the main branch**. Pull request analysis and branch analysis start at Developer Edition. You can still run a scan in a `pull_request` job, and `sonar.qualitygate.wait=true` will fail that job on a red gate. But it isn't a pull request analysis: with no `sonar.pullrequest.*` support the results are filed against the project's single main branch, so the gate is evaluated against main's new code and the run overwrites main's stored state with PR code. Using Sonar as the PR gate destroys the trend you're keeping Sonar for. The Sonar way quality gate does include `New code test coverage is greater than or equal to 80.0%`, but "New Code" there means everything since the project's baseline, not the lines this pull request changed. That gate also ignores coverage entirely until there are at least 20 new lines to cover. !!! warning "Never pass branch parameters to Community Build" `sonar.branch.name` and `sonar.pullrequest.*` are a hard scanner error against Community Build, not a silent no-op. Brimyr doesn't pass either, but `sonar_args` would let you. --- # Troubleshooting Source: docs/troubleshooting.md URL: https://docs.magmamoose.com/brimyr/troubleshooting/ Symptom, cause, fix. Exit codes first, because they tell you which half of the page you're in. | Exit code | Meaning | | --- | --- | | `0` | Patch coverage met the threshold, or there was nothing coverable to gate. | | `1` | Patch coverage fell below the threshold. This is the gate doing its job. | | `2` | Broken run, bad setup, or a usage error. **Never** a coverage verdict. | Exit `2` is the one to read carefully. A failed, empty or unparseable test run is a tool error, not 0% coverage, so it turns the build red instead of quietly failing the gate. ## The gate passed but the number looks too good The most important entry on this page, because nothing fails and nothing is logged as an error. Patch coverage only counts changed lines the coverage report actually mentions. A file the report never names contributes nothing, so anything that drops files from the report inflates the result instead of failing it. Check the denominator first. `Covered / changed executable lines` in the PR comment, or `total_lines` in the outputs. If it's `0` or much smaller than the diff, one of these is happening: - **A report was dropped.** A multi-project .NET solution writes one report per test project, and a Maven reactor writes one per module. Count them: `find . -name coverage.cobertura.xml | wc -l`, or `find . -path '*/target/site/jacoco/jacoco.xml' | wc -l`. - **The paths don't match.** Coverage paths are often absolute or rooted differently from `git diff` paths. Use `strip_prefix` to peel a known root. - **A JaCoCo report was read as Cobertura.** Both use `.xml`. Brimyr picks by root element, but an explicit `coverage_file: report.xml:cobertura` overrides that and produces an empty report. Drop the format suffix or use `:jacoco`. - **`exclude` is too broad.** A glob like `*Migrations*` uses `fnmatch`, where `*` crosses `/`, so it matches deeper than you might expect. ## `the coverage report(s) named no files at all` The report parsed but describes nothing, so there is no coverage to gate on. That's a broken report, not 0%, and it exits `2`. The usual causes: - **JVM:** a surefire `` that overrides instead of appending `@{argLine}`, which detaches the JaCoCo agent while the build stays green. - **.NET:** `dotnet test` ran without `--collect:"XPlat Code Coverage"`, or the `coverlet.collector` package isn't referenced by the test project. - **Any:** `coverage_file` points at a report from a different run, or at a file the coverage tool wrote before it instrumented anything. ## `tests did not finish within Ns and were killed` The suite hit `test_timeout` (default 3600 seconds) and was killed. This is a broken run, exit `2`, not 0% coverage. Raise it if the suite is genuinely that slow: ```yaml with: test_timeout: '7200' ``` Set `'0'` to wait indefinitely, which restores the old behaviour: a hung suite then holds the runner until the job timeout, six hours by default on GitHub-hosted runners. ## `History is shallow, so the merge-base is unavailable.` `actions/checkout` defaults to `fetch-depth: 1`, and patch coverage needs the merge base. Set `fetch-depth: 0`: ```yaml - uses: actions/checkout@v7 with: { fetch-depth: 0 } ``` If you let Brimyr do its own checkout, leave `fetch_depth` at its default of `0`. Locally, run `git fetch --unshallow`. ## `No merge-base could be determined for the given base and head.` The two refs share no history. Usually the base branch isn't fetched, or you're comparing across unrelated histories such as an orphan branch. Fetch the base, or pass `--no-merge-base` to diff the two refs directly without a common ancestor. ## `No test suite detected`: the run passed but nothing was gated Not an error. Auto-detection found no ecosystem marker, at the root or in a [project in a subdirectory](index.md#projects-in-subdirectories), so no tests ran and there is nothing to gate: the job is green, `gate_result` is `skipped`, and the summary says so in place of the coverage table. This is what makes Brimyr safe to provision across a whole org rather than adopt one repo at a time. A charts, Terraform, prompts or docs repo has nothing to test, and failing it would mean a permanently red check on a gate it can never satisfy. **If the repo DOES have tests, this is the bug, not the fix.** The warning is loud on purpose. Detection needs one of: - `pyproject.toml` / `setup.py` / `setup.cfg` / `requirements.txt` / `tox.ini` **plus a real Python test signal**: a `test_*.py` or `*_test.py` file belonging to *this* project, outside `.venv` / `node_modules` / other vendored directories, or a pytest section (`[tool.pytest.ini_options]`, `[pytest]`, `[tool:pytest]`). "Belonging to this project" means no nested project sits between the repo root and the test file. A subdirectory with its own `pyproject.toml` / `setup.py` / `setup.cfg` (a separate deployable such as `broker/`) is a different project whose dependencies are not in this environment, so `pytest` at the root would collect its files and then fail importing them. Depth alone is fine: `backend/tests/test_api.py` with no `backend/pyproject.toml` is still this repo's suite. A nested project whose root detects nothing is found on its own and run from its own directory. Under a root that IS detected it is not: the root's verdict stays as it was. To gate it from the root anyway, add a root pytest section pointing at it (`testpaths`), since an explicit config wins outright, or set `ecosystem:` plus a `test_command:` that enters the directory. - `package.json` **plus** a jest/vitest config or a non-placeholder `test` script. - `pom.xml` for Java (`build.gradle` is recognised but not auto-run, see above). - `go.mod` **plus** a `_test.go` file the root module's `./...` reaches (not under `vendor/`, `testdata/`, a `_`/`.` directory, or a nested module). See [Go](go.md). - `*.sln` / `*.slnx` / `*.csproj` in the repo root for .NET. - A real, non-vendored `*.bats` file for [shell](shell.md). A `tests/` directory on its own is never enough, and a vendored bats-core submodule under `test/bats` is deliberately ignored. Force it with `ecosystem:`, or skip detection entirely with `coverage_file:`. A forced `ecosystem:` that then fails is still an error: explicit intent is never downgraded to a skip. ## `the test command did not run: ... command not found` Nothing on the runner could launch the test binary. Usually that is `pytest` or `jest` on a job that never installed the repo's dependencies. Nothing was measured, so this is a broken run (exit 2), not 0% coverage. Brimyr normally installs them for you: with `provision` on (the default) it uses the repo's own dependency manager, so a `uv.lock` repo runs under `uv run` and a `package.json` repo gets an `npm ci` first. The line above this error says why that declined. The common ones: | Line | Fix | | --- | --- | | ``` `uv` is not on PATH ``` | The action installs `uv` for you when `provision` is `true`, so you'll only see this outside the action, e.g. `brimyr local`. Install `uv`, or install your test dependencies. | | `no installable Python project found` | No `pyproject.toml`, no `requirements*.txt`. Add one, or set `test_command`. | | ``` poetry project, but `poetry` is not on PATH ``` | A pre-2.0 Poetry layout (`[tool.poetry]`, no `[project]`). Install Poetry in the job. | | ``` `npm` is not on PATH ``` | The action installs Node when the runner has none and `provision` is `true`, so you'll see this with `provision: 'false'` or outside the action. Install Node in the job. | You can always take it over yourself: install the dependencies in an earlier step and set `provision: 'false'`, set `test_command` to something that works (which disables provisioning too), or skip the test run with `coverage_file`. The full table is in the [Action reference](action.md#dependency-provisioning). ## `dependency install failed` The provisioning step itself exited non-zero: `npm ci` against a stale lockfile, `poetry install` on an unresolvable graph, `uv run` on a lockfile that no longer matches `pyproject.toml`. The tests were **not** run, so again this is a broken run, not a coverage number. The failing command and its exit code are in the message; run it locally to see the real error. ## The tests ran but the build is red with exit 2 That's the broken-run rule. The suite failed, produced no coverage file, or wrote something unparseable. Look at the test output above the Brimyr step: the underlying failure is there, and Brimyr is refusing to convert it into a coverage number. The one exception is an ecosystem that cannot produce coverage in the first place. A bats suite emits nothing without `kcov`, so a passing one is a pass, not a broken run. See [Shell / bats](shell.md). ## `--repo ... is not a directory` The path given to `--repo` (or the `repo` the action ran in) does not exist, or is a file. Exit 2, and deliberately not a green skip: detection looks for marker files, a path that is not there has none, and before this was checked a mistyped `--repo` reported "no test suite detected" and passed the build. ## `could not run git ...` Git could not be started at all, which is different from git running and failing. Either it is not installed in the job (a slim container without it), or the working directory is unreadable. Exit 2, naming the path. Nothing was measured. ## `could not write the ... JSON` The `--json-out` or `--quality-json-out` path could not be written: a permission problem, a read only filesystem, or a component of the path that is a file rather than a directory. Missing parent directories are created, so those are not a cause. This is a **warning and nothing more**. The gate is already decided by the time the artifact is written, so the run keeps its own exit code and the summary and PR comment still go out; only the file is missing. The artifact upload in `action.yml` is guarded on the file existing, so nothing downstream reads a stale one. ## `Tests ran, but no coverage was measured` Not an error, and not 0%. The suites that ran passed, and none of them measures coverage: today that means [bats without `kcov`](shell.md). `gate_result` is `skipped`, the exit code is `0`, and the summary replaces the coverage table so the run cannot be mistaken for a well tested one. Install `kcov` on the runner to turn it into a measurement: ```yaml - run: sudo apt-get update && sudo apt-get install -y kcov ``` ## `Not everything was measured` A polyglot run where one ecosystem measured and another did not, for example `dotnet,shell` on a runner with no `kcov`. The percentage above the warning is real, but it covers only the measured half: a file the coverage report never mentions contributes nothing to the denominator, so the unmeasured half's changed lines are absent rather than uncovered. ## Small pull requests aren't being gated Working as intended. `min_lines` defaults to `20`, so a diff with fewer changed executable lines than that isn't gated, and the summary says so: ```text ⚪ Only 3 changed executable line(s), below the 20-line minimum, so the 80.0% threshold was not applied (patch coverage was 33.3%). ``` Set `min_lines: '0'` to gate every diff. See [Patch coverage](patch-coverage.md). ## `skipped (sonar-scanner not found on PATH)` The scanner install didn't happen or didn't succeed. It only runs when `sonar_url` is set, and it's deliberately non-fatal, so the job stays green with no analysis uploaded. Check that `sonar_url` is non-empty, and look for an earlier `::warning::` from the install step. On a container or self-hosted runner without .NET, add `actions/setup-dotnet` before Brimyr for .NET repos. ## `skipped (no token in $SONAR_TOKEN)` `sonar_url` is set but `sonar_token` is empty. Non-blocking by design, so nothing failed. Pass the secret, or clear `sonar_url` if you didn't mean to enable the leg. ## `skipped (Java / JVM needs sonar.java.binaries)` `sonar-scanner` can't analyse Java without compiled classes. Supply them: ```yaml sonar_args: '-Dsonar.java.binaries=**/target/classes' ``` See [SonarQube](sonarqube.md#java-needs-sonarjavabinaries). ## No HTML report artifact appeared `html_report` needs ReportGenerator, which needs the .NET runtime. On GitHub-hosted runners the SDK is preinstalled. Elsewhere you'll see: ```text ::warning::html_report needs the .NET runtime, add actions/setup-dotnet ``` Add `actions/setup-dotnet` before the Brimyr step. The gate is unaffected either way. ## The PR comment isn't posted The comment is failure-isolated, so it never fails the run. Check, in order: 1. `pr_comment: 'true'` is set. 2. The workflow has `pull-requests: write`. 3. The run is on a pull request. There's no PR to comment on for a push. 4. The event is `pull_request`, not `pull_request_target`. Fork pull requests get a read-only token, so the comment can't post. That's a GitHub restriction, not a Brimyr one. ## The comment says `github-actions[bot]` instead of `Brimyr[bot]` The token broker didn't answer, and the run fell back to the job token. That fallback is silent and deliberate: the comment still posts, only the byline changes. Check that `token_broker_url` is set and the workflow has `id-token: write`.