Skip to content

ci(secu): speed up dependency and gitleaks scans - #74

Merged
sc979 merged 7 commits into
mainfrom
MON-211442-optimize-security-scans
Oct 7, 2026
Merged

ci(secu): speed up dependency and gitleaks scans#74
sc979 merged 7 commits into
mainfrom
MON-211442-optimize-security-scans

Conversation

@kduret

@kduret kduret commented Oct 6, 2026 •
edited by atlassian Bot
Loading

Copy link
Copy Markdown
Contributor

Description

What are your needs or what are you planning to do in this PR?

Fixes: MON-211442

Speed up the two reusable scans that run on every pull request of many repositories. Measured on centreon-modules over 5 days: 434 dependency-analysis runs (13.7 h of runner time, on the module pipelines critical path) and 89 gitleaks-analysis runs (1.5 h, 1.3 GiB downloaded per run).

Before / after on centreon-modules

Job Metric Before After
dependency-analysis Job duration 98 s median (163 s p90) 9 s
Blocklist scan step 64-118 s 1 s
Checkout 6-10 s 7 s (unchanged)
Network per run ~58 MB (depth 1) ~58 MB (unchanged)
gitleaks-analysis (pull requests) Job duration 58 s median 12 s
Checkout 38-48 s 6 s
Network per run ~1.3 GiB (full history, all branches and tags) ~55-60 MB (PR commits + 2)
Both, over 5 days Runner time 15.2 h (434 + 89 runs) ~1.4 h
gitleaks network ~116 GiB ~5 GB

"Before" figures come from the 523 scan jobs run on centreon-modules from 2026-10-01 to 2026-10-06; "after" durations from the test PR centreon-modules#9507; network sizes from clones of the same refs. The 5-day "after" totals apply the new per-run figures to the same number of runs.

dependency-analysis.yml

  • The blocklist scan spawned one grep per IOC and per lockfile (1679 IOC x 30 lockfiles = ~50,000 processes, 64-118 s per run).
  • It now builds a name@version pattern file once and runs a single grep -oF -f per lockfile: 96 s -> 0.24 s on the centreon-modules tree.
  • Same substring semantics as before: the 1679 patterns generated from the real blocklist are byte-identical to the ones the old loop built.
  • Old (main) and new steps run side by side with bash -eo pipefail on the centreon-modules tree: same fail_the_build and exit code for clean, infected (40 IOC), overlapping IOC, lockfile compliance findings (before and after the enforcement date), OVERRIDE_DEPENDENCY_SCAN=true and an empty blocklist. Only difference: when an IOC is contained in another one (e.g. ngx-bootstrap@19.0.4 in @rxap/ngx-bootstrap@19.0.4), the old loop also reported the shorter, false-positive name.

gitleaks-analysis.yml

  • gitleaks-action only scans first_pr_commit^..head on pull requests, but the checkout fetched the whole history (1.3 GiB, 38-48 s on centreon-modules for a 4 s scan).
  • On pull requests, fetch-depth is now the PR commit count + 2 (merge and base commits). Push, schedule and workflow_dispatch keep fetch-depth: 0.
  • On centreon-modules: about 60 MB downloaded instead of 1.3 GiB, whatever the PR size (the shallow clone carries one full snapshot of the repository).

Verification

  • 4 centreon-modules PRs (7 to 57 commits): same commit range scanned as with the full history.
  • Test PR centreon-modules#9507: checkout 6 s instead of 38-48 s; after a GitHub "Update branch" merge, depth and range are recomputed and the same commits are scanned.

kduret added 7 commits October 6, 2026 14:54
@kduret
ci(secu): match blocklist with a single grep per lockfile
d4926e6 
The blocklist scan spawned one grep per IOC and per lockfile (1679 IOC x 30
lockfiles on centreon-modules), taking 60-120s per run. A single fixed-string
grep per lockfile gives the same matches in under a second.
@kduret
ci(secu): limit gitleaks checkout depth to pull request commits
db1f872 
gitleaks-action only scans the pull request commits, but the checkout fetched
the whole history (40-50s on centreon-modules for a 4s scan). Pull requests
now fetch their commits plus the merge and base commits; other events keep
the full history.

Refs: MON-211442
@kduret
fix(secu): scan every pull request commit with gitleaks
062c715 
gitleaks-action lists pull request commits without pagination, so only the
first 30 commits were scanned (gitleaks/gitleaks-action#187). Pull requests
now run a checksum-pinned gitleaks 8.24.3 on the full first^..head range;
other events keep using gitleaks-action.

Refs: MON-211442
@kduret
ci(secu): keep full gitleaks history with a blobless checkout
fb64844 
Pull requests keep fetch-depth 0 but fetch the history without file
contents and only check out the gitleaks config files; the contents of the
scanned changes are fetched in one batch before the scan (63 MB instead of
1.3 GiB on centreon-modules).

Refs: MON-211442
@kduret
ci(secu): scan pull requests with gitleaks-action again
d84f5ae 
Drop the direct gitleaks run: gitleaks-action scans the pull request again,
after a step that batch-fetches the file contents of the pull request
changes into the blobless checkout.

Refs: MON-211442
@kduret
ci(secu): fetch only pull request commits for gitleaks
cde8880 
Go back to a PR-sized fetch-depth with file contents (about 60 MB on
centreon-modules) instead of the blobless full-history checkout, which needed
an extra authenticated fetch step for the same size.

Refs: MON-211442
@kduret
fix(secu): keep the dependency scan green with an empty blocklist
d5e7171 
Under bash -eo pipefail, grep -v exits 1 when every blocklist line is a
comment, which failed the step where the previous loop scanned nothing.

Refs: MON-211442
@kduret
kduret marked this pull request as ready for review October 7, 2026 07:45
@kduret
kduret requested a review from a team as a code owner October 7, 2026 07:45
sc979
sc979 reviewed Oct 7, 2026
View reviewed changes
Comment thread .github/workflows/gitleaks-analysis.yml
pull-requests: read

steps:
# Pull requests only scan their own commits: fetch them plus the merge and base commits, full history otherwise

@sc979 sc979 Oct 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@sc979
sc979 merged commit 137aa25 into main Oct 7, 2026
5 checks passed
@sc979
sc979 deleted the MON-211442-optimize-security-scans branch October 7, 2026 08:41
sc979 added a commit that referenced this pull request Oct 7, 2026
@sc979
Merge remote-tracking branch 'origin/main' into SECU-gitleaks-custom
9489601 
Bring in #74 (shallow PR checkout with a computed fetch depth, faster
blocklist scan). The gitleaks conflict keeps the pull_request_target
guard as the first step, then the fetch-depth step, and keeps the job
token scoped to contents: read.
sc979 added a commit that referenced this pull request Oct 7, 2026
@sc979 @claude
ci(secu): avoid false leaks on shallow PR checkout
ecd2b5a 
Since #74 the PR checkout is shallow (fetch-depth = PR commits + 2).
When the base branch moved further than that since the PR forked, the
merge-base was unreachable: the scan concluded "no shared history" and
ran a full scan of the shallow clone, where gitleaks reads boundary
commits as roots and reports every pre-existing secret in the tree.
A clean PR then failed on secrets already on the base branch.

The scan now deepens the PR history (32 to 1024 commits, doubling)
until the fork point is reachable from both sides, unshallows as a
last resort, and never scans a range holding a shallow boundary
commit. "No shared history" is only concluded on a complete clone.
Deepening goes through the authenticated fetch and stops when a fetch
brings nothing, so the existing retry loop takes over.

Also shorten the pull_request_target error message.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Assisted-by: Claude Code (claude-opus-5-5)
@sc979 sc979 mentioned this pull request Oct 7, 2026
Merged
sc979 added a commit that referenced this pull request Oct 12, 2026
@sc979 @claude
ci(secu): avoid false leaks on shallow PR checkout
197d7fc 
Since #74 the PR checkout is shallow (fetch-depth = PR commits + 2).
When the base branch moved further than that since the PR forked, the
merge-base was unreachable: the scan concluded "no shared history" and
ran a full scan of the shallow clone, where gitleaks reads boundary
commits as roots and reports every pre-existing secret in the tree.
A clean PR then failed on secrets already on the base branch.

The scan now deepens the PR history (32 to 1024 commits, doubling)
until the fork point is reachable from both sides, unshallows as a
last resort, and never scans a range holding a shallow boundary
commit. "No shared history" is only concluded on a complete clone.
Deepening goes through the authenticated fetch and stops when a fetch
brings nothing, so the existing retry loop takes over.

Also shorten the pull_request_target error message.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Assisted-by: Claude Code (claude-opus-5-5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sc979 sc979 sc979 approved these changes
@BaptisteCentreon BaptisteCentreon Awaiting requested review from BaptisteCentreon BaptisteCentreon is a code owner automatically assigned from centreon/owners-security

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kduret @sc979