Add read-only smoke tests for the modules written from the spec by davidnmbond · Pull Request #51 · panoramicdata/Codacy.Api

davidnmbond · 2026-10-12T12:02:00Z

The ~170 operations added in 4.0.13 were written from the OpenAPI spec and never exercised against the real API. This adds NewModulesSmokeTests, which calls 43 read-only endpoints across the new modules against the configured test organization.

Behaviour

A refusal (401, 402, 403, 404, 409) is reported but does not fail: the organization may lack the feature, plan or permission.
A 400/422 (this client sent a malformed request), any other exception, or a 2xx whose body cannot be read (a model that does not match the real response) fails the test.
It also fails if 5 or fewer calls succeed, so it cannot pass vacuously on bad credentials.
Logging is off and only outcomes are written, never response bodies, because some of these endpoints return secrets or capability URLs. Mutating endpoints and ones that return credentials (API tokens, billing card) are left out.
CODACY_SMOKE_REPORT=<path> writes the outcome lines to a file, since a passing test shows no output.
Category=Integration, so CI skips it like the other integration tests.

Result against a real organization

36 OK, 7 refused, 0 failures. The refusals are the expected missing features: no Jira or Slack integration (404), no autoconfig run (404), no SBOM (404), no enterprise (404), and DAST (403).

Limits

"OK" means the response deserialized; it does not assert on the content. Endpoints that returned empty lists prove little about their item models. The search POSTs (AI inventory, SBOM, reports) and anything needing ids this organization does not have are not covered.

🤖 Generated with Claude Code

Calls 43 read-only endpoints across the new modules against a real organization. A refusal (401/402/403/404/409) is reported, but a 400/422, an exception, or a 2xx whose body cannot be read fails the test. Logging is off and only outcomes are written, never response bodies. CODACY_SMOKE_REPORT writes the outcome lines to a file. Category=Integration, so CI skips it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

codacy-production · 2026-10-12T12:07:39Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 20 complexity · 0 duplication

Metric Results

Complexity 20

Duplication 0

View in Codacy

🟢 Coverage ∅ diff coverage · +0.00% coverage variation

Metric Results

Coverage variation ✅ +0.00% coverage variation (-1.00%)

Diff coverage ✅ ∅ diff coverage

View coverage diff in Codacy

Coverage variation details

Coverable lines Covered lines Coverage

Common ancestor commit (dc1df57) 206 135 65.53%

Head commit (aeb3f07) 206 (+0) 135 (+0) 65.53% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

Coverable lines Covered lines Diff coverage

Pull request (#51) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

codacy-production

Pull Request Overview

Codacy reports the PR as not up to standards because of one new issue. The main merge-blocking concerns are that HTTP 5xx responses can be treated as successful and that segment-key endpoints retrieve credentials despite the requirement to exclude sensitive endpoints.

The test scope description also does not clearly reconcile the stated endpoint count with the probe sites in the diff, and coverage data is unavailable for independently verifying conditional probes.

Test suggestions

Smoke-test the listed read-only module endpoints and deserialize their real responses.
Report expected 401, 402, 403, 404, and 409 refusals without failing.
Fail on HTTP 400 and 422 responses.
Fail on unexpected exceptions and response deserialization failures.
Require more than five successful probes.
Disable body logging and avoid writing response bodies to output or the smoke report.
Write outcome lines to CODACY_SMOKE_REPORT when configured.

Low confidence findings

Reconcile the documented endpoint count with the actual probe sites and add or expose coverage for conditional probes so the complete smoke-test scope can be verified.

_{TIP Improve review quality by adding custom instructions}
_{TIP How was this review? Give us feedback}

codacy-production · 2026-10-12T12:07:58Z

+		await Probe("Segments.GetSegmentsKeys", () => client.Segments.GetSegmentsKeysAsync(p, o, null, 5, null, ct));
+		await Probe("Segments.GetSegmentsKeysWithIds", () => client.Segments.GetSegmentsKeysWithIdsAsync(p, o, null, 5, null, ct));


_{🟡 MEDIUM RISK}

Remove the segment-key probes, or replace them with a read-only endpoint that validates the module without retrieving secret material.

codacy-production · 2026-10-12T12:07:58Z

+		catch (ApiException ex) when ((int)ex.StatusCode >= 500)
+		{
+			_outcomes.Add($"SERVER    {name}: HTTP {(int)ex.StatusCode}");
+		}


_{🟡 MEDIUM RISK}

HTTP 5xx responses are currently reported but do not fail the test, contrary to the requirement that any other exception must fail. Add these responses to _failures or remove this special catch.

codacy-production · 2026-10-12T12:07:58Z

+	public async Task ReadOnlyEndpoints_DeserializeRealResponses()
+	{
+		var p = TestProvider;
+		var o = TestOrganization;
+		var r = TestRepository;
+		var ct = CancellationToken;
+
+		// Logging off: the shared client logs response bodies, and some of these hold secrets or capability URLs.
+		var baseOptions = GetClientOptions();
+		using var client = new CodacyClient(new CodacyClientOptions { ApiToken = baseOptions.ApiToken, BaseUrl = baseOptions.BaseUrl });
+
+		// Repository settings, files, coverage and analysis
+		await Probe("RepositorySettings.GetQualitySettingsForRepository", () => client.RepositorySettings.GetQualitySettingsForRepositoryAsync(p, o, r, ct));
+		await Probe("RepositorySettings.GetBuildServerAnalysisSetting", () => client.RepositorySettings.GetBuildServerAnalysisSettingAsync(p, o, r, ct));
+		await Probe("RepositorySettings.GetRepositoryLanguages", () => client.RepositorySettings.GetRepositoryLanguagesAsync(p, o, r, ct));
+		await Probe("RepositorySettings.GetRepositoryIntegrationsSettings", () => client.RepositorySettings.GetRepositoryIntegrationsSettingsAsync(p, o, r, ct));
+		await Probe("RepositorySettings.GetBranchRequiredChecks", () => client.RepositorySettings.GetBranchRequiredChecksAsync(p, o, r, TestBranch, ct));
+		await Probe("RepositoryFiles.ListDirectories", () => client.RepositoryFiles.ListDirectoriesAsync(p, o, r, null, null, null, null, null, 5, ct));
+		await Probe("RepositoryFiles.ListIgnoredFiles", () => client.RepositoryFiles.ListIgnoredFilesAsync(p, o, r, null, null, null, 5, ct));
+		await Probe("RepositoryCoverageReports.ListCoverageReports", () => client.RepositoryCoverageReports.ListCoverageReportsAsync(p, o, r, 5, ct));
+		await Probe("AnalysisActions.GetAutoconfigStatus", () => client.AnalysisActions.GetAutoconfigStatusAsync(p, o, r, ct));
+		await Probe("AnalysisActions.GetLatestAutoconfigRunSummary", () => client.AnalysisActions.GetLatestAutoconfigRunSummaryAsync(p, o, r, ct));
+		await Probe("AnalysisActions.HasQuickfixSuggestions", () => client.AnalysisActions.HasQuickfixSuggestionsAsync(p, o, r, TestBranch, ct));
+		await Probe("Sbom.GetRepositorySbomPresignedUrl", () => client.Sbom.GetRepositorySbomPresignedUrlAsync(p, o, r, ct));
+
+		// Gate policies, segments, integrations
+		var policies = await Probe("GatePolicies.ListGatePolicies", () => client.GatePolicies.ListGatePoliciesAsync(p, o, null, 5, ct));
+		if (policies?.Data.FirstOrDefault() is { } policy)
+		{
+			await Probe("GatePolicies.GetGatePolicy", () => client.GatePolicies.GetGatePolicyAsync(p, o, policy.Id, ct));
+			await Probe("GatePolicies.ListRepositoriesFollowingGatePolicy", () => client.GatePolicies.ListRepositoriesFollowingGatePolicyAsync(p, o, policy.Id, null, 5, ct));
+		}
+
+		await Probe("Segments.GetSegmentsSyncStatus", () => client.Segments.GetSegmentsSyncStatusAsync(p, o, ct));
+		await Probe("Segments.GetSegmentsKeys", () => client.Segments.GetSegmentsKeysAsync(p, o, null, 5, null, ct));
+		await Probe("Segments.GetSegmentsKeysWithIds", () => client.Segments.GetSegmentsKeysWithIdsAsync(p, o, null, 5, null, ct));
+		await Probe("Jira.GetJiraIntegration", () => client.Jira.GetJiraIntegrationAsync(p, o, ct));
+		await Probe("Slack.GetSlackIntegration", () => client.Slack.GetSlackIntegrationAsync(p, o, ct));
+		await Probe("Dast.GetDastTargets", () => client.Dast.GetDastTargetsAsync(p, o, null, 5, ct));
+		await Probe("Images.ListOrganizationImages", () => client.Images.ListOrganizationImagesAsync(p, o, null, 5, ct));
+
+		// Organization settings, billing and platform
+		await Probe("OrganizationSettings.GetProviderSettings", () => client.OrganizationSettings.GetProviderSettingsAsync(p, o, ct));
+		await Probe("OrganizationSettings.RetrieveOrganizationOnboardingProgress", () => client.OrganizationSettings.RetrieveOrganizationOnboardingProgressAsync(p, o, ct));
+		await Probe("OrganizationSettings.GetGitProviderAppPermissions", () => client.OrganizationSettings.GetGitProviderAppPermissionsAsync(p, o, ct));
+		await Probe("OrganizationSettings.CheckIfUserCanLeave", () => client.OrganizationSettings.CheckIfUserCanLeaveAsync(p, o, ct));
+		await Probe("OrganizationSettings.ListOrganizationJoinRequests", () => client.OrganizationSettings.ListOrganizationJoinRequestsAsync(p, o, null, 5, null, ct));
+		await Probe("OrganizationSettings.CheckSubmodules", () => client.OrganizationSettings.CheckSubmodulesAsync(p, o, ct));
+		await Probe("OrganizationSettings.GetAiRiskCheckList", () => client.OrganizationSettings.GetAiRiskCheckListAsync(p, o, ct));
+		await Probe("Billing.ListPaymentPlans", () => client.Billing.ListPaymentPlansAsync(ct));
+		await Probe("Platform.ListConfiguredLoginIntegrations", () => client.Platform.ListConfiguredLoginIntegrationsAsync(null, 5, ct));
+		await Probe("Platform.ListProviderIntegrations", () => client.Platform.ListProviderIntegrationsAsync(null, 5, ct));
+		await Probe("Platform.GetConfigurationStatus", () => client.Platform.GetConfigurationStatusAsync(ct));
+		await Probe("Platform.Health", () => client.Platform.HealthAsync(ct));
+		await Probe("Enterprises.ListEnterprises", () => client.Enterprises.ListEnterprisesAsync(p, null, 5, ct));
+		await Probe("Metrics.ReadyMetricsForOrganization", () => client.Metrics.ReadyMetricsForOrganizationAsync(p, o, ct));
+
+		// Tools and patterns
+		await Probe("Tools.ListLanguagesWithTools", () => client.Tools.ListLanguagesWithToolsAsync(ct));
+		await Probe("Tools.ListDuplicationTools", () => client.Tools.ListDuplicationToolsAsync(ct));
+		await Probe("Tools.ListMetricsTools", () => client.Tools.ListMetricsToolsAsync(ct));
+		var tools = await Probe("Tools.ListTools", () => client.Tools.ListToolsAsync(null, 5, ct));
+		if (tools?.Data.FirstOrDefault() is { } tool)
+		{
+			var patterns = await Probe("Tools.ListPatterns", () => client.Tools.ListPatternsAsync(tool.Uuid, null, 5, null, ct));
+			if (patterns?.Data.FirstOrDefault() is { } pattern)
+			{
+				await Probe("Tools.GetPattern", () => client.Tools.GetPatternAsync(tool.Uuid, pattern.Id, ct));
+			}
+
+			await Probe("RepositoryToolPatterns.ListRepositoryToolPatterns", () => client.RepositoryToolPatterns.ListRepositoryToolPatternsAsync(
+				p, o, r, tool.Uuid, null, null, null, null, null, null, null, null, null, null, null, 5, ct));
+			await Probe("RepositoryToolPatterns.ListRepositoryToolPatternConflicts", () => client.RepositoryToolPatterns.ListRepositoryToolPatternConflictsAsync(p, o, r, tool.Uuid, ct));
+		}
+
+		foreach (var line in _outcomes)
+		{
+			Output.WriteLine(line);
+		}
+
+		// A passing test shows no output, so the outcomes can also be written to a file.
+		if (Environment.GetEnvironmentVariable("CODACY_SMOKE_REPORT") is { Length: > 0 } reportPath)
+		{
+			File.WriteAllLines(reportPath, _outcomes);
+		}
+
+		var ok = _outcomes.Count(line => line.StartsWith("OK ", StringComparison.Ordinal));
+		Output.WriteLine($"{ok} OK, {_outcomes.Count - ok} refused or server-side, {_failures.Count} failed");
+
+		_failures.Should().BeEmpty("a failure here is a request Codacy rejected as malformed or a model that cannot read the real response");
+		ok.Should().BeGreaterThan(5, "if almost everything is refused the credentials or organization are probably wrong, and nothing was actually checked");
+	}


_{🟡 MEDIUM RISK}

Suggestion: Extract endpoint groups into focused private async helpers grouped by domain, then keep this method as a short orchestration method while preserving probe order and conditional behavior.

See Issue in Codacy

Six focused tests share the same helpers. Each asserts no failures and a per-area minimum of OK calls; CODACY_SMOKE_REPORT now appends, since several tests write to it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

codacy-production Bot reviewed Oct 12, 2026

View reviewed changes

davidnmbond and others added 2 commits October 12, 2026 13:18

Merge remote-tracking branch 'origin/main' into test/new-modules-smoke

aeb3f07

davidnmbond merged commit 9d90298 into main Oct 12, 2026
8 checks passed

Navigation Menu

Repository navigation

Add read-only smoke tests for the modules written from the spec - #51

Add read-only smoke tests for the modules written from the spec#51
davidnmbond merged 3 commits into
mainfrom
test/new-modules-smoke

davidnmbond commented Oct 12, 2026

Uh oh!

codacy-production Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

codacy-production Bot left a comment

Uh oh!

codacy-production Bot Oct 12, 2026

Uh oh!

codacy-production Bot Oct 12, 2026

Uh oh!

codacy-production Bot Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		await Probe("Segments.GetSegmentsKeys", () => client.Segments.GetSegmentsKeysAsync(p, o, null, 5, null, ct));
		await Probe("Segments.GetSegmentsKeysWithIds", () => client.Segments.GetSegmentsKeysWithIdsAsync(p, o, null, 5, null, ct));

Conversation

davidnmbond commented Oct 12, 2026

Behaviour

Result against a real organization

Limits

Uh oh!

codacy-production Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

codacy-production Bot left a comment

Choose a reason for hiding this comment

Pull Request Overview

Test suggestions

Uh oh!

codacy-production Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

codacy-production Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

codacy-production Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codacy-production Bot commented Oct 12, 2026 •

edited

Loading