Repository navigation
ci: notify Slack when release jobs fail, are cancelled, or never run - #353
Merged
Merged
Conversation
posthog-ruby-sync Compliance ReportDate: 2026-11-10T14:31:47.193961+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 8ms |
| Format Validation.Event Has Uuid | ✅ | 5ms |
| Format Validation.Event Has Lib Properties | ✅ | 4ms |
| Format Validation.Distinct Id Is String | ✅ | 8ms |
| Format Validation.Token Is Present | ✅ | 6ms |
| Format Validation.Custom Properties Preserved | ✅ | 5ms |
| Format Validation.Event Has Timestamp | ✅ | 7ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 5ms |
| Retry Behavior.Retries On 503 | ✅ | 5239ms |
| Retry Behavior.Does Not Retry On 400 | ❌ | 5ms |
| Retry Behavior.Does Not Retry On 401 | ❌ | 6ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8012ms |
| Retry Behavior.Implements Backoff | ❌ | 294ms |
| Retry Behavior.Retries On 500 | ✅ | 5156ms |
| Retry Behavior.Retries On 502 | ✅ | 5150ms |
| Retry Behavior.Retries On 504 | ✅ | 5149ms |
| Retry Behavior.Max Retries Respected | ❌ | 530ms |
| Deduplication.Generates Unique Uuids | ✅ | 13ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5113ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10295ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5119ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 13ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 7ms |
| Compression.Sends Gzip When Enabled | ✅ | 5ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 4ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 2ms |
| Batch Format.Multiple Events Batched Together | ❌ | 12ms |
| Error Handling.Does Not Retry On 403 | ❌ | 4ms |
| Error Handling.Does Not Retry On 413 | ❌ | 4ms |
| Error Handling.Retries On 408 | ✅ | 5112ms |
Failures
retry_behavior.does_not_retry_on_400
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.does_not_retry_on_401
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.implements_backoff
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
retry_behavior.max_retries_respected
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
batch_format.multiple_events_batched_together
Expected 1 requests, got 5
error_handling.does_not_retry_on_403
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
error_handling.does_not_retry_on_413
500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 8ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 7ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 6ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 6ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 4ms |
| Request Payload.Groups Round Trip | ✅ | 5ms |
| Request Payload.Groups Default To Empty Object | ✅ | 4ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 5ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 4ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 5ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 1ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 4ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 7ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 4ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 108ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 146ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 6ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
posthog-ruby-async Compliance ReportDate: 2026-11-10T14:32:30.315053+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 108ms |
| Format Validation.Event Has Uuid | ✅ | 106ms |
| Format Validation.Event Has Lib Properties | ✅ | 107ms |
| Format Validation.Distinct Id Is String | ✅ | 106ms |
| Format Validation.Token Is Present | ✅ | 106ms |
| Format Validation.Custom Properties Preserved | ✅ | 106ms |
| Format Validation.Event Has Timestamp | ✅ | 106ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 6ms |
| Retry Behavior.Retries On 503 | ✅ | 5306ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2109ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2108ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8014ms |
| Retry Behavior.Implements Backoff | ✅ | 15722ms |
| Retry Behavior.Retries On 500 | ✅ | 5212ms |
| Retry Behavior.Retries On 502 | ✅ | 5212ms |
| Retry Behavior.Retries On 504 | ✅ | 5211ms |
| Retry Behavior.Max Retries Respected | ✅ | 15622ms |
| Deduplication.Generates Unique Uuids | ✅ | 110ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5211ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10416ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5214ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 110ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 106ms |
| Compression.Sends Gzip When Enabled | ✅ | 106ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 106ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 4ms |
| Batch Format.Multiple Events Batched Together | ✅ | 110ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2109ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2110ms |
| Error Handling.Retries On 408 | ✅ | 5211ms |
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 107ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 106ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 107ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 106ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 106ms |
| Request Payload.Groups Round Trip | ✅ | 106ms |
| Request Payload.Groups Default To Empty Object | ✅ | 106ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 107ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 107ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 106ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 104ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 110ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ✅ | 105ms |
| Retry Behavior.Retries Flags On 502 | ✅ | 256ms |
| Retry Behavior.Retries Flags On 504 | ✅ | 243ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ✅ | 109ms |
Failures
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']
SDK performance: posthog-ruby2 changes worth reviewing: 2 observed regressions. Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate.
View all results
Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals. Each shard uses its own runner; compare Before PR against With PR within a row, not across shards. Compared 102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in |
ioannisj
force-pushed
the
chore/release-alert-on-cancelled
branch
from
November 10, 2026 12:35
7cf1bdd to
ffd5ac1
Compare
ioannisj
changed the title
ioannisj
marked this pull request as ready for review
November 10, 2026 14:10
![posthog[bot]](https://avatars.githubusercontent.com/in/1203049?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
PostHog Review · Standard · reviewed a18b145: 2 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
Comment on lines
+14
to
+18
| const attempts = readdirSync(directory) | ||
| .map(name => /^release-slack-thread-(\d+)$/.exec(name)?.[1]) | ||
| .filter(value => value && Number(value) <= attempt) | ||
| .sort((a, b) => Number(b) - Number(a)); | ||
| if (attempts.length === 0) return undefined; |
Contributor
There was a problem hiding this comment.
P2 · Handle the single-artifact download layout
When a run has exactly one thread artifact, the pinned actions/download-artifact v8 extracts thread-ts.txt directly into THREAD_DIRECTORY, without an artifact-named subdirectory. readThreadTimestamp therefore returns undefined despite a valid saved timestamp, and the alert starts a new channel message. Support the flattened single-artifact layout.
Comment on lines
+71
to
+77
| const publishedGems = publishJobs | ||
| .filter(job => job.conclusion === 'success') | ||
| .map(job => /\((.*)\)$/.exec(job.name)?.[1]) | ||
| .filter(gem => gems.includes(gem)); | ||
| if (!gemsPublished && publishedGems.length > 0) { | ||
| problems.push(`• Published: ${publishedGems.join(', ')}`); | ||
| problems.push(`• Not published: ${gems.filter(gem => !publishedGems.includes(gem)).join(', ')}`); |
Contributor
There was a problem hiding this comment.
P2 · Determine published gems from publish outcomes
When gem push succeeds but a later Rails step fails, publishedGems excludes Rails and the alert says Not published: posthog-rails even though RubyGems already has it. Successful jobs that skip publishing also count as published on Rails-only releases. Track the publish step outcome separately from the overall job conclusion.
Comment on lines
+74
to
+75
| const publishJobs = jobs.filter(job => isJob(job, publishJob)); | ||
| const gemsPublished = publishJobs.length > 0 && publishJobs.every(didPublish); |
Contributor
There was a problem hiding this comment.
P2 · Exclude intentionally skipped packages from publication requirements
On a Rails-only release, the core job intentionally skips Publish posthog-ruby. If Rails publishes successfully but tagging or GitHub release creation fails, publishJobs.every(didPublish) stays false: the alert claims the release failed and lists the unchanged core gem as unpublished. Exclude intentionally skipped packages from publication requirements and missing-gem reporting.
turnipdabeets
approved these changes
Nov 10, 2026
turnipdabeets
left a comment
turnipdabeets
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Every job and step name the notifier looks up matches this repo's release workflow, and the privileged workflow only runs code from the default branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
A release job that gets cancelled never reaches Slack. The failure notification is a step inside the release job gated on
failure(), and GitHub reports a job timeout, a runner it can't schedule, or a manual stop ascancelled, which stops the job before that step runs. That is howposthog-cliwent five versions without publishing before anyone noticed (PostHog/posthog#132516).Ports the release failure notification workflow from posthog-ios. It runs on
workflow_runafter every main-branchReleaserun and replies in the approval thread when a job failed, timed out, was cancelled, or never ran.release.ymlgains apreserve-slack-threadjob that saves the approval thread timestamp as a run artifact so the alert can find the thread. The in-job Slack failure replies are removed, since the notifier covers them; the PostHog failure events stay. The publish job now has a per-gem name, and a partial publish lists which gem went out and which did not. RELEASING.md gains a short section on the alert. No feature flag controls this change. CI-only, so no changeset.💚 How did you test it?
actionlintreports nothing new andnode --checkpasses on the script. I haven't triggered a cancelled release to watch it fire.📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset file🤖 Agent context
DRI: @ioannisj
Autonomy: Human-driven (agent-assisted)