[klesgidis]

Summary

How we identified

A consumers pod received SIGTERM while it was creating a HelloSign signature request. The process exited immediately, before that handler finished and before Kafka committed the offset.

HelloSign had already created the request and sent signature_request_sent. The pg-boss job looked up the employee document by that request id, found nothing, and failed after three retries. Kafka then redelivered the same message to another pod, which created a second request and saved that one. The employee got two signing emails. Only the second is linked in HRIS.

What happened

The exit was not a graceful shutdown. orka's SIGTERM handler is registered with process.once and does not wait for disconnect(). Once it runs, the listener is gone. Honeybadger then sees no other SIGTERM listener and calls process.exit(). The in-flight handler is killed instead of being allowed to finish and commit.

After the change

The handler now stays registered, waits for every consumer to finish and commit, then disconnects the producer. The wait is capped by kafka.shutdownTimeoutMs (default 25 seconds). shutdownKafka() is exported for the app to await before it exits. Consumers are tracked in a Set, because the old Map was keyed by group ID and dropped an earlier consumer that shared one.