Feature Description
When the same event arrives multiple times, there is no guarantee that it will not be re-inserted into the DB, or re-delivered to our consumers multiple times. There is no idempotency guarantee.
Problem / Opportunity
The EventGate Lambda handling events and performing writes, which is the central to the EventGate system, currently perform triple write: DB, EventBridge, and Kafka. If there is a failure, e.g. Kafka write fails, the Lambda finishes (the two writes are done) but it constructs response payload with HTTP status 500.
Then, the producer is notified about 500 and maybe it stores the msg into DLQ or redelivers the message again immediatelly, doesn't matter - the message is processed again and the triple write happens again - even if it previously succeeded for some of the writes. Therefore, we'll have duplicit items in our DB and our EventBridge/Kafka consumers might receive the message multiple times potentially.
Something like this happened with Unify recently (Unify as consumer), although no producer complained about this. But this certainly is something we should deal with and redesign.
Note: DB tables contain almost no constraints and very little auxiliary timestamp-based columns - like when a message arrives, a new column arrivedAt with default NOW() should be present - an idea - because otherwise we DO NOT KNOW when the message actually arrived (well, in this case, when it was stored in DB, the arrival happens seconds earlier). The processing start, column that is present, is set by producer, not by eventgate system. On producer's retries, this is the same value!
Acceptance Criteria
- Scenario with msg-redelivery is handled in better and more expected way, idempotency based ideally.
- Documentation
- Either programmatic tests exist, or the final solution was tested at least manually and evidence is attached somewhere, maybe into this ticket even.
Feature Description
When the same event arrives multiple times, there is no guarantee that it will not be re-inserted into the DB, or re-delivered to our consumers multiple times. There is no idempotency guarantee.
Problem / Opportunity
The EventGate Lambda handling events and performing writes, which is the central to the EventGate system, currently perform triple write: DB, EventBridge, and Kafka. If there is a failure, e.g. Kafka write fails, the Lambda finishes (the two writes are done) but it constructs response payload with HTTP status 500.
Then, the producer is notified about 500 and maybe it stores the msg into DLQ or redelivers the message again immediatelly, doesn't matter - the message is processed again and the triple write happens again - even if it previously succeeded for some of the writes. Therefore, we'll have duplicit items in our DB and our EventBridge/Kafka consumers might receive the message multiple times potentially.
Something like this happened with Unify recently (Unify as consumer), although no producer complained about this. But this certainly is something we should deal with and redesign.
Note: DB tables contain almost no constraints and very little auxiliary timestamp-based columns - like when a message arrives, a new column arrivedAt with default NOW() should be present - an idea - because otherwise we DO NOT KNOW when the message actually arrived (well, in this case, when it was stored in DB, the arrival happens seconds earlier). The processing start, column that is present, is set by producer, not by eventgate system. On producer's retries, this is the same value!
Acceptance Criteria