Transactional outbox pattern, apply or not to
Posted by Aki59@reddit | ExperiencedDevs | View on Reddit | 17 comments
Our one microservice is implemented with a classic dual write problem. They are updating dynamodb table status to received first and then sending message to sqs. If messege sending failed then in catch block we are again calling dynamodb to update the status as failed.
My rationale:
To use aws dynamodb stream push event to eventbridge and then to sqs.
Only drawback is it will incur extra cost and effort.
Their rationale:
1.Pushing to sqs rarely or never fails and even if it fails then updating db will not fail almost always (it just never happens). I also think that it will work 99 percent of the time.
2.They say even if it fails we are logging error message which will trigger alert to prod support which can manually mark the status to failed.
I do believe outbox pattern is a clear pattern to be implemented but how to counter the above points is a challenge.
6a70@reddit
send the message to SQS first, and then update dynamo
if you fail updating dynamo, then your process should just try dynamo again on a message that was already written. But the systems processing SQS should be idempotent
Megamygdala@reddit
Makes the most sense to me
PmanAce@reddit
This is backwards. Always send the event after if not doing the outbox pattern. If the event fails you can always replay the deadletter.
6a70@reddit
it depends on the semantics of your message
PmanAce@reddit
No because you're coupling the exception handling with two different domains which you shouldn't do. Each should have independent strategies, which sending the message last permits.
6a70@reddit
we're obviously on different pages because what you wrote has nothing to do with what I thought we were talking about
OP has not clearly articulated the use case and why they are writing to Dynamo nor what the SQS message represents. That'll influence what is appropriate
anubus72@reddit
weird that you’re downvoted, this is a legitimate solution depending on the problem
auto_off@reddit
Hmmm… all the other answers seem missing guidance. Don’t pull from ddb event stream as it couples your table data model and makes migration 3x harder. Write a separate events table for outbox listener if u want or do a poller.
Checklist to determine what u should do:
IdeaJailbreak@reddit
I will say the transaction outbox should be considered a last resort. It’s great at creating better guarantees, but at great cost if the intent is to keep something else up to date with the same information. I would only recommend it in a case like this where you’ve inherited a bad situation where the data is just incoherent and you need to fix it “fast”.
I would rather have an eventually consistent event based system OR make sure ownership of data is as separate as possible so that there are as few possibilities for incoherence as possible. E.g. datastore A and B share an ID but own entirely separate information about that resource. If a client needs information about the resource, consider standing up a single endpoint for them to hit that reconciles the information from the two backends etc.
subma-fuckin-rine@reddit
Why doesn't it send to sqs once the outbox record is read later?
Mountain_Sandwich126@reddit
"Hardly fails" is not a answer. The team needs to think about when it fails. And also place the burden on them to update the status. Dev shit , op shit
RelevantJackWhite@reddit
What will it cost you? What is the cost of the failed messages that trigger prod support?
Will these costs change as the business changes?
IMO those are the two factors to consider here
dreamoforganon@reddit
These are the right questions. I’m not familiar with sqs or dynamodb, but have seen systems where things that “just don’t happen” when the system is healthy can start happening all the time if there is an outage somewhere.
AlistairX@reddit
Seems to me you’re overthinking it - write the data to Dynamo including the message content and then relay from there to SQS via EventBridge pipes.
There is a cost for that of course, is your product team willing to pay extra for the guarantee? It’s pretty cheap though 🤷♂️
gfivksiausuwjtjtnv@reddit
Instead of outbox canyou just publish duplicate messages? Makes it simpler
ok_annie@reddit
> To use aws dynamodb stream push event to eventbridge and then to sqs.
I'm confused about how this is a transactional outbox.
aookami@reddit
aint dynamo a BASE db?