Handling Billing Race Conditions and Duplicate Bill Number Issues
Posted by retro_rude007@reddit | ExperiencedDevs | View on Reddit | 16 comments
I’ll keep it short. I’ve been working on a billing race condition issue for the past 1 month. There are duplicate bill numbers being generated, and some bill numbers are getting skipped.
We don’t have a tester and cannot afford Kafka, so I used Redis + BullMQ to replicate a queue for processing bills. However, this has made the process slower. My plan was to let the system stabilize with this slower version and then move to a faster version later.
The issue, though, still persists. I test it on my system as much as I can, but whenever it goes live, it fails. I understand that there is a huge difference between local testing and production environments, but I don’t know how to ensure this doesn’t happen again.
Honestly, I’m at my wits’ end. My manager scolds me every day for it, and I feel stuck with no clear way out.
Flashy-Whereas-3234@reddit
You haven't articulated your technologies or the sources of the doubles/skips.
There's an infinite combination of solutions for this from where you are.
Consider semaphores, mutex, atomic incremental redis values, optimistic write and pessimistic read, preallocations, uuids, giving each node an allowance of IDs before pulling from the pool.
Idk man, zoom out a bit and think.
retro_rude007@reddit (OP)
we are using mongo as database so the unique id part is not the issue we have and internal logic that generates the bill no as per the choice of the distributor the can set there series and then according to that series and specifications the bill no are generated forward on now the thing is that my initial first fix was making the bill no unique so the one issue of bills being duplicate would resolve but the thing is that there are distributors who have identical bill requirements so there bills can be duplicate so that fix failed and the skipping race condtion was not getting fixed the race condition was occuring during bulk billing where all bills were trying to secure the bill no for them selves and did not which bill is taking which no so i serialized it using a queue and gave them order and if any bill gets missed they are resend back to the end of the queue but the processing has become very slow and many time it fails and i am not able to find solution for it
F0tNMC@reddit
I’m sure this makes sense to you, but your system needs to be expressed more clearly for us to understand and help.
BoBoBearDev@reddit
Are you saying UUID is duplicated? Or are you saying you didn't use UUID? I am not in the field, but my immediate idea is to use UUID so client cannot guess other invoices.
retro_rude007@reddit (OP)
we are using mongo as database so the unique id part is not the issue we have and internal logic that generates the bill no as per the choice of the distributor the can set there series and then according to that series and specifications the bill no are generated forward on now the thing is that my initial first fix was making the bill no unique so the one issue of bills being duplicate would resolve but the thing is that there are distributors who have identical bill requirements so there bills can be duplicate so that fix failed and the skipping race condtion was not getting fixed the race condition was occuring during bulk billing where all bills were trying to secure the bill no for them selves and did not which bill is taking which no so i serialized it using a queue and gave them order and if any bill gets missed they are resend back to the end of the queue but the processing has become very slow and many time it fails and i am not able to find solution for it
dbxp@reddit
Oh, mongo and billing do not go well together. I would look into what sort of consistency model you've got.
A common practice when working with NoSQL is have Mongo handle the high traffic bits before you have a sale but as soon as it comes to spending money you use something like Postgres or MySQL.
dbxp@reddit
I think that may result in issues when trying to integrate with some accounting systems
dbxp@reddit
How many workers do you have consuming the queue?
retro_rude007@reddit (OP)
i have created only one worker the billing worker that consumes the queue i batch process three bills at a time
dbxp@reddit
ok, what are you using as a store for the latest bill number?
Perhaps the duplicate figures are linked to fresh startups of the worker after it has been killed for being idle too long
retro_rude007@reddit (OP)
we are using a seperate schema that holds the lates bill no count for the distributor when he started this bill series and other information about his billing but we store the latest billno over there seperately
dbxp@reddit
Add a bunch of log lines to the worker to track what is coming in from the queue and the DB. Ideally add a check to the end of the log to check for duplicates too, then you can have it log an error so you get alerted.
It's going to be one of those silly little things you kick yourself over.
alien3d@reddit
i would scold back who design the system.1st,did you really need redis and mq. . 2. How much mini whatever api processing to complete all thing. 3 . how do you handle atomic transaction between one server or many server or define a queue flag e.g service 1 complte , service 2 complete , all service -complete , fail .
Dry_Author8849@reddit
You can't have race conditions for billing. You need a transactional system. You know, begin transaction, commit, rollback.
There is a reason those exist. And stop running in circles and learn SQL. Your next problem will be reporting.
Cheers!
ninetofivedev@reddit
Can’t wait to hear what product you’re shilling from your sock puppet account.
retro_rude007@reddit (OP)
no it is not sock puppet account i am a looking for some help and suggestion