Point of sale systems going down every time internet hiccups, what are modern solutions?
Posted by NoFerret8153@reddit | sysadmin | View on Reddit | 223 comments
We manage IT for a retail chain, about 40 stores. Our pos system is cloud based which is great until internet goes out and suddenly nobody can ring up customers, happens maybe once a week at different stores, sometimes for 10 minutes, sometimes more.
Looked at backup lte connections but that's expensive to put everywhere and doesn't really fix the main problem that the system needs internet all the time to work. I know there must be a better way to build this where the pos terminals can work by themselves and sync back to main office when connection comes back, but our vendor basically says too bad, their system needs internet. Thinking we might need to switch vendors completely but want to understand what's actually possible before we start that whole process. What are retail people running these days that handles spotty internet well?
The business impact is real, we're losing money and customers are getting mad got yelled at by regional managers last week after a store was down for 3 hours on a saturday.
EffectiveNo8515@reddit
If your system depends fully on internet, outages are always going to hit you. Even a short drop can slow orders break payments and mess up kitchen flow.
What usually works is:
A POS with offline mode
A backup internet connection LTE/5G
Some systems such as Epos Now offer offline support, which at least keeps things running when the network drops...
ConsistentCoat5608@reddit
I used to work for a POS software company, and one of their best features is offline capabilities. Allows operations to run even when internet drops, or you can also setup mobile carts and true up sales at end of the night. DM me for more information, do not want to spam their name here.
mhkohne@reddit
The problem is that the POS vendor has taken the fact that they need Internet to do credit cards as an excuse to put everything in the cloud, so that everything hoses up, instead of just failing to finish the transaction at the end.
sargcj@reddit
This is becoming more of a problem as more systems want to go to Cloud based systems and architecture.
It drives me insane that companies ignore the fact that outages and offline capabilities are important.
If you're not in a position to change PoS solutions, then failover and possibly some SDWAN solution are the directions to go in. Honestly, I'd recommend some form of failover regardless unless it's a very low impact interruption.
Want to validate the cost? Then do an analysis figuring the lost revenue per minute/hour/etc for every downtime and store you have. Does it justify the cost?
You should probably think more than just raw revenue, what about lost productivity and other potential issues that can arise? Labor cost for idle employees gets expensive quickly on top of loss of revenue.
Regarding PoS, this is a crazy world of it's own and so many that claim "works offline" are nothing more than marketing bs, or they simply mean it'll accept a card without knowing if it will actually take or decline.
In my opinion/experience, even "Cloud" systems should have a local system where devices can still work via Local network, and ideally some sort of internal awareness to failover devices between one or more acting as a temporary "server" or "controller". When internet comes back, it can re-sync everything to the cloud and then continue to run normally. (There's some terminology for this that I'm brainfarting on.)
Ultimately at minimum, I'd look into a 4G/5G and/or satellite failover options. Bonus if you can get on a 4g/5g provider that doesn't share same local fiber lines with local ISP.
CPAtech@reddit
Sounds like you're running residential internet for a business if it goes down once a week. If uptime is critical, get actual business internet backed with an SLA.
arkmtech@reddit
So management is just fine with losing many thousands of dollars in sales, repeat customers due to inconvenience, and potentially also front-end workers due to stress, just so they can save a few thousand on Internet each month?
If not, they need to pull their heads out of their asses and get a more reliable ISP + failover connections in place immediately.
When things like this keep happening, people stop coming back.
Useful-Process9033@reddit
This is the conversation that needs to happen in dollars. A single 3-hour Saturday outage probably costs more than a year of backup LTE at every location. If the POS vendor does not support any offline mode at all, that is a massive vendor risk worth flagging to leadership too.
nuttertools@reddit
Doesn’t even need to be a robust business solution that costs thousands per mo. We spend ~$60/mo/site for 2 consumer grade wireless plans as backup. Cloud POS retailers don’t have complex needs.
Expensive_Plant_9530@reddit
I agree that they need to just suck it up and get backup internet.
However, they may not even have the same ISP across stores, given that it’s retail. Or they might not really have a lot of choice when picking ISPs.
Still, a backup connection is a must, and way easier than trying to get the vendor to re-engineer their cloud POS software with offline backup.
loosebolts@reddit
Not only easier, but likely cheaper. If they’ve been cloud only so far they’re unlikely to have servers in stores so budget for a server on each site to host an instance of the software… short term financial impact would be much higher.
mike9874@reddit
We have two connections with different ISP for each and LTE backup at 200+ stores
gonewild9676@reddit
It takes 100 $10 combos to make $1000. They don't get the revenue you are talking about.
asdlkf@reddit
You can get a backup LTE/5G connection from rogers for $99 per month, no contract, 500Mbps down, 30Mbps up, and 999TB/month transfer cap.
You are telling me a business can't afford $100 a month for backup internet? That is $3 per day. If the internet goes out for 10 minutes a week and that 10 minutes costs a $15 sale, then it pays for itself.
Graymouzer@reddit
That would be true only if at least $10 of a $15 sale is profit.
asdlkf@reddit
Doesn't matter because a 5$ profit is a 5$ profit.
A 5$ non-sale is a loss of a future customer and far more dollars.
JMeucci@reddit
If internet is a requirement for the business to function then backup internet at each site should be the solution. No different than people who work remotely 100% of the time. Internet is your livelihood so plan for outages accordingly.
No ISP on the planet guarantees 100% uptime.
enby_dot_local@reddit
Ideally a proper business grade internet connection with decent SLA's, AND a failiver connection like 5G or Starlink
Mindestiny@reddit
Yeah, I'm going to imagine most of these retail locations are just using "business" internet from Comcast or whoever because they don't want to pay for a proper dedicated uplink with an SLA.
OPs business needs to do a proper business impact assessment, and decide what's more impactful - 10 minutes of downtime here and there, or the cost of proper multi-WAN networking to maintain uptime.
angrydeuce@reddit
10000%. This is an issue because the bean counters didnt want to pay more then the bare minimum for the service they require.
I support a couple hundred clients across the US and this is always a pain point...I literally am dealing with this right now, with an internal person that went rogue and signed up for fail over LTE service without running it by us. Dude picked the 20mbps plan because it was the cheapest option...for an office of 200 fucking people both onsite and remote connecting back with VPN.
Imagine their surprise when the failover didnt work how they expected!
Enterprise grade uptime carries enterprise grade costs.
Useful-Process9033@reddit
The bean counter argument goes away real fast when you can show them exactly how many minutes of downtime each store had last month and multiply by average revenue per minute. Most shops just dont have that data because their monitoring is garbage.
timbotheny26@reddit
As someone who is currently stuck in retail, this 100%. Barring a handful of possible exceptions, retail companies run everything as cheap and outdated as possible and will refuse to fix or update anything until they absolutely have to.
Case in point at my store:
* Old, dusty desktops in the back office that look like they're from the 2010s though I can't say for sure. Still running Windows 10 of course and hopefully on extended support but I have no idea.
* Punch-down blocks (M-type 66 blocks) for the VoIP system. (At least that's what I think it's for; the copper leads to a server rack in the manager's office which by the way is always unlocked.)
* What look like early 20 year old Toshiba POS boxes running an OS that I believe is from 2011.
* An equally old Toshiba box which serves as the POS control terminal and is running what I believe is a version of MS DOS.
* What I believe to be nearly 30 year old Fujitsu self-checkout machines running WEPOS 2009. These things in particular are constantly going down and have gotten more unstable since I started working here last year.
halxp01@reddit
Starlink is my goto. 60$ a month. And doesn’t die also when Johnny doesn’t look digger cuts the fiber in town and takes down cell service also.
danieIsreddit@reddit
My business was quoted 50GB for $200/month using Starlink as a backup.
Frothyleet@reddit
For our sole customer using it, they pay $120/mth for 500GB (metered). That's through a broker but seems like the normal cost for Starlink from what I can tell.
Superb_Raccoon@reddit
Cheap as chips for a backup.
GhostDan@reddit
Just when there's a lot of clouds And hope you can live with lower speeds for more price!
Superb_Raccoon@reddit
Starlink is above the cloud!
Chris92Merch@reddit
We haven't been able to get a Starlink business connection for that good of a price. We have to use cellular as our backup. \~$22 a month.
trebuchetdoomsday@reddit
lol SLAs. fiber's out, thanks for the $2.15 statement credit you worthless financial-backed SLA, my fiber's still out. backup circuit is the only answer, preferably configured w/ seamless failover via SD-WAN.
RaNdomMSPPro@reddit
Failover will disconnect the POS session, but SD-WAN will leave that connection up and even optimize it to the best path when things operate normally.
Expensive_Plant_9530@reddit
Even if it does disconnect the POS, that’s a short outage vs waiting for the internet to come back on (10 minutes+ according to OP).
JMeucci@reddit
Exactly.
fmdeveloper25@reddit
When we have clients with Internet uptime requirements, we use up to 3 connections. 1: primary, 2: diverse path wired connection (if available) and 3: 5G backup connection
DJzrule@reddit
It’s literally a no brainer. It’s a non starter for us to open ANY location in my company without primary and secondary ISP providers, both diverse. I’ve even gotten approval for tertiary via 5G/LTE/Starlink at all sites in case a shared telephone/power pole gets hit, and we’re stuck on generator power with no hardwired internet. Broadband circuits are literally <300-500 a month for a gigabit business circuit or similar. You’re probably losing that in less than an hour of being down, probably minutes of being down.
butter_lover@reddit
some downtime is assured with any vendor, if you really can't do a backup connection with a diverse ISP then choose a POS with an on-prem instance that can store and forward transactions when the ISP connection is interrupted.
i'm surprised your vendor says that isn't possible bc i know we've been doing it literally for more than a decade maybe two.
we actually have at least two ISP vendors at each site, some MPLS and some Business class cable modems, and even a few highly managed starlink terminals for really out of the way sites.
if you have 40 sites you should be able to make a deal with a second vendor to become your primary or give you some inexpensive backups.
we used to use a lot of cradlepoints but we've gone away from those over time bc the hw kept getting more expensive.
i was exposed to a project some years back with a US bank with many branches that actually used some small serial terminal console servers with LTE modem that had the ability to route a bit of critical traffic over a tunnel back to services when the main connectivity went down. something to consider i guess.
Brook_28@reddit
In some cases having a 3rd fail over option may be required. It's the cost of doing business. Or they can lose money when they are down and unable to process transactions. If a full back connection is not cost effective, then maybe a cellular? Or get enterprise sla fiber runs to all stores. That won't go down weekly to begin with.
burnte@reddit
Why buy one when you can have two at twice the price?
JMeucci@reddit
Needs to be a different provider/technology.
burnte@reddit
Yeah, definitely need diversity there.
Expensive_Finger_973@reddit
But that is expensive right now, the outage is future someone’s problem.
slykens1@reddit
OP when confronted with the consequences of their design decisions.
31nz163@reddit
XD lol
Expensive_Plant_9530@reddit
I’m with this.
Yeah maybe the vendor could create an offline fallback mode, but I can envision a lot of complications with that, reconciling stock and purchase transactions after the fact.
If they’re losing money, backup internet is the solution.
LTE, or a different wired connection type than the primary.
D0ri1t0styl3@reddit
That’s kind of beside the point, as mentioned by OP. Local caching with delayed sync should be table stakes for a POS system.
Solkre@reddit
Yep. WFH as much as I can. Primary fiber is 2Gbps. Secondary fiber is 50Mbps. LTE backing those up if needed.
Useful-Process9033@reddit
The real question is why you're finding out about outages from store managers calling in angry instead of catching them before anyone notices. Backup LTE is the obvious fix but you also need monitoring that tells you the second a store drops so you can failover or at least get ahead of the call. We built IncidentFox to do exactly this kind of thing for distributed environments. https://github.com/incidentfox/incidentfox
deepwat3r@reddit
I manage IT for a similarly sized retail company, and we use Cradlepoint cell backup units for connectivity failover. ~$700 for the units, plus cell service & small annual fee for remote management has been quite effective for us.
Bottom line, you can't run credit cards with no internet, regardless of what POS you have .
Distracted-User@reddit
Yeah I also support a small retail chain, about half of what OP supports, and all of our locations have 5G failover, and in case everything goes down, each location has a portable cellular card reader.
The person suggesting calling in each transaction has never actually worked retail. There is no way anyone would tolerate that for more than 30 seconds. And with how bad scammers are, most people aren't going to hand over their card wait for you to call some mystery phone number. Never mind the fact that your underpaid cashiers are going to remember how to do it once a year when there's an issue.
deepwat3r@reddit
Yeah I need to add cellular card readers into the mix, but I'm somewhat constrained by the devices supported by Lightspeed Payments. And re: phoning the bank, as you called out, that is a non-starter. We train our staff to never ever handle a customer card.
In the worst case scenario, we have the customer buy the item on our website with their phone and then reconcile the inventory later.
EscapeFacebook@reddit
Yes you can. You just have to call the credit card companies to authorize each sale. We do it all the time when power is out.
traumalt@reddit
If the internet is down, how do you place a call then?
My POTS landline service got discontinued 10 years ago lol, its all VOIP nowadays.
yawnnx@reddit
They'll make you call on your personal cell phone lol.
yawnnx@reddit
Having to call in to authorize each card sale would be annoying and I’m sure not every customer would be on board with that.
CARLEtheCamry@reddit
Wait... Really?
I am imagining in OPs situation, someone comes up to the register, hands you a MasterCard and you call some MasterCard number and read them the card info?
FarmboyJustice@reddit
Most credit card processing worked this way for a long time, usually using a card reader connected to a land line phone. The transaction was performed over a phone line similar to sending a fax.
You can also collect the information locally, save it temporarily, and enter it later on. Obviously that creates more risk, but it's an option.
EscapeFacebook@reddit
Yes. Credit card processors still allow manual authorization by calling in. We have businesses that operate on completely paper credit card invoices for multiple days during disasters like tornados if required. If internet services are down we can run a generator to power the store and we just call in the cards.
trebor_indy@reddit
PCI compliance steps into the room....
EscapeFacebook@reddit
Our company is over 60 years old, we got this.
CARLEtheCamry@reddit
OK, that makes sense in a catastrophe, I learned something today.
I don't think it really fits for OPs ask, when they are too cheap to pay for a backup circuit though.
EscapeFacebook@reddit
Moto card payments shouldn't cost a ton of extra money to set up, it's just something you have to set up with your payment processor. I think the biggest offense for OP is the web based POS, that needs to go. It's also clear they have no contingency plan for when power and internet is out. This business has just determined if they don't have access to their technology they're not selling product. This is a failing of the operations team, they should be training their stores to write invoices manually if they have to.
GeekgirlOtt@reddit
They're usually IVR, so not reading it out, but keying it in.
vintagerust@reddit
If this is high volume like a gas station that already has a customer through the line every thirty seconds or so, this won't save them. They'll keep 1/20th of the sales because everyone else will walk away
EscapeFacebook@reddit
It's not about saving them, it's about being able to process sales for the time the internet's down versus selling nothing at all.
vintagerust@reddit
Sure, technically you can run a card over the phone. But this is about saving sales for the company which that won't do overall. Most customers will go somewhere else and stop stopping at the place that only works some of the time.
Capable-Ad-5344@reddit
Depending on what country you are in.
EscapeFacebook@reddit
I can't find a single law that mentions MOTO being illegal in any country.
GeekgirlOtt@reddit
True, OP should see if they have MOTO capability already in their plan.
draggar@reddit
Huh?
I used to support restaurant POS systems. They had an offline mode (it verified that the credit card number was valid) and then when they went back online it would process them. Yes, some would fail but it was very few.
Or - some sites would have an analog (non VoIP) phone line as a backup. Yes, it was slower, but it would still run credit cards. Some would have something similar, like a wireless modem that connected to a wireless provider.
You can run credit cards with no internet - it just takes extra work but that's just making sure your business can continue to run.
katarh@reddit
I think eating the cost of a few failed CC transactions over the course of 3 hours of downtime is probably less than losing all sales during 3 hours of downtime.
draggar@reddit
Yep. I'd help a lot of sites with recoveries and typically the rejection rate was less than 1% of all transactions and the vast majority of them went through if they manually re-ran them.
RaNdomMSPPro@reddit
The very few retail and "have to take payments even if internet is down" have your same setup - Cradlepoint + 5g. We resell verizon too, so we can sell them the 5G with a static and tap into a pool of data they can share to keep the expense down vs. everyone having their own 5G account.
GeekgirlOtt@reddit
"backup lte connections but that's expensive" "we're losing money"
Is it really that expensive after all if it saves you those sales ? You'd be higher risk rated if you needed to allow manually entering due to risk of employee fraud when they manhandle cards.
SpeechEuphoric269@reddit
Yeah, this is stupid. A 5G cradle point costs less than a thousand dollars (I believe), and then a minuscule amount per month. A business thinking that is too expensive when they lose thousands due to ISP outage is ridiculous.
OP, you have a cloud based system. Invest in architecture to keep it reliable.
FastFredNL@reddit
Thousand dollars for whatever a 5G cradle point is, times 40 stores.
SpeechEuphoric269@reddit
And across those 40 stores, whats their revenue? In the hundreds of millions?
Stupid logic when we are talking about enterprise networking.
FarmboyJustice@reddit
Not sure I'd call 40 small business locations in the same franchise enterprise networking...
GhostDan@reddit
Don't know, if i was with a company that had 40 offices I would say that's a enterprise
FarmboyJustice@reddit
Not 40 business offices, 40 retail stores. Big difference.
GhostDan@reddit
Is there?
I mean they need connectivity, most retail stores now have email and IP phones, apps their employees use, etc. Even a small chain.
Honestly their biggest issue was their decision process, who chose a SaaS product knowing they have connectivity issues in their stores. Fix those issues before you do anything else
FarmboyJustice@reddit
Yes, there absolutely is, and the fact you think there isn't tells me you've got limited or no experience with retail.
https://www.cisco.com/site/us/en/learn/topics/networking/what-is-an-enterprise-network.html
GhostDan@reddit
I managed one of the largests convenience stores infrastructure for the north east, but sure.
Not treating it like an enterprise network explains why there are so many complaints.
FarmboyJustice@reddit
If you did in fact manage a large convenience store chain, then you are absolutely aware of what I'm talking about.
Comparing that environment to a college campus or a major bank makes it extremely clear that it's not even remotely in the same category.
Feel free to continue telling me these are the same and being confidently incorrect. I know better.
SpeechEuphoric269@reddit
Sorry, not enterprise technically, but this is still the scale of corporate networking with corporate levels of profit loss due to downtime. AKA: this is the time to stop being cheap
JMeucci@reddit
Enterprise 5G appliance.
And you don't need to spend that much. Plenty of options available for $250 - $400.
Frothyleet@reddit
Yes. That is a very reasonable cost of doing business for a chain with 40 stores.
GeekgirlOtt@reddit
I'm in Canada; we have some of the most expensive internet and mobile plans. Yet even here we can get a $50-$60 CAD/month ($35-$45 USD) non-contract Rogers cell mifi hotspot.
HealthAndHedonism@reddit
I used to work for a company with around 600 retail stores across Europe. Our larger stores had Meraki with a 5G failover, but at least 500 of our stores had cheap Zyxel 5G routers (around €150) setup in case the fixed line went down. We'd have data only SIM plans with whichever local provider was cheapest.
thecodemonk@reddit
Even a backup starlink would work. You can get the roaming one, never use it, pay $5 a month then turn it on when shit goes south and only pay $50 that month. Or just pay the regular rate..
gmc_5303@reddit
Or just leave it in standby mode and realize that half a meg of service is enough for the POS system to work, and have backup internet for $5 per month. Worst case sign up for roam for $50 and get 100 gigs.
Frothyleet@reddit
Ideally you start with a firewall (can be a cradlepoint) that has an integrated cell modem in the first place! But you don't even have to add a robust solution like a cradlepoint, for backup purposes a $200 5G modem with ethernet that is on your firewall's WAN2 works just fine too.
Although then again if I've got 40 locations I'd like a single pane of glass on managing all components.
katarh@reddit
I WFH and use my 5G hot spot in this exact manner when the internet goes down.
All you can eat data plan from AT&T is like $80/month.
KingZarkon@reddit
Don't some mobile providers sell service for exactly that purpose? It's meant to be a backup to your main connection so it's priced lower than a regular line, but with lower caps (still should be plenty for the POS connections) since you shouldn't be using it much.
DuckDuckBadger@reddit
Can the system not process orders offline, and sync them once internet is restored? Most point of sale systems will support this up to a certain dollar amount. They won’t be able to verify payment methods in real time but that becomes a second ISP to accept the risk business decision.
nadji-bl@reddit
offline first architecture, your terminals need to be the source of truth with cloud as the backup not the other way around. We went through this problem and decided to build our own pos instead of relying on vendors, the messaging layer matters more because that's what handles the sync when connection drops and comes back, we built ours using nats because it has automatic store and forward so terminals buffer locally and replay when online, more reliable than trying to maintain constant websocket connections that break every time wifi hiccups, look for messaging systems that were designed for intermittent connectivity not just cloud first scenarios.
Murky_Bid_8868@reddit
Is the issue not placing orders or credit card transactions? All pos systems have an off line function for placing orders and can batch credit card transactions until the internet comes back up based on a max credit card amount configuration. I usually have the restaurants purchase a carrier tablet and config a square account to process any cc transactions when normal internet goes down. Hope this helps.
bbbbbthatsfivebees@reddit
If there's money on the line, and there absolutely is with POS systems, this is the solution. It genuinely doesn't matter if it "doesn't really fix the main problem", it's a bandaid on the bullet hole of having an unreliable ISP but it's a DAMN good one because it's seamless and will allow the business to keep working with zero interruptions, even if the downtime is only 10 minutes.
Having worked with POS systems in the past, friction reduces sales which reduces revenue. A wireless backup solution is probably appropriate here, see if the ISP offers a good wireless failover solution or just roll your own using TMobile or Verizon.
It's also a lot less expensive than getting the ISP to run new reliable fiber to the location.
nwmcsween@reddit
Easy: Throw a piece of hardware that has 5g connectivity, Starlink, etc. Hard: Architect resilient IT infrastructure similar to what Chick-fil-a did with Kubernetes and move away from SaaS.
enforce1@reddit
SDWAN
loosebolts@reddit
You’re a business that relies on internet connectivity to function, but backup internet lines are too expensive?
That’s a decision you have to make. Either you spend big to reorganise and decentralise your system and bring it all in house and local (buy servers for each site) or get in backup internet lines.
dartdoug@reddit
We're going to be rolling out Verizon's 5G (also works on 4G) internet backup service. Cost is $ 30 per month with a 3GB per billing period data cap. If you exceed the cap in a given billing period you get billed an additional $ 70 and then you have unlimited data. If your outages are brief you would be paying $ 30 per month. They sell a router (which can be set in bridge/IP passthrough mode) for $ 349 or you can BYOD.
nyckidryan@reddit
What's more expensive, a backup internet connection or being shut down for 3 days because someone cut a fiber line in the area? 🤣
Sorry, you rely exclusively on the cloud and you deserve what you get. Maybe an employee will be nice and share their phone's wifi hotspot. 😝
accidentalciso@reddit
What?! A backup internet connection is dirt cheap compared to downtime where a business can't generate revenue. Downtime has a cost that can be calculated. The business needs to quantify downtime in terms of dollars per minute (or hour) so that they can make a real decision about how much to invest to mitigate the risk of downtime.
discgman@reddit
You are looking at the problem all wrong. POS is cloud based which means it needs internet 100 percent of the time. You need better internet that is more reliable, better wifi infrastructure and better backup internet solution.
FarmboyJustice@reddit
Nah, there's plenty of POS systems that can handle offline processing for periods of downtime. And the idea that every single small business must be able to have redundant ISPs is unrealistic. In some rural areas there's literally only one provider.
discgman@reddit
You cant do credit cards without a steady connection.
FarmboyJustice@reddit
This is absolutely false.
jeffrey_f@reddit
Look into lesser internet for backup and you need to set your routers to fail-over to the secondary connection on the fly. A cheap LTE or even 25 to 100MB broadband. Secondly, you need to revisit your POS vendor to talk about this.
Barbarian_818@reddit
"cloud based" that's your problem right there. I'd look into setting up local copies of your cloud set up.
I'd lean towards having the POS units always linked to the local server. Then the local server maintains fidelity with the cloud. A cloud first with fail over to local would preserve the control that head office is used to, but I don't know if your POS has the capability.
eyeteadude@reddit
Cellular failover router and service is your best option for DOS due to backhoe.
2cats2hats@reddit
I was an IT director for a nationwide chain before covid. We had redundant internet in all locations because an abandoned cart at the till was unacceptable for us. It's worth the extra money. We used ECRS back then and they provided a local 1U server for store transactions. Throughout the day it would sync with HQ server(at our head office). Hope this helps.
DrunkenGolfer@reddit
"The business impact is real, we're losing money" and "Looked at backup lte connections but that's expensive to put everywhere"
Seems like a simple cost-benefit analysis to me.
There are dozens of on-prem POS solutions and they mostly work by periodic work by polling. LS Retail, NCR Counterpoint, Retail Pro Prism, Windward System 5, EPOS Now, Clover, Shopify/Square.
MekanicalPirate@reddit
SD-WAN?
OldGeekWeirdo@reddit
A backup system is going to cost money (along with the networking to do the change-over).
But changing vendors is likely to be expensive too. If not in cash, then in the amount of work to get everything transferred over and retrain everyone.
You might ask your question about systems in a sub that's more focused on retail.
Leopold_Porkstacker@reddit
The pos should have a “store and forward” function that will complete a transaction when the internet comes back online, it might need to be enabled.
But yeah, you need better internet for a 40 store chain.
Walbabyesser@reddit
Secure USB at a local PC, tracks all transactions locally - if internet is gone, it still works. If internet is back, it sync data again.
Simplified version of a system I helped deploy
scytob@reddit
sounds like your org is making mistake of looking for cheaper not better
you need to invest in backup internet links, multicloud capability or running your own stuff in your own data centers and dedicated comms lines with backups
there is no free lunch
and ALWAYS know what your manual process is
FarToe1@reddit
Each site will need a primary internet link, a secondary link that is is not supplied by the same wires, and a decent router that can A) Failover and back automatically, and B) doesn't crash and that you can remotely keep updated.
Secondary needn't be anything special. If you're fighting a battle for funding, then cost it out. A PAYG 4g sim so that it hardly costs anything whilst it's not being used. Then when it is, the cost is lost by the continued ability of the site to earn money. Honestly, this stuff is a no brainer to supply.
If this isn't your ballpark, then there are a lot of third parties around that will do all the sites for you and give them support - at a cost. But again, even that cost has to be weighed against not just immediate profit loss, but long term loss of business and employee welfare. It's no fun being the face customers want to shout at.
slykens1@reddit
Is this comment for real?
All I can think of here is the stick in the bicycle wheel meme.
Your client chose a 100% internet based POS. I suspect your client also chooses the least expensive consumer grade internet service at each store. As their IT, your company failed to make them fully aware of the risk. And now they wonder why they’re losing sales.
Maybe you’re not in the US - here, with careful planning and execution, you could provide cellular backup across all locations for a few hundred dollars a month. If that’s too expensive then your client should just close up shop.
hung-games@reddit
Even if the POS wasn’t cloud based, any POS will need network connectivity to authorize payment transactions like credit/debit cards. There are offline models for some card based payments, but they often aren’t well supported and still occasionally need to connect to the network. OP needs backup internet if downtime is unacceptable.
Graymouzer@reddit
Retail used to do credit cards with no network connectivity at all. I worked for a major restaurant chain about 25 years ago and we did batch processing of credit card transactions. We just needed to connect once or twice a day to t he POS systems and pull back the transactions for batch processing. That's uncommon now but the need for always on internet connectivity makes for a pretty brittle system. There should be an offline mode that can work for a few hours or a couple of days. That will happen pretty much anywhere from time to time.
traumalt@reddit
"Used to" being the key word.
Cards also were embossed for the use with an imprinter, and had mag stripes.
Nowadays if it's not chip&pin or tap, then my card is useless and I have to pay with cash.
hung-games@reddit
Card networks support offline models, but they are a PITA for everyone, especially the banks that issue cards. Because of that, they are historically mostly used in markets with poor/expensive connectivity. So they are poorly supported in the US for example which has generally cheap/ubiquitous connectivity.
EMV chip cards allow for offline transactions if the issuer wants to enable that. If so, there is a maximum amount that can be spent before the card has to come online to talk to the network & ultimately issuing bank. So if that limit is $50, then you could charge $15 3 times (for a total of $45) before the card would have to go online for any charge over $5 could be approved. If the terminal can’t go online for the fourth transaction, it will be declined.
There is also a similar counter for the number of offline transactions before the card has to come online.
In some cases like Europe’s PSD2, the issuers are mandated to come online in various situations as well (e.g. high value transactions) to ensure the transaction is secure.
They also don’t work for digital wallet transactions like Apple Pay.
pspahn@reddit
Don't know what sort of retail OP is doing, but for us we still need Internet to do sales tax lookups as a large number of our customers are getting delivery.
FarmboyJustice@reddit
Authorize still provides these capabilities, I'm sure others do as well.
imnotonreddit2025@reddit
It's not real. It's Engagement Farming.
ratshack@reddit
No way this is real. “Oh, we are using ‘cloud’ how can we not rely on ‘cloud’”
I mean what platform are they even running on, is this an Acme POS system or a bunch of thin clients running excel on a Remote Desktop?
It could be anything and without that how is this even a question?!
OP is either a clanker or incapable of doing the needful.
gwildor@reddit
its TOAST - the POS is TOAST. running on the same ancient network stack from 20 years ago that the old POS before TOAST was using. They invested nothing other than the bare minimum to install toast, i guarantee it.
SlendyTheMan@reddit
Toast will hold transactions and will process them once reconnected. The handhelds can also have cellular sims..
gwildor@reddit
"can" - they think a single facility cell connection is too expensive, you think they are paying for 1 per unit?
EscapeFacebook@reddit
Wow what idiot went with a cloud based POS? Get rid of that asap.
ATL_we_ready@reddit
You’re getting downvoted because most cloud POS also support offline and offline cards. Same as an on prem. You have reduced functionality during an offline which is always the case for on prem as well.
EscapeFacebook@reddit
OK? His doesn't. So what's "most?"
ATL_we_ready@reddit
Or he doesn’t know the right questions to ask…
bfodder@reddit
Yeah the people in here saying "fIx uR iNtERnET" don't get it.
raginghavoc89@reddit
This is how we do things at the Fortune 500 I work for. Some stores have up to three modems but we can still sell parts if we dont have internet due to on-premise POS servers and we can still make credit card sales by calling them in. If power is out completely we just write paper invoices. This might shock a lot of people but businesses still operated before the internet and cash registers.
ResolvedByReboot@reddit
Idk either, nothing you said is incorrect. How the hell can a company not have a contingency plan for when the power/internet is out so they can still sell product? If you're dumb enough to leave your entire business on a web based POS and not have a back up plan they deserve the mess. This is a business plan issue, not just an IT issue.
thatfrostyguy@reddit
Its very early in the morning where I am, but did you look at alternative ISPs? Fix the thing thats broken, which is internet access.
bfodder@reddit
It isn't unreasonable to want to be able to accept payments during an outage.
Cyberprog@reddit
You can't have things both ways.
1) Your POS requires an internet connection - DSL is unreliable sometimes, so invest in 4G/LTE backup. Here in the UK I'm paying I think £5-10 a month for a data SIM with a shared data plan with all our other mobiles.
2) Find a POS that has an offline mode - you may have less features while offline (can't place PO's etc.) but you should still be able to sell stuff.
We are in the first camp, as that's the best for us and the cost of a backup is peanuts compared to the loss of business.
nighthawke75@reddit
Offline payment option, local servers to keep things humming.
ultradip@reddit
I used to do installs for retail. We used Cradlepoint routers that failed over to cellular if the primary Internet connection dies.
Worked in 90% of locations. The rest were mall locations that were interior with no windows and got little to no signal.
Klintrup@reddit
I'm sure you could find a new POS setup that works offline, though you would likely still not be able to accept credit cards when the internet is down - depending on your country that may or may not be a problem.
Replacing your POS is expensive, you need to get new POS terminals for all stores, retrain all customer facing employees, re-integrate with your central IT systems as well. Storing data locally might also present new compliance issues (PCI/GDPR) depending on your location. Furthermore you'd be adding a big risk of losing transaction history if one of your POS devices broke down while offline. You could have a local server in each store, but that would be adding even more cost to the solution.
What I would do in your situation is to sketch up a couple of solutions that you believe you can actually manage, estimate the opex/capex for each store and present these solutions to management.
Depending on your store layout/revenue streams you may present a tiered system, so the stores with the highest revenue (which would have the highest risk of revenue loss on during outages) could select the most reliable solution, which would likely come with the highest cost (dual homed with SLA + 5G backup), medium revenue stores could select a 24x7 SLA + 5G backup solution and low revenue stores could stay on the current solution (no backup).
If you want to go the extra mile, talk to finance/store managers and try to get an estimate of the lost revenue from internet outages for the past 12 months per store, and pre-sort them into high/medium tiers as a recommendation, you could also provide an estimate of what a full outage allowed by the current SLA (if there is one) - what if the highest revenue store is offline for an entire weekend ?
FarmboyJustice@reddit
Nice! An answer that doesn't summarize as "git gud" for a change.
jedimarcus1337@reddit
I have seen 2 solutions to this:
- One system has a database replication to the store, so the store POS is totally independant of the master database, can work offline for days and then replication will kick in once connection is restored. Note: this is not a web app
- Another system (web based) will have enough data stored locally to continue working with some downsides (maybe not all the pricing rules will be applied or customer store credit not working)
maglax@reddit
I mean no Internet means no credit card processing. Sure they can still sell without Internet, but only to a minority carrying cash.
FarmboyJustice@reddit
Credit card processing without live internet is still possible today. The risks are a bit higher but you can absolutely do this.
ibreatheintoem@reddit
This is not entirely true. Toast, Revel, and I'm sure others support 'deferred' payments. Meaning if the internet is down for whatever reason terminals can ring up customers paying with tap or card, but the charge only gets sent/validated once the device is back online.
Sounds scuffed but cards getting declined for whatever reason is pretty rare as a percentage of tractions at the places I've worked. Losing 0-2% of card payments during an outage due to payment declines not being caught is much preferable to losing 100% of card payments during an outage.
jedimarcus1337@reddit
OP didn't specify where he is from. I see a lot of card terminals that have 3/4/5G by default, because the terminal provider doesn't want to debug your network connection too.
tropicbrownthunder@reddit
Those are not very common in big surfaces or medium -large retail. You use the integrated reader with the POS hardware.
Definitely a couple of those mobile card readers might help when ISP kicks the buck
Pure_Fox9415@reddit
Will this local database run on thin air? They think lte router is expensive and you advise them to have a database server + ups + wiring + whatever this setup needs.
Imbrex@reddit
The pos systems in familiar with supported a saf feature to cache transactions for approval until connectivity resumed. Of course they could be denied, but the risk was worth it. This was like 10 years ago though.
bensode@reddit
If the losses in combination with negative brand hits are greater than cost of redundancy and resiliency, it’s a no brainer. Do the cost analysis and present your findings. There is no magic wand for this. This is the cost of business - either spend it for infrastructure or lose money transactionally.
zer04ll@reddit
Clover and Square have offline payments, most modern POS do. You have 48 hours to get it back online for the transactions to be processed after that they are lost. Honestly LTE is a solution you jus haven't found the right provider. If you interested I can get you in contact with a company that handles this.
Its simple how much money on average is being lost, find a solution that comes under that and it technically is paying for itself and then some. So if they are out 4k per month across all stores then spending an extra 3k on backup internet solutions makes them money
jocke92@reddit
It seems like the one who bought the system didn't specify the correct requirements. But you've got two options. Look for a new system that is able to work without a server and catch up when the internet is back again. Sounds like something like that should be available.
And if you want to stick with the current system, get a backup internet connection. Starlink or 5G. Your firewall does also need support for dual ISP.
TrustMeImAnOnion@reddit
…laughs in Win XP POS with serial receipt printers…
Dave_A480@reddit
Even if your POS wasn't close based it is still going to be dependent on connectivity back to the main office (I mean, you could do some really hacky 1980s style batch processing fuckery, with local servers in each store updating a central system at HQ each night - but no modern system does that)....
Dual internet connections (ideally one being satellite or cellular) and the appropriate network gear to automatically switch between them is the most affordable way to go with this.....
cubic_sq@reddit
Need the right solution that can handle communications outages. Which if the thousands of retails solutions out there probably less than 10 handle this well.
It can still be an app on a tablet. But needs to he coded to handle this.
RandomGen-Xer@reddit
What's the cost of these outages versus, say, installing Starlink as a backup internet provider for these 40 stores?
Coldsmoke888@reddit
I have some big sites running cloud based server solutions. Each site has 2 landlines with diversified connections and an LTE external cradlepoint or Starlink. The cost is insignificant compared to downtime and lost productivity/sales.
The lines are on hot failover so no manual intervention needed.
AcidBuuurn@reddit
Generally the cheapest backup internet I’ve seen is a 5g ISP set up as either failover, load balancing with failover, or SDWAN. Get a few quotes- some of them are less than $50/month.
kona420@reddit
SDWAN and stack cheap internet, whatever is cheapest quickest to install. This is what the fire and forget cloud solutions were made for as they are expensive per mbps of throughout but you really dont need much for transactions. Otherwise roll your own at the head end.
braytag@reddit
Cloud based POS is the stupidest thing ever.
What's the point? The advantage?
Nothing on prem prevents you to:
Sending logs to a server. Updating clients once a day.
Am I missing something? Any upsides?
a60v@reddit
This. It seems like a massively stupid design failure. And if the vendor failed at this, one must wonder what other design mistakes were made. Change vendors now.
braytag@reddit
A decade ago, I was approached to work on a cloud based POS, I thought it was the dumbest thing ever back then, and 10 years later, I feel validated.
You can always have a hardline phone backup for cards transactions with on prem.
But 100% cloud based, wtf do you do if some construction worker rips your fiber out while digging and you are down for a week?
Bankruptcy ?
Away-Ad-3407@reddit
what do you consider expensive? what kind of prices are you being given?
Ad-1316@reddit
*please be weary of low priced options! They can have insanely low bandwidth caps, and high fees for going over that cap!
JJHall_ID@reddit
Our POS systems all have local databases so they can continue to operate as normal in the event of a network outage. The only thing that will stop working is online card processing. To combat that, we have two connections to every one of our stores, typically one being a business fiber, and the second one usually a broadband connection (cable, shared fiber, etc) from a different provider. Our Meraki gateways also have cellular failover so in the event that both hardlines are cut (like if agents are trying to trap someone) there's still a wireless option. We're testing using Starlink for the secondary line in a couple of our stores and so far it's working well. If push comes to shove, our registers are capable of being put into "offline" card processing, sometimes called "store and forward." In this scenario we take on some risk in that cards can be declined after the fact when we go back online and push in the batch of offline transactions, but in our experience the loss is dwarfed by the sales we would have lost if we just had to turn card customers away.
If you're interested in our solution, feel free to reach out and I'll get you in touch with them. I have no skin in the game if you go with our vendor, other than they may buy me a dinner next time they're in town. :-)
Centimane@reddit
That's how you end up with a bunch of bounced credit card charges.
Tower21@reddit
I'm hoping this is reposted on shittysysadmin, it really deserves to be there.
DonkeyTron42@reddit
A pencil, paper, and a calculator. Otherwise second ISP.
Frothyleet@reddit
We have a customer with about 70 retail locations in our state. Every store we set up for them, their primary circuit is usually business grade shared fiber, and the firewalls are models that accept a SIM card for cellular failover.
Their POS is also cloud based. I am not sure if it works offline, but if you have an internet dependency for your locations to function, you need resiliency in your WAN.
Anxious_Youth_9453@reddit
Why is backup LTE too expensive? You don't have to partner with a carrier directly. There are MVNOs where you can pay by the gig. You can get 1 or 2GB of data for under $20/month. Be creative.
Puzzleheaded-Sink420@reddit
Idk most POS Systems I work at can store like 5transactions and Need Connection to the cash register. If the register needs always on, then there is NO way around a Backup Internet Connection.
Pure_Fox9415@reddit
Sounds like this "retail chain" is a chain of village micro-shops. How is it possible, that router with usb port, + modem (or native sim support) + sim with prepaid 1gb of traffic is too expensive? It's like 100$ once and 10$ monthly? Huge money! Ok, if it's enterprise-grade lte-router with outdoor case, maybe $200-300?
qkdsm7@reddit
"backup lte is expensive" but probably one of the top short term solutions, to something that likely should have been scienced out before implementation...
Could get a tunnel over mptcp or some such SD-WAN with a secondary ISP, and never even hiccup when WAN1 goes down.
Exploding_Testicles@reddit
A secondary circuit (vsat or cellular) just for POS and essential systems.
stromm@reddit
I suspect your POS provider offered a hybrid cloud/on-premise solution and the powers that be chose to save money by not getting that.
amw3000@reddit
What is the cost of losing internet access/sales vs supplying LTE connections? What is the cost of switching POS vs LTE Connections?
Not sure where your located but in most places, when your dealing with 40 stores, you can get some pretty good rates for a stable connection + LTE failover. Are you working with a broker?
FastFredNL@reddit
Failover to 5G/Starlink is the only way
What is the connection being used for?
- If the usage is file access, you need to locally store all the files and sync them when there's a connection, this can lead to sync errors and conflicts so not an option. Also, you need infrastructure to store and manage all files which is waaaaay more expensive then a firewall/router with a SIM card and a data subscription.
- You talk about calling customers (by phone?), in that case your proposed solution is not gonna work anyway (also the backup solution to that is to call them using cellphone)
- You're talking about retail? Bank transfers/payments at the checkout need to happen instantly. Pretty sure you can't hold those back and send them out when the customer has already left and you have internet back an hour later.
eums@reddit
The provider should offer an offline solution. We use epicor and they have offline modes for both cloud or hosted (propello or eagle).
RaNdomMSPPro@reddit
SD-WAN, a real SD-WAN service will probably fix this with the minimal investment. It doesn't probably need a ton of bandwidth, but you are going to have to spend money on it for the SD-WAN, plus at least one more internet connection to have that redundancy. 5G is a decent option if you have solid signal. shared fiber isn't much more these days. If you have enterprise wan circuits you could switch to two shared fiber from different carriers (make sure they are truly separate) plus sd-wan service and you've got redundancy that is much better than you have now. Some sd-wan let you involve a 3rd or 4th connection.
If the POS will run on a tablet you could have that w/ 5G and a stripe reader to ring up folks in an emergency to limp along.
What are the BCP plans for these stores? Do you have ways to ring anything up and take a payment w/out internet - think stripe for example on a phone w/ 5G.
Steve----O@reddit
Might as well cancel the main internet circuits. Just think of all the money you will save.
newtekie1@reddit
A backup 5G connection cost me $42 a month. Is the business lost from outages totaling more than what it would cost to put 5G backup at each location?
RetroHipsterGaming@reddit
As much as I agree with everyone that you should have a backup connection if your business requires internet to function, I will say that it sounds like these businesses are using unreliable internet. If it were me I would start by figuring out which internet everybody's using and if there is any consistency in which providers are causing you grief, have them switch from that internet provider. Other than that, getting some kind of backup internet connection can be really invaluable.
When you get a backup internet provider, make sure they go from different backbones than the primary. A lot of Internet providers especially in rural places will use the exact same backbone to provide the service.
Dermotronn@reddit
Cellular fail over is the best solution. In a similar industry but without almost 100% internet the terminals become almost useless (live SQL lookups for accounts). Since going to broadband primary and cellular as backup there's rarely been a transaction missing due to internet outages. The cost is about 25k per year but that would be lost in sales a less than a month without it.
gwildor@reddit
i mean: you've got a few options.
1) call the ISP and get the issue resolved.
2) call the POS vendor and ask them their preferred solution
3) obtain backup internet via hardline.
4) obtain backup internet via cell
5) obtain backup internet via starlink
Xidium426@reddit
Pretty common with cloud software, this question should have came up during the hunt for new POS software.
You only really have two options, swap POS to something that can support offline mode or get redundant internet.
alemonaday@reddit
We use t-mobile Inseego devices as back up/failover. They’ve been pretty stable with speeds of 200mbps or higher. It’s around $50/mo with no data caps.
Canuck-In-TO@reddit
Get another ISP to quote on and install another connection. Either as a failover or to replace the problematic existing connection.
I have a client that has had terrible 25Mb service from AT&T. AT&T had nothing faster.
Also, with such a slow connection, they still had employees connecting their cell phones to it, which would completely choke the connection.
I finally got Comcast in there and they put in a 300Mb service. Everything runs much better now and I don’t get complaints about slow service.
Macia_@reddit
Cloud POS systems are a PITA, but the solution you're describing is going to be a far greater headache.
I found 2 solutions to this problem:
Cellular internet backup (Recommended): POS systems don't typically need a lot. All but the most rural of stores should be able to run off a cellular internet backup. At some locations you may need to mount an exterior antenna, but it's a solid backup solution.
Bluetooth card terminal. Think Square or something similar. They connect to a smartphone app. It won't hook up to your POS software, but we never had any issues that weren't solved by a few knowledgebase articles. The account manager for your payment gateway should be able to hook you up.
A third option is to have the employees write down card information to charge later, but there are a LOT of problems with this approach. Number 1, you're introducing PCI-DSS requirements to the store. Number 2, you're screwed if they give bad information or the cards don't work.
I definitely recommend option 1. A national provider can send you these systems in advance for store buildouts and you'll need almost no config on your part. Still, make sure whoever you go with gives your admins access to configure the WWAN device remotely
ratshack@reddit
“…but I’m not going to state what software we are using because that would be a clue”
TheEvilAdmin@reddit
I use to install P.O.S. systems and all the data from the registers were sent to a server in the office of the store. This being moved to cloud is so dumb. I went to a circle k in my area and there was a big internet outage. No one could buy anything. People were trying to pay with cash, but they weren't even taking cash. Either cause they just couldn't use the register or they didn't now how to math and enter it later. Maybe both.
As others said, maybe a backup internet connection, but if there is a big internet outage, how would that help?
the_doughboy@reddit
I got out of Retail just as SaaS based POS systems were a thing but if I remember correctly a lot of them had Internet Failure modes where one of the terminals acted as quasi server.
You still needed internet for payment processing though so you'd be cash only until internet was back.
Rabiesalad@reddit
If a business relies on internet it should have a failover connection.
An on-site POS system isn't a bad idea but is it really going to be worth it? You'll introduce additional complexity to maintain at each location and I can't see it possibly being less cost/labour than a backup connection.
Right now it's a pain with the network goes down but how does that compare to your in-house self-maintained solution going down? If there's no expert available at a moment's notice you could be looking at full day or multi-day outages when it goes down.
Affectionate-Cat-975@reddit
If a second internet connection is not an option then you have to point up for something like NCR’s Aloha which does do offline transactions. And it will not be cheap
sam_knighthood@reddit
We also have a cloud based EPOS but it syncs a local product info database on each till so that the tills can work without internet. The loyalty, customer info, and giftcards didn’t work though.
Before we got a Leased line per site we had 2 card readers per site that had a 4G connection so weren’t reliant on internet, but these could not be integrated so transactions had to be manually typed in. These cost about £15 a month each so easily paid for themselves if/when needed.
Turbojelly@reddit
Unfounded Conspirqcy: The person that created Point Of Sale devices decided on that name just so the initials would reveal it's true nature. POS.
Dr-Webster@reddit
Having a local server at each store to "cache" transactions may seem like a good idea until you consider credit card processing. Even if you could cache those, is management OK with the idea of letting a transaction go through while the Internet connection is offline, but having the card end up getting declined when it comes back? Worse, what if someone figures out they can get away with using a stolen/deactivated credit card when the store's Internet goes down? The shrink cost could end up being more than a backup Internet connection.
chodu_editz@reddit
offline first architecture, your terminals need to be the source of truth with cloud as the backup not the other way around. We went through this problem and decided to build our own pos instead of relying on vendors, the messaging layer matters more because that's what handles the sync when connection drops and comes back, we built ours using nats because it has automatic store and forward so terminals buffer locally and replay when online, more reliable than trying to maintain constant websocket connections that break every time wifi hiccups, look for messaging systems that were designed for intermittent connectivity not just cloud first scenarios.
Xev007@reddit
have you calculated the actual money lost from these outages? sometimes that number is what it takes to get budget approval for switching vendors. If you're losing $10k a month in sales because the system is down, spending $50k on moving makes sense really fast.
SevaraB@reddit
“Business broadband” and DIA (usually dedicated fiber) are not the same thing, and this is a good example of that. Broadband is built around oversubscription, with logic to scale up capacity or regularly update CG-NAT servers to meet demand, which can look an awful lot like a daily 10-minute “blip.”
But the core issue you’re describing is your circuits are a bad match for your POS solution and one or the other needs to change. You either need a journaling POS system that can cache transactions long enough for the circuit to recover, or you need to invest in higher-quality circuits with SLAs.
bazjoe@reddit
Change POS to local or install backup like starlink or cellular. When I managed it for this kind of thing all locations had cellular and a dual wan router and cellular was up for ping all the time. It’s likely under $200/mo to have a full time auto switch backup. I have heard of people investing in iPads with cellular in them so they have one terminal up but how to take cash, credit cards, scan things…
n1ck-t0@reddit
Choosing a POS system that requires internet was a business decision
Choosing to not have internet reliable enough to ensure uninterrupted operations is also a business decision
The role of IT here is to make a recommendation for the business to make the decision, let them own the consequences not IT
Bulky_lenda_@reddit
offline first architecture, your terminals need to be the source of truth with cloud as the backup not the other way around. We went through this problem and decided to build our own pos instead of relying on vendors, the messaging layer matters more because that's what handles the sync when connection drops and comes back, we built ours using nats because it has automatic store and forward so terminals buffer locally and replay when online, more reliable than trying to maintain constant websocket connections that break every time wifi hiccups, look for messaging systems that were designed for intermittent connectivity not just cloud first scenarios.
LeeKingbut@reddit
Run pos system locally. Then update each shift to main.
maddler@reddit
secondary connection
non-cloud based POS
There isn't really much more.
povlhp@reddit
We have a store server which sends data to POS. Our setup is designed/built to work offline.
Cloud/online can query store server for stock and local prices. If up.
mikevarney@reddit
Get better internet. That much downtime just isn’t the norm these days. But as others have said, failover LTE is your answer. It can be built into routers these days, and realistically is really inexpensive.
talin77@reddit
Local database on a NUC in shop and a sync, that is how i solved it for 200 stores with an POS directly connected to the main server (in another country) we always had time-outs, stores in shopping streets with a lot of digging, internet outage etc.
Timely-Film-5442@reddit
this is common with cloud only pos systems. the vendors love them because it's easier for them to manage but it totally screws retailers in places with bad internet. you need offline first setup where the terminal is in charge and cloud is just for collecting data and reports.
poizone68@reddit
I'd think you'd want at least a SIM based ISP backup. Exchanging your POS for 40+ systems is bound to be more disruptive to your business than adding in a data plan subscription.
brentaarnold@reddit
If you’re on Toast you can get a second internet connection and Toast will configure failover for you.
WiskeyUniformTango@reddit
I've setup added redundancies like this for cloud based pos retailers.
Active passive firewalls, wired ethernet with wireless fail over, 2x installed physical circuits for internet + cellular fail over as a 3rd internet. Further in some instances also added cellular sim cards in some of the endpoints if the store was extra critical and had prior isp issues.
Then go down the stack with N+1. Ie 2 UPS, on-site generator, etc.
MrJingleJangle@reddit
Having a local system with uploads to central hubs used to be The Way, but it was expensive, an IBM System/36 which was a common store machine will male your LTE cost worries look trivial. Redundant Internet connections is now The away, welcome to the cloud.
Accurate-Ad6361@reddit
In which country are you located?
starfish_2016@reddit
Netgear lbr20 + $20/mo 15gb on Verizon > plug into your router wan2. Bam. $20/mo fail over
BOOZy1@reddit
Get better internet or get redundant internet, or both. Since your POS is cloud based, and can't gracefully handle outages and you can't change POS vendors you can only change your internet connection.
Use 4G or Starlink as your secondary internet connection and block all non-POS devices from using this connection.
It'll cost you a few 100$ extra per month but that's peanuts compared to lost sales and lost trust from customers.
Bonobo77@reddit
It sounds like an on prep solution like Aloha could work for you. Advantage would be everything works locally and then syncs up to the NCR cloud or whatever provider are you end up going. Only issue is if an LTE backup is considered expensive, I’m not sure what your locations would feel about the cost of maintaining 40+ servers.
You could also look at upgrading your entire Internet solution to an enterprise MPLS network. Then build out a single on-prem Aloha database, centralizing, and hopefully simplifying your infrastructure. And with some larger MPLS providers you can always negotiate an LTE backup.
FelisCantabrigiensis@reddit
If the essential service is remote, then you need redundant connectivity - from two carriers, with failover between.
Or get a service that can run offline locally for a while. That requires a different POS design with a local server providing service to the actual POS devices and syncing to a central system every so often if it goes offline, or some other way to work without connectivity.
You can phone in card authorisations if your point of sale system is down, or have a merchant contract that allows you to accept a certain amount of transactions offline.
CertifyWithAI@reddit
Are these stores on fiber? What gives that's very poor. If its the ISP replace them, but if you provided the routers perhaps check there is no odd config issue there.
RevolutionaryWorry87@reddit
If this is thus business critical, two ISP lines seems the best way to go. Shouldn't be too expensive either.
ryalln@reddit
Does the POS have any like on prem caching options. Old system I used tills could store details them phone home or export details to manually upload. Or is this a flat web browser based system? Cause if your loosing money and there isn’t a solution change platforms.
noodlyman@reddit
Do you accept cash?