Customer doing my job like a pro
Posted by Independent-Wish-725@reddit | sysadmin | View on Reddit | 78 comments
Soooo, i have a customer that's a dentist, i stopped working for them a while back cause every invoice became a debate and i don't have the energy for that. Turns out during the "forgotten time" (3 months) said dentist installed antivirus that included a SQL db on the server, you can imagine how many things that broke.
TLDR my first day back included a 3 way call hearing that they had to pay £12k to upgrade their software so the business could function again :)
davidflorey@reddit
Remind the dentist why you don’t perform your own teeth work 🤣
CosmologicalBystanda@reddit
I went to the dentists and asked em to rip a tooth out. The dentist said, no, I dont recommend that, I can fix it. Then she tried to justify it, and I said, you're the expert, if you're telling me you can fix it and it's worth it I believe you.
She thanked me for saying that as most people ignore their advice and want what they want.
dj_shenannigans@reddit
On the other hand, if you put a crown on mostly decayed tooth, you know you have a potential customer from then on that had to get the crown redone every few years. Monetarily, it makes more sense for them to save it, but for the person, it makes more sense to just pull it off you don't wanna keep spending the money and dealing with it
andrewa42@reddit
WTF sort of AV needs an SQL back end?
Viharabiliben@reddit
I’ve managed some big enterprise AV. Trellix uses a SQL db to keep track of the thousands of clients, their scan results, updates, module versions, settings, etc.
3Cogs@reddit
We have about 5000 end user devices and use cloud based Sophos. It works well for us. I have no idea how much the subscription costs though.
Viharabiliben@reddit
We were on Sophia and we’re happy with it but couldn’t keep it once they moved to cloud management. We are not allowed to use anything cloud connected, so had to move to Trellix. It’s been a PITA.
Stonewalled9999@reddit
A lot of but it’s better than Cylame and a fair number of others
andrewa42@reddit
That makes sense, but what the heck did a dentist manage to find that uses one…a typical dental office (around here at least) is eight to twelve endpoints.
cpayne22@reddit
One that debates every invoice
Droid126@reddit
You don't end up buying SQL if you're the kind of person that debates every invoice. I HAVE YET TO SEE A SOFTWARE THAT REQUIRES SQL COST MORE THAN THE SQL LICENSING ITSELF
Dal90@reddit
SQL Express most likely.
\:hangs head in shame since I use it to back some of my big scripts because I don't want to deal with the paperwork to have the DBAs spin databases up for me on the corporate high availability, far more storage SQL cluster\:
thepotplants@reddit
DBA here. Giving you the squinty side-eye.
"You only feel guilty... when you are"
Mr_Squinty@reddit
Same. It’s because they use stored SQL procedures and SQL scripts to do 90% of what the product says it does!
Main-Tank@reddit
It's this. I worked in door access control and indeed 80% of the business logic lived in the database.
RaNdomMSPPro@reddit
Remember, he knows more about this stuff that the it folks.
CaptainZhon@reddit
Yet spend money on frivolous crap
aretokas@reddit
I'm surprised anyone in health wanted Anti-Virus tbh.
Greedy-Lynx-9706@reddit
lol
fresh-dork@reddit
google and a complete lack of understanding his own needs?
officialbrushie@reddit
One that might be tied into a centralized repository
Wyatt_LW@reddit
Gdata i think
deucemcsizzles@reddit
/r/MSP has some golden dentist client horror stories lol
UltraSPARC@reddit
An MSP can confirm. Health care providers in general are a PITA. You would be surprised just how blatantly most providers are violating HIPAA. We’ve had to have many “you cannot use a free Gmail account and a consumer google drive to conduct business with patient data” conversations too many times to count only to be met with blank stares along with “so I’m still confused why we can’t do it this way”. We do not take on any new health providers because of all the issues we’ve had. I’m not going to be liable for some massive data breach. I’ll let some teenager they find on Facebook do that for them LOL
Argyros2@reddit
Can confirm very true.
Financial-Chemist360@reddit
I had a doctor client , many years ago, who scoffed at my objections saying that he had helped to write the HIPAA regulations and knew everything about them. Yes sir, I’m quite certain a one man practice with an x-ray machine so old the only parts source is eBay is so noted in his field that he was consulted as a subject-matter expert on patient privacy refs. /s
Stopped dealing with him soon after when he wanted a failed desktop repaired with a motherboard, you guessed it, only available on eBay.
l337hackzor@reddit
My health care clients, particularly dentists, have been the worst. I replaced their whole office with new PCs in preparation of win 10 EOL. Turns out their Pano scanner can't work with 11. Spent hours on the phone with support to determine that.
Had to keep one old machine running and I bet they will keep it running long after EOL just to avoid new Pano.
To be compliant here data has to be kept in country. They all violate that with Google Workspace or Microsoft 365. I have a msp cloud backup software I resell, one is cheaper and better but it's in USA (not compliant) but I offer a compliant alternative in Canada for $5-$10 more a month, they still go with cheaper option.
Shit drives me crazy.
UltraSPARC@reddit
LOL your customers opt in to back their data up?
l337hackzor@reddit
I've never had push back on server backup. I've had some not want to go cloud backup but they've all been sufficiently terrified of losing everything.
Capable_Stranger9885@reddit
I once had a job in a university lab with some analytical testing tool connected to a 386 tower. The thing booted up in 11 seconds though
l337hackzor@reddit
I have a bio company client that has two windows 7 computers. Has to stay on to 7 to run the software for that particular machine.
386 is crazy old though, irreplaceable.
MegaThot2023@reddit
The machine can't work with W11, or the manufacturer refuses to update the drivers/firmware in hopes that offices will just have to buy a new one?
l337hackzor@reddit
The machine didn't support a high enough firmware version, is what the manufacturer told me. Insisted they need a new one.
Sillyb if you ask me. Windows 10 and 11 after so similar.
Financial-Chemist360@reddit
I think the last straw for me - besides the risk of getting dragged in to the legal swamp of their complete lack of understanding and observance of HIPAA - was the cardiologist’s office with at least (guessing) $1,000,000 of specialty medical equipment and a stapler holding the receptionist’s monitor cable in just the right place to keep a stable image on the screen. Touch it and get screamed at but not allowed to replace it. See ya. Life is too short to deal with that level of stupidity.
kalamiti@reddit
In healthcare can confirm. The amount of small dental clinics using free email providers is insane.
0RGASMIK@reddit
Yeah I see them and it makes sense why I hate my dentist. They always mess up my bill and it takes them 12 months to figure it out.
tnmoi@reddit
Uhm, I am no sys admin so forgive me if I ask this question: what is wrong with installing AV with SQL server database? We have this on all our SQL servers and no issues.
wildflowersinparis@reddit
Also have this same question. Genuinely curious.
bzomerlei@reddit
If i had to guess, when the AV was installed, it broke the earlier version of SQL. It is also not a good idea to install other apps on a SQL Server, especially if it is running production critical services. There is a reason virtualization is so popular, spin up a small VM to run the AV would have been better assuming they use that environment. The IT support cost for the fix was probably more than the cost of another server license.
Got2Bfree@reddit
What would be the point of having an AV on a completely isolated machine?
My guess would be that the dentist wanted to protect his server.
bzomerlei@reddit
The OP did not explain the specifics of the AV. Most stand-alone AV clients have no need for a database as part of the install, but they do exist on enterprise AV systems that keep track of clients and their status; that was the assumption I made that could explain why a newer version of SQL was installed as part of the AV.
Got2Bfree@reddit
Fair enough, seems kind of overkill for a few PCs at a dentist.
bzomerlei@reddit
Agreed. For a small office, it is more likely they have a single server hosting many service and probably not a VM.
Got2Bfree@reddit
I'm not a sysadmin but I have a home lab with proxmox.
Full VM backups have saved me countless times after I fucked up.
Pushing encrypted offside backups is also extremely easy and cheap.
I think I would even set this up in a dentist practice.
Jayteezer@reddit
Or my thoughts was stupid developer and it upgraded the version of sql installed instead of installing a new instance. If the updated version was beyond the drivers the software was compiled with this would not be good...
Have seen it before - only once in 30 years at least and we had a full backup of the dB and server before the third party installed their app.
BBO1007@reddit
I think the “AV” or whatever software was upgraded their SQL DB to. Version past support for their old dentist software.
SydneyTechno2024@reddit
Sounds like whatever AV they installed brought along its own copy of SQL server.
From the SQL perspective, it should be fine having both 2014 and 2022 installed.
From the software side, it was probably written by a useless company that decided they didn’t need to specify which ODBC driver to use. Newer version gets installed and their application automatically tries to use that instead despite the older compatible version still being present.
Or something similar.
wizardglick412@reddit
That was my question. Why did the new SQL server version mess up the old one?
pr06lefs@reddit
Either nuked the old sql server instance, or installed the AV db into the wrong instance which is too old and already has tables in it, or the existing db software tries to read/write to the wrong sql server instance. Or something else along those lines.
CoffeePizzaSushiDick@reddit
Like EDR? Or like some Norton or MalwareBytes shizz?
progenyofeniac@reddit
Symantec used to do this, I’m sure plenty others do too.
jepperepper@reddit
it replaces the original one, and it's a different version so things will break.
gboccia@reddit
It scans a shit ton of files and can really throw things through a wrench if it’s a very active DB. Most AV have a list of exceptions and guides on where to consult for an exclusion list.
EitherExamination343@reddit
Not one either but it’s likely that something fails during the changeover or conflicted with their other application. I work in application support and it wasn’t uncommon to find an on-premise software stopped working because they installed a new version of SQL Server and decided to uninstall the other…where their database was.
Main_Ambassador_4985@reddit
Is antivirus a problem on an SQL server?
Is it the lack of CPU, RAM, or the IOPS on the SAN?
I have Microsoft Defender Server 365 EDR antivirus protecting all of my MSSQL servers on-prem. In Azure they are SaaS Databases with Azure Defender protections.
We have many TB of Databases without issue. The VMs do average 256 GB of RAM, NVMe SAN, and at least 16 vCPU each.
Stonewalled9999@reddit
SAN? You mean 60 apps on one server with 16 GB RAM and 5400rpm 1 tb desktop drive
bluedefender8@reddit
Haha its a dentist office, likely a single server with AD, DNS, the practice management system, the imaging software and god knows what other “add-ins” and “SAN” is 3x 7200rpm drives in a raid 5 if you’re lucky. Maybe server 2016 with 16 GB of RAM
ace00909@reddit
I'd consider anything with 2 disks in RAID1 in a dental office a god send. All of them I worked with that brought their own equipment looked like they were running on prayers and a bad sneeze away from going out of business.
datOEsigmagrindlife@reddit
Read the post again.
Also do you honestly think a dentist office has a SAN..
andrewa42@reddit
I handle around 100 dental offices. Exactly none of them have a SAN.
notHooptieJ@reddit
One of them has a hygienist named Sam though.
andrewa42@reddit
How did you know?
She's pretty easy-going (for a hygienist).
zeus204013@reddit
I suffered small health providers. Like calling for a backup of something, but I never was paid for that... Calling family members about my schedule, ignoring my privacy. Really not worth.
coyote_den@reddit
lol damn I used to do SCO servers for a MSP. We had a lot of dentists. Every single one of them had the root password set to “canal”
Even tho the customer knew that, they also knew better than to ever log in as root unless we were telling them exactly what to do over the phone.
kagato87@reddit
Update database set compatibility level 160.
I'll only charge them 10k for that fix.
Shaggy_The_Owl@reddit
‘It’s just an update. What could go wrong?’
colossalpunch@reddit
Just keep clicking Next!
SuccessfulLime2641@reddit
cue my last post
Shaggy_The_Owl@reddit
We’ve all been there…
--Chemical-Dingo--@reddit
Dentists & Vets are my cheapest clients by far
1a2b3c4d_1a2b3c4d@reddit
But
So, they learned their lesson? Or will you have the same issue with getting paid in the future?
huntinwabbits@reddit
Dentists earn enough, they can afford it! :)
DogThatGoesBook@reddit
Hope he didn’t have to sell his Pinarello
Sinister_Nibs@reddit
PRS Private stock
Mukimpo_baka@reddit
Comes down with lack of change control, and dare I say lack of backups, not sql or av issue, just incompetency at work
zakabog@reddit
Isn't that officially supported by Microsoft? I've dealt with this before with servers running legacy software and some new piece of software needed the latest SQL. I don't recall it being a problem, the apps just talk to their relevant databases.
ugus@reddit
lol
SuccessfulLime2641@reddit
bargain demons when will they learn