FIX: Welch Allyn / Mortara Diagnostic Cardiology Suite - Service Crashes and Server Connection Guide
Posted by bensonGpixel@reddit | sysadmin | View on Reddit | 13 comments
In case anyone needs this info in the future, here ya go. No clue how helpful this will be, but here is what I found after days and days of troubleshooting..
The "Leetspeak" Bug (Startup Crashes)
The Symptom: You try to start the CorScribeAppServer service on the server, or launch ExamMgrUI.exe on the client, and it crashes instantly.
The Error: Event Viewer shows a System.IO.FileNotFoundException for misspelled files like ntd1l.dll (with a "one"), msc0ree.dll (with a "zero"), or kern3l32.dll.
The Cause: There is a hardcoded typo in Mortara’s diagnostic module. It tries to inventory 32-bit DLLs in C:\Windows\SysWOW64, but because the names are misspelled, the .NET framework throws an unhandled exception and kills the app.
The Fix: You have to "satisfy" the typo by creating dummy files with those misspelled names.
Run this in Command Prompt (Admin) on BOTH the Server and the Laptop:
cd C:\Windows\SysWOW64
echo. > ntd1l.dll
echo. > msc0ree.dll
echo. > kern3l32.dll
Server Not Available (even after you have configured the server)
The Symptom: Your firewall is open on Port 7502 and the service is running, but the client still throws "Server not available" and tries to connect to localhost.
The Cause: The main EXE configuration is often ignored. The application instead pulls network settings from a half-dozen "Ghost DLL" config files hidden in the ModalityMgr folder. These are all hardcoded to localhost by default.
The Fix: Bulk-update every config file in that directory using PowerShell. Do NOT do this manually in Notepad; copy-pasting into XML often introduces invisible "non-breaking space" characters (U+00A0) that will crash the app parser.
Run this on the CLIENT PC in PowerShell (Admin):
(Change 0.0.0.0 to your Server’s actual IP)
$serverIP = "0.0.0.0"
$files = Get-ChildItem -Path "C:\Program Files (x86)\Mortara Instrument Inc\ModalityMgr" -Include *.xml, *.config -Recurse
foreach ($file in $files) {
(Get-Content $file.FullName) -replace 'localhost', $serverIP -replace '127.0.0.1', $serverIP | Set-Content $file.FullName
}
The "PGDBInterface" Login Crash
The Symptom: You finally get the login prompt! But when you hit OK, you get a WCF FaultException: "The type initializer for 'Mortara.ExamMgr.IntegrationApi.PGDBInterface' threw an exception."
The Cause: This error comes from the Server. It means the client successfully hit the server, but the server crashed trying to talk to its own Postgres database. This usually happens because the DBConnectionString in the server's config was changed to the network IP. Since the DB is on the same machine, it must stay as the local loopback (127.0.0.1).
The Fix: Force the database string back to localhost on the server and bounce the services.
Run this on the SERVER in PowerShell (Admin):
$configPath = "C:\Program Files (x86)\Mortara Instrument Inc\ModalityMgr\Mortara.ExamMgr.IntegrationApi.dll.config"
$configData = Get-Content $configPath
# Replace whatever network IP was there back to the local loopback
$configData = $configData -replace 'Server=[0-9.]+;Port=5432', 'Server=127.0.0.1;Port=5432'
Set-Content -Path $configPath -Value $configData
Restart-Service -Name "CorScribeDBSvc", "CorScribeAppServer" -Force
Disclaimer: I did all the trouble shooting and I did fix everything myself. I then explained everything to Gemini and had it write this up for me. I checked for any errors and hallucantaions and it looks clean to me :)
Tensoneu@reddit
My thoughts coming as a former Healthcare IT.
While appreciate the writeup it sounds like a misconfiguration issue from the start or something was changed. These were my questions when reading the post:
1) Has it ever worked before or worked in production?2) Is this this a new project to bring it into production? 3) Was the vendor asked if their other customers experienced this issue?
If there's any future updates or software that alters this or causes other issues this can be a potential hands off from vendor for unapproved/unauthorized customer fixes.
For others reading this, always go through the vendor first and have everything documented. Often times I find it is rare medical software has trouble communicating with its clients. Usually when this happens it's a misconfiguration (related to either security/permissions, network, or workflow). Most medical server side software has a test system available for troubleshooting purposes. If they don't always request for one or have one built.
bensonGpixel@reddit (OP)
Hey, I really appreciate the input! I totally agree that going straight to the vendor is usually the best move. I did actually reach out originally but I could tell it was going to take forever for a resolution to come about. Which is why I decided to start tinkering on my own.
To answer your questions:
1 &2: Yes, this setup was previously working on a standalone laptop., but it got water-damaged by a broken pipe. Unfortunately, there was zero documentation from the previous techs on how it was configured, so I was pretty much flying blind. I decided to move the Manager to an actual VM on our host server and make the new laptop just a client (which is how it should have been set up originally, but never was).
3: I did reach out to their support after the fact as well to see if anyone else had experienced this issue. Like I mentioned in the edit, they replied and explained what usually causes that initial bug. They did not answer about the other issues.
One last thing, I'm totally open to hear how I could have handled this situation better/faster. I am new to a lot of these systems in Healthcare IT, so I really do appreciate any input :)
Tensoneu@reddit
There's other factors but is this a smaller practice/department? The data entered in the software, does it automatically get sent to the patients chart in the EMR (via HL7/interface engine) or is it done manually?
Considering the software what sounded like was all in one solution. Have you tested with multiple laptops being able to connect to the software on the server side?
If yes, then this is good for scaling purposes, server side VM backup, and easily replace a damaged or outdated laptop.
The negative would be more places you have to troubleshoot if things go awry (laptop, network, server/VM).
There's many factors to determine if the configuration suits the situation. But given that it was time sensitive or quick resolution. I personally would've just done a 1 to 1 replacement without introducing more variables to the original configuration. This is just based on the limited information.
Usually before I make technical decisions I would check: 1) what existing agreement was made with the vendor with purchasing this device. (Yearly maintenance contract, equipment leased?, is it up for renewal, will there be a newer version of the software or equipment.)
2) Is the practice/department happy with using the device/software or were they planning to go for another vendor?
In the end you fixed the issue. Just hope the fix introduced doesn't cause extra work down the line.
tuxedo_jack@reddit
The fact that it's looking for those particular DLLs - and versions with incorrect, obfuscating spelling - is particularly worrisome.
One may have been okay. Two or three isn't a coincidence.
bensonGpixel@reddit (OP)
Yeah good point, I didn't really think too much about it at first actually.
But I did double-check the installer - the digital signature is valid and directly from Welch Allyn. The file hash also came back totally clean on VirusTotal.
I also installed everything on a brand new Windows Server, but I'll look into anything that may be sketch.
mortsdeer@reddit
It's not unknown for software providers to get infected, as well. The local check for viruses is good, but there are stealth or newer viruses. I'd still be awfully concerned seeing this sort of error.
mfinnigan@reddit
This could absolutely be a supply-chain attack (although a weirdly-bad one.) The alternative is that the Welch-Allyn devs are 12 years old on the inside.
tuxedo_jack@reddit
If you've ever had to support CardioPerfect on XP Tablet Edition and have it export to eClinicalWorks, you KNOW their devs have issues.
tuxedo_jack@reddit
Signatures don't mean they're not malicious, only that they're from who they day they are.
Run it in a VM and use InCtrl or similar to track changes.
Kumorigoe@reddit
The amount of people (even on here) that think "valid signature" equals "safe" is pretty bad...
bensonGpixel@reddit (OP)
Yes, agree! I have contacted their support but heard nothing back. Crazy😵💫 It is entirely possible I didn't install something correctly, but idk I'm pretty confident I followed all the manuals and everything.
Down_B_OP@reddit
I have never touched this software, but I would like to thank you for doing the work. Witjout admins doing write-ups like this, we would be living in the dark ages.
bensonGpixel@reddit (OP)
I am just starting out and learning a ton. But it feels good to be able to actually start contributing after years of reading old forums reddit threads like this haha