LDAP keeps breaking and we have no idea why

Posted by DDRDiesel@reddit | sysadmin | View on Reddit | 26 comments

So, we have LDAP set up on several copiers throughout the company so users can scan to their email. We also use it on our SonicWall for user authentication against AD as well as few other appliances on the network. I'll get a call from a user that the copiers aren't pulling up any results, go to check using the LDAP tools in the copiers web interfaces, and confirm the issue. Then within 10-15 minutes, it resolves itself, and everything works again. The AD server isn't going down, resources aren't getting tied up, and there's nothing running that shouldn't be. This only started happening recently, so I was thinking maybe an update was to blame, but nothing comes up in any search results. Server is running Windows server 2019 standard, if that helps. It is also used for DNS, DHCP, and primary domain controller

Reply to Post

26 Comments

[-]

No_Winner2301@reddit

Can you not use port mirroring on the copier port and use wireshark on the mirrored port to see what is happening?

[-]

MinidragPip@reddit

When a machine starts to have this issue do other machines have it too? If it's happening to everything at once, that points to the DC or something in between like a core switch. If it's just some machines, but not all, look at the network segment they are on.

[-]

DDRDiesel@reddit (OP)

Yes, all the copiers at once will have issues with LDAP lookups. They are all plugged into different switches due to their physical locations throughout the building,and the DC isn't going down. It's pingable and accessible through RDP. Event Viewer logs on the server also show nothing wrong

[-]

MinidragPip@reddit

Ping and rdp? But can you do ldap lookups from a PC or server? Just because ping works doesn't mean ldap is accessible. Do you have more than one DC? If you point a machine at a different one will it work?

[-]

ez12a@reddit

I've caught a bunch of obscure and hard to repro transient issues with a script with a loop before. Def recommend at least something doing some sort of synthetic testing for troubleshooting, heck even implemented somehow as LDAP health monitoring long term.

[-]

DDRDiesel@reddit (OP)

We used to have multiple DCs, but the secondary was taken offline about a year ago. At first I thought the secondary server being listed but unavailable was causing the issue, so I took it out of the LDAP configuration, but the problem persisted. I can see if LDAP lookups from the other network appliances work when the copiers go down, that would at least help me narrow it down a bit

[-]

MinidragPip@reddit

Get that second DC back. Living with one DC is just asking for trouble.

[-]

Apachez@reddit

And also how is the network setup? Have you segmented it into VLANs or is everything a puke galore where one evil client can either send broadcasts to block all other traffic or send PAUSE frames to screw up printers? Not to forget the ip spoofing issue... Do you perhaps have any firmware updates available for your printers?

[-]

DDRDiesel@reddit (OP)

The network setup is... not great. No VLANS, but instead different subnets are set up for different purposes. Set up as a class B address scheme but using class C subnet masking. Both my director and myself have identified that the network segmentation needs cleaning up, but at this point the monster has grown so large and the tentacles so entangled that unraveling and doing it properly would simply be too big of a project with how little manpower we have. HOWEVER: that has not posed an issue in the past and only just recently are we experiencing this LDAP issue

[-]

Apachez@reddit

We left the classes back in the 80's. Nowadays we use CIDR :-) The classics would then be "did you turn it off and on again"? And I would prioritize to VLAN segment your network so printers sits by themselves along with private/protected vlan so printers wont hog each other - anyone who want to print something will talk to the printerserver instead of directly talking to the printer. Also firmware update the printers and perform factory reset on them and reconfigure them to whatever settings are needed. There is also this perhaps affecting your environment? https://www.bankinfosecurity.com/patch-alert-remotely-exploitable-ldap-flaws-in-windows-a-27221

[-]

thomasmitschke@reddit

Maybe the issues are your copiers? Most of them have shitty software.

[-]

zerotol4@reddit

Do you have an AD account lockout duration set? The AD account the devices are all using to bind to AD may be getting locked somewhere and once the lockout duration is triggered the account is automatically unlocked

[-]

DDRDiesel@reddit (OP)

Now that you mention it, one of the appliances set up for LDAP was only done so recently, and we do have a lockout duration set in our policy. I'd have to check the settings on that tomorrow to see if the credentials are the same that the copiers are using. This is the only potential lead I've had in the past week. Everything else has been a dead end

[-]

wrt-wtf-@reddit

This here is the point to start at. Look at the common touch points and work from there. NTP can be another source of pain but I wouldn’t expect it to recover unless there’s something else broken upstream.

[-]

DDRDiesel@reddit (OP)

I checked the other appliance and no problems there. LDAP lookups work with no issue and no lockouts. I've enabled auditing for user accounts on the DC and going to monitor those logs the next time the issue pops up. I've also seen that different accounts were used to set up LDAP for other appliances so I'm going to test lookups on those if the copiers throw a fit to confirm if it's an account thing or something bigger

[-]

Tr1pline@reddit

Each service account should be for each appliance, don't use the same account or it will be harder to troubleshoot.

[-]

pdp10@reddit

Though a very good idea, this is also extremely troublesome to set up and manage at scale. Luckily, service accounts with passphrases aren't a thing outside of MSAD.

[-]

Sinsilenc@reddit

Not to mention easier to laterally move.

[-]

TrueStoriesIpromise@reddit

And...you can set up more than one LDAP account. you can even create one per copier!

[-]

KStieers@reddit

Altools will help you track down which printer is doing it. https://www.microsoft.com/en-us/download/details.aspx?id=18465

[-]

glirette@reddit

Account lock out policy could certainly be causing issues Also the correct tool to use to reproduce LDAP activity is ldp.exe not anything else It takes some getting used to but is pretty easy once you do

[-]

brainstormer77@reddit

You may be victim of external password spray attack if you have Sonicwall SSL VPN enabled or any open authentication web page. Combine this with account lockout policy and you will get multiple accounts locked for a while. The DC/LDAP security logs will show all locks coming from the one server. Check Sonicwall for failed authentication logs and that might be the issue. If that's the case, lock down Sonicwall ASAP.

[-]

DDRDiesel@reddit (OP)

We use SSL VPN with MFA through NetExtender client. I'll go through the logs and see if I can find authentication issues

[-]

LDAP keeps breaking and we have no idea why

Reply to Post

26 Comments

No_Winner2301@reddit

MinidragPip@reddit

DDRDiesel@reddit (OP)

MinidragPip@reddit

ez12a@reddit

DDRDiesel@reddit (OP)

MinidragPip@reddit

Apachez@reddit

DDRDiesel@reddit (OP)

Apachez@reddit

thomasmitschke@reddit

zerotol4@reddit

DDRDiesel@reddit (OP)

wrt-wtf-@reddit

DDRDiesel@reddit (OP)

Tr1pline@reddit

pdp10@reddit

Sinsilenc@reddit

TrueStoriesIpromise@reddit

KStieers@reddit

glirette@reddit

brainstormer77@reddit

DDRDiesel@reddit (OP)

brainstormer77@reddit

Quirky_Oil215@reddit

fuzzylogic_y2k@reddit