question on SSSD, keytab refresh and host tickets
Posted by Unexpected_Cranberry@reddit | linuxadmin | View on Reddit | 9 comments
So, I'm trying to get smart card authentication working reliably in an environment with Redhat 9.x clients joined to Active Directory.
We've now gotten to a point where we can get it working, but only for a while.
The issue we're seeing is a case mismatch between entries in the keytab and a jproxy implementation trying to authenticate.
When a machine is freshly joined, the keytab contains records for the client in both upper and lower case, like so
host/COMPUTER\$@REALM
HOST/COMPUTER\$@REALM
With that, everything works fine. However, once the password rotation happens and the keytab is refreshed, we're only getting the upper case ticket. This breaks authentication and you see an error in the secure log
credential verification failed: Cannot find key for host/COMPUTER\$@REALM kvno x in keytab
Looking in the keytab, I can see that there is no entry for kvno x with a lower case host/, only upper case.
I've been trying to figure out what's going on. We are currently joining the machines using net rather than realm, not sure if that's what is tripping us up. I'm wondering if this is something anyone has seen before and knows how to solve. If there's something I can add to sssd.conf that would be easier than trying to convince the Linux team to switch from net to realm...
I have a test environment, and I haven't seen the issue there yet. I'm not sure how to simulate a password refresh to see if I can break my test environment in the same way as prod is currently broken.
sudonem@reddit
Yeah generally most don’t use net join.
You can do realm join, but I would instead I’d advise creating templates for sssd.conf, and krb5.conf with the settings you need - then use adcli for the domain join.
If you use realm join, then you want to script the entire process because realm join will modify your sssd.conf file and if you aren’t scripting it you can end up with heavy config drift.
IDEALLY you’d be doing this automatically via Ansible - which makes templating those conf files and applying with adcli pretty straight forward.
Also - look into joining the domain using certificates rather than having to enter domain admin user creds.
The next step would be auto joining using Ansible-pull during server provisioning.
Unexpected_Cranberry@reddit (OP)
To clarify. This is automated by saltstack which is managed by our Linux team.
They're currently using net ads to handle the domain join, and I haven't gotten an answer yet how they're handling password rotations.
The reason I'm looking into this is that I've been tasked with implementing a new solution that requires authenticating using virtual smart cards. Turns out, this is dependent on having lower case host/ records in the keytab, otherwise pre authentication fails because it can't find a matching entry to authenticate against the domain. This is annoying for the Linux guys since as far as they're concerned, everything is (mostly) working with the current setup.
I'm a Windows guy. I'm fairly well acquainted with AD, but Linux is fairly new to me. I'm doing my best to learn as much as I can since if this POC goes well I'll be supporting Linux as well.
Right now, everything works because we've rejoined our machines to the domain, so the keytab contains both upper and lower case entries. I've tried recreating the issue in my lab, however even after running net ads changetrustpw my keytab looks "ok". As in I have both upper and lower case entries for the new kvno in keytab.
However I'm getting the sense from you that the upper case entries might not even be desired? On machines where I've joined using realm I get only lower case entries, which works great for all my stuff. But I don't know if there's something else somewhere in the environment that needs the upper case entries. Regardless, at some point we'll need a dedicated config in salt for our machines anyway, as we require additional config in sssd.conf to enable the smart card authentication. But I'd like to understand why this is happening in prod. If for no other reason than once this solution hits prod from our perspective, I have a feeling that this stuff is a bit finicky and the more I know the easier it will be to troubleshoot once it breaks.
Since I know, if there are issues the Linux guys will look at our stuff and go "Oh, that's that annoying solution. I don't know anything about that."
sudonem@reddit
Hrm
I can’t help with anything saltstack related since I live & breath Ansible most days.
Generally though, using
net ads joinis considered to be a very outdated approach. It technical works as the tool hasn’t been fully deprecated, but best practice is to userealm joinor ‘adcli` instead.In either case the keytab file should be generated & updated automatically.
The thing is, in all case, the domain in the keytab file should indeed be uppercase as a requirement for Kerberos (as well as just common convention).
I have never implemented smart card authentication - but I’d really revisit that documentation, or have a discussion with the vendor because I am almost certain that these lowercase entries are going to be a problem with regards to a Linux system staying authenticated on the domain.
Unexpected_Cranberry@reddit (OP)
So, I've done some more experimentation, and I've managed to recreate the issue in my lab after getting some more details from our Linux team.
Apparently the setup we have in prod is this.
nets ads join is used to join Active Directory. This works, on join the keytab contains entries with both upper and lower case host/ components. As far as I can tell from the man page, this is the expected behavior for compatibility reasons.
However, after that sssd and krb5 is used to handle authentication and computer password rotations. If I rotate the computer password using nets ads changetrustpw it works, entries in the keytab will again contain both upper and lower case entries for the new kvno.
But, if I let sssd handle the rotation automatically, the keytab will only get the upper case HOST/ records after the rotation, and kerberos is broken.
If it was up to me, we'd move away from net ads and go to realm or adcli instead. However, at least so far in my testing, that will result in the keytab containing only lower case host/ records. And I'm not familiar enough with what other things we have running that rely on kerberos, and if they expect or require the upper case records to exist. That might well be the case as it's a decades old environment.
So now I'm experimenting with adjustments to the sssd-configuration to see if I can get it to work with a small adjustment there. It's slow going though, as I've only been able to recreate the issue when sssd rotates the password off of the ad_maximum_machine_account_password_age setting, and as far as I can tell the lowest value I can provide there is one day, and I haven't figured out how to trigger a computer password rotation in a way that reproduces the issue. So at the moment all I can do is make a change to sssd.conf and then wait two days for the password to be rotated. Which is... less than ideal.
sudonem@reddit
Right. Thats… terrible.
The obvious solution is to move away from net join - it hasn’t been a recommended / best practice solution in many years at this point.
Having said that, I think you’re shooting yourself in the foot by trying to interact with sssd.
Joining a domain with net ads join relies on samba and winbind - not sssd.
I couldn’t tell you the last time I did any domain joins using net ads join - but IIRC you want to put the machine password parameters in the smb.conf, not sssd.conf. (Seriously check my work on that though because it’s been ages)
But ultimately your treating symptoms not working on a cure. 😑
Unexpected_Cranberry@reddit (OP)
I've managed to recreate the issue now. Which is nice, because I can confirm it breaks kerberos not only for our stuff.
I am a bit confused though. You are saying you typically standardize on upper case? If we are talking about the same thing, the default as far as I can tell is lower case, and if the keytab contains upper case only, gssapi fails pre-auth for kerberos because it's expecting lower case. Everything I read states lower case is recommended.
To clarify, the spn in AD is as upper case is the default there, like so HOST/COMP@ad.domain.com.
The entry in keytab is host/COMP@AD.DOMAIN.COM.
This works because most things on the Linux side are case sensitive and seem to expect lower case host/ and uppercase REALM. And as far as AD is concerned host/ and HOST/ is the same thing, as is @realm and @REALM.
Now I'm curious how you've managed to standardize on HOST/COMP@REALM. I don't think that's what I would opt for. I can get adcli to create it in the keytab by specifying - - service-name=HOST, but if the lower case entries aren't there pre auth for kerberos fails, so there are probably more config files in not aware of for gssapi and spnego that would need to be adjusted to get uppercase to work. If you'd care to share at least which man pages or config files I should look at I'd appreciate it.
Unexpected_Cranberry@reddit (OP)
Yes, the realm part should be upper case. But that's not the part causing trouble for us. It's the host/ part, as in host/COMPUTER@REALM vs HOST/COMPUTER@REALM
Unexpected_Cranberry@reddit (OP)
This is automated already using saltstack. Right now we're trying to figure out what we need to put in the files for this to work consistently, as the current config isn't working right.
hortimech@reddit
Active Directory started out as Windows thing and Windows is case insensitive, so your two example SPNs are duplicates. I am fairly sure that some work has gone into Samba to stop such duplicates, but cannot easily track the changes. A newly joined Samba machine in my domain only has uppercase 'HOST' SPNs.
There is also the problem that if you are joining using 'net ads join' then you are probably also running Samba. If this the case, then, in my opinion, you shouldn't be using sssd, it isn't required with Samba and can in fact cause problems.