Dont tie your Password Manager to SSO
Posted by sysacc@reddit | sysadmin | View on Reddit | 108 comments
I recently did a table top DR exercise with a client. The goal of the event was to see what could operate during a SSO outage and for how long.
The first thing that was caught was that the mandated password manager was SSO only and only 2 people had non SSO accounts. Those two saved their non-SSO accounts in said password manager.
I may still have a bump on my head from my head hitting the desk...
gnopgnip@reddit
You should have SSO on a password manager
You should all have a break glass account(s) for emergencies
mtgguy999@reddit
It sounds like they did but the break glass account was stored in the password manager that was only accessible via sso
siedenburg2@reddit
Break glass accounts should be physical (in a vault with a seal or behind such a fire emergency glass thing), not digital.
sssRealm@reddit
We have a Yubikey zip tied to the inside of a server rack for break glass. Director has physical key to server room, if badge access goes down. What could go wrong? I guess a few things.
siedenburg2@reddit
What do you plan if your server room burns down or "if a plane crashes into it"? You now (should) have backups stored in a different location but no access to the systems and accounts.
Also what happens if the director while driving to the server room gets into a car crash, who can unlock the room now?
sssRealm@reddit
It's the 3rd admin account and we have 3-2-1 (offsite and cloud) In such an extreme disaster, would I even have a job to come back to?
what_dat_ninja@reddit
But the vault code is in the password manager!
fresh-dork@reddit
add a process where each of three keyholders is required to open the vault under supervision monthly. solved
Igot1forya@reddit
And the account can only be accessed if one of them is sacrificed... Blood for the blood gods!
walking_on_a_wire@reddit
Blood for the blood god
ImpossiblePudding@reddit
https://i.redd.it/awcy3p3k82vg1.gif
what_dat_ninja@reddit
By God, have you even thought about your bus factor?! Smh
fresh-dork@reddit
3 people minimum on DR with monthly review. do we need 5?
TheFluffiestRedditor@reddit
You get a dependency! You get a dependency! Everyone gets a dependency!
It’s dependencies all the way ~~down~~ round.
ne1c4n@reddit
Chefs kiss, this is my workplace. Lol
Hangikjot@reddit
The company Accountant, CFO or HR head tend to have a small safe. That's were we store the printed copy in a sealed envelope.
Fuzzmiester@reddit
Just remember to have some fish for the seal.
sofixa11@reddit
Or digital but in individual password managers, or printed pieces of paper stored in wallets.
siedenburg2@reddit
Digital only as an additional location (but with rigid checks on the account to log every use), never put your emergency access only in a digital location or you probably end with nothing in an emergency.
Also I'm not a huge fan of giving the emergency root access to someone that has to keep it to themself (except if they own a safe and put it there), the responsibility is high for a single person and there is the risk of losing the data together with identifiable material.
WantDebianThanks@reddit
Director at a previous place created a break glass account, put the username and password on a piece of paper, handed it to the owner, explained what it was, and asked him to put it in the company's box at a bank.
Six months later director followed up with owner just to make sure the owner remembered he had this account. The owner did not.
I don't know if the owner actually put the creds in the company's box or not
Matazat@reddit
100% chance that piece of paper is sitting on the owner's desk right now
WantDebianThanks@reddit
This was in 2019, so I would be surprised.
kevvie13@reddit
I think the best practice was to use break glass account with another sso provider than the standard ones.
goingslowfast@reddit
That’s great for IT, but how does Joe in accounting log in to his online accounting tool if he can’t get into his password manager?
gnopgnip@reddit
You use the break glass account to fix sso??
goingslowfast@reddit
I think the scenario he is talking about is what happens if say Entra is down for an extended period.
gnopgnip@reddit
Doesn’t matter if a password manager is on a separate login. Can’t login to email or anything else with SSO is entra is down. And the same is going to be true for many of the other businesses if their is an entra or azure outage.
abr2195@reddit
I feel like if Entra is down for an extended period of time, the world is probably ending - regaining access to SSO-backed accounts is the least of my concerns.
goingslowfast@reddit
That is also a fair point. This may be a risk you choose not to control for because of that.
bfodder@reddit
I guess we should also stop using computers in case they ever go down.
Hollow3ddd@reddit
It’s easy to Forget or accidentally to apply CA policies over it in the crazy day by day.
Need a quarterly test of pulling the creds from a safe and logging in
ISeeDeadPackets@reddit
Yep. I actually had to use the break glass when we missed the SSO cert expiring. It was super convenient vs having to contact their support and spend an untold number of hours trying to get access back.
magataga@reddit
To protect our company we're enforcing arbitray password complexity with a password manager! (This is good). For ease of administration we require SSO for access to the password manager! (This is minimally good better than nothing, but in the event of a breach is a compromise once, compromise many which is bad).
Aggressive_Ear2395@reddit
yeah, break glass accounts with their physical tokens/info and a regularly backed up copy your PW database of things you might need in physically secured spots that only get opened for emergecy/update.
CharcoalGreyWolf@reddit
This. We have break glass accounts with master passwords and the appropriate privileges. Other accounts should all be SSO.
Far-Bug8297@reddit
imagine needing sso to access the passwords for when sso goes down
maxstux11@reddit
We got rid of our password manager. After rolling out a SAMLless SSO - everything is behind Entra SSO.
My end-users are much more likely to get phished then Entra is to go down (even with Microsoft's relentless desire to ruin their products)
abr2195@reddit
This is great for applications that use SAML, what about all the accounts they have that don’t support this?
maxstux11@reddit
A SAMLless SSO IS for all the applications that don't support SAML or charge an arm and a leg for it - there are a bunch (Aglide, Cerby, etc.)
abr2195@reddit
I misread your post. Sorry for the confusion and thanks for the response!
AbsoluteProbability@reddit
So.. Where do you put your other secret info? Keys, client secrets, cert passwords, api keys..
Password manager is not only username password storage?
And there are of course applications that can't, won't or just refuse to go sso..
maxstux11@reddit
Everything that isn't a username and a password goes into HashiCorp.
The purpose of the SAMLless SSO is for all the apps that don't support / charge too much for SSO. We're running Aglide; but we also looked at Cerby as an alternative
stingray75ma@reddit
.... Sorry..... That was the best laugh I have had for a long time...
I had to read it 5 times... in disbelief.... Every time... I had a great laugh 😂😂
Thank you and your customer!!
Sorry 😔
man__i__love__frogs@reddit
Sounds like you took the wrong lesson from this experience lol.
EViLTeW@reddit
I think you took the wrong lesson out of this experience.
The real lesson is: Do tabletop exercises.
SSO for the password manager wasn't a failure. Not having a clear DR/BC plan in place for when SSO is unavailable was the failure. It was found by doing a tabletop. Deficiency identified, deficiency corrected. Next time, you'll have new deficiencies to bump into.
sysacc@reddit (OP)
I probably could of written down the title of this post better.
There's a series of things that lead to this:
The teams who manages the password manager is inexperienced
Policies do not allow to write down passwords
They removed all other password managers
EViLTeW@reddit
I get all of that, but it's really irrelevant to your post.
Your post is saying don't use SSO. That is not good advice.
The good advice is to do tabletop exercises. That way you learn about your issues (which in this case is the lack of DR/BC plan for the password manager) when the stakes are zero instead of when things are on fire.
aquila421@reddit
We use Keeper. Keeper forces break-glass for this exact reason.
talin77@reddit
Or! Sync your passwords once a month to a other password manager!
quantumhardline@reddit
SSO should be used for password manager, the admin of the tenant can have a backup account and hardware key. SSO outages are rare, the benefits far outweigh risk of using non SSO example enforcing CA policies etc.
orion3311@reddit
Admin accounts have a breakglass code, and technically you could go in and disable SSO if need be.
sysacc@reddit (OP)
Want to guess where those codes were stored?
Mindless_Consumer@reddit
So, keep sso on your password manager, but keep your break glass password in locked cabinet.
Yeseylon@reddit
Bonus point if it's a glass cabinet and you break the glass when you need it.
Hollow3ddd@reddit
Reminds me of the codes they crack open in War Games movie. Idk why
NaturalIdiocy@reddit
Wait wait wait... where do I store the baby and where do I store the bathwater?
Mindless_Consumer@reddit
Server room i guess
Reedy_Whisper_45@reddit
brrr.
SquashNo7817@reddit
At some point you need to use common sense...
sysacc@reddit (OP)
https://en.wikipedia.org/wiki/Wikipedia:Common_sense_is_not_common
theunquenchedservant@reddit
correct, that's why you have DR exercises.
orion3311@reddit
I thought thats the actual job description.
MrShlash@reddit
In a physical safe like any sane org?
goingslowfast@reddit
Even if you disable SSO, how many of your users have non SSO creds setup within the password manager?
SaintEyegor@reddit
We just had a major outage caused by an EPO test that wasn’t supposed to trip the EPO and it took two days to fully recover. Management finally realized that having critical infrastructure hosted wholly in VMs wasn’t too bright and not having break glass accounts was monumentally stupid. I earned a buttload of comp time and had a healthy case of schadenfreude.
davy_crockett_slayer@reddit
It’s fine. You just need a break glass account, and store the creds elsewhere.
Tessian@reddit
Not only this, but most implementations of SSO for password managers are very insecure.
If the password manager, like LastPass (don't know if this is still the case but it was a few years ago), just lets you set up SSO and it works without a master password - you now gave the vendor all your vault encryption keys. They're holding them somewhere and handing them out to the client when SSO succeeds, but most companies don't realize the control and security they just gave up.
Alternatively Bitwarden supports SSO options that don't compromise security, either by still requiring a master password or by forcing you to build a server that does the key escrow.
That all being said, yes this is another reason I don't recommend SSO on password managers. Storing DR documentation is a nice benefit and you need to be able to access it in an emergency.
abr2195@reddit
LastPass splits the encryption keys. One half is held in your LastPass vault while the other half is held by your IdP.
I’m sure you were right in the past, but this does not appear to be the case now.
Maverick0984@reddit
That's just LastPass doing LastPass things. 1Password doesn't behave this way either.
FatBook-Air@reddit
Will 1Password let you SSO? They didn't in the past.
Maverick0984@reddit
We've been on it for a few years now.
Specialist_Guard_330@reddit
Yes they do.
FatBook-Air@reddit
They had SCIM before they had SSO. There was a time when they didn't support SSO because they thought it was inherently not secure. Guess they changed their minds. They did support SCIM itself just for convenience (but it didn't support SSO, which always felt weird).
Specialist_Guard_330@reddit
They did not have hosted SCIM you needed to setup your own SCIM bridge separately.
Specialist_Guard_330@reddit
1password the goat???
HKChad@reddit
I’m just going to pile on. Everything sso, you learned the wrong lesson.
bfodder@reddit
Your password manager should definitely use saml auth with your IDP, but you also need a "break glass" account that can be used in an emergency like this to get things back up and running.
HeyLuke@reddit
OK question: how are orgs doing this with for instance Bitwarden? Because when I enabled SSO for Bitwarden, I still had to use the master password to log in. I've read it's because the data in your vault is encrypted with your master password and that makes total sense. But it seems to me that it's a lot to ask from users to do two login methods to gain access to their password manager every day.
Klynn7@reddit
We use Trusted Devices. Once you enable it, users can sign in with SSO, and the first time they do on a new device the admin has to approve it. Going forward they can just SSO, no Master Password required.
The caveat is this makes it possible for an Admin who controls the IdP and Bitwarden to get access to any users’ vaults. In our org that’s fine but we make sure to warn users not to store personal passwords in the work password manager.
FarmboyJustice@reddit
Bitwarden is actually pretty flexible. It makes a distinction between signing into the platform and unlocking the platform. So you can use SSO to sign into your Bitwarden account, but then use your master password to unlock your vault while you are signed in.
You can also configure it to allow other unlock methods, like using a PIN or hardware token but that's up to whoever administers the platform. You also control how long users are able to stay logged in for, you can make them re-authenticate frequently or rarely. Or you can do that in your SSO platform instead, or you can mix both together to create whatever level of annoyance you want.
bfodder@reddit
I can't really speak to how it works for Bitwarden.
BlackV@reddit
SSO the the vault is fine
SSO being the only access is not
Main_Ambassador_4985@reddit
We store the BreakGlass to the password manager in a non-SSO account.
DueBreadfruit2638@reddit
Bitwarden can be configured to force admins to use a master password to login while non-admins are forced to use SSO. This works well.
ansibleloop@reddit
1Password tell you this in big bold text when you're setting it up
Which is why the people who have admin accounts in 1Password can't have SSO - they use KeePass for their break-glass creds
RCTID1975@reddit
Breakglass accounts shouldn't be tied to people. They should never be used unless in a DR situation
RCTID1975@reddit
Disagree. Everything should be SSO.
Critical services like this should have a break glass stored with the DR plan.
Sigma186@reddit
The break glass accounts are in a binder in the safe with a seal.
Secret_Account07@reddit
I’ve brought this up soo many times
All of our admin accounts for our on prem infra users SSO. We legit have no other way in.
It’s gone down several times and we can’t do shit. No local option
SpadeGrenade@reddit
Uh.. no.
agingnerds@reddit
In m limited experience with SSO admin or owner accounts are typically excluded from SSO. In our environment if we cant access 1password at all because of microsoft we have other issues besides passwords. So It works.
CountGeoffrey@reddit
disagree 100%
FartInTheLocker@reddit
Surely it's always better to PW managers tied with SSO so you can have some level of protection in place, assuming CA's with Entra etc, that's better than just not having SSO there.
Probs just always need to have some level of break-glass access into the platform, same as Entra etc.
attathomeguy@reddit
If your SSO is down don't you have bigger problems? Every environment I have worked in has had a admin account that is not SSO locked in a physical safe for just such an emergency. However if everything else is tied to SSO and you can't access SSO then you can't get into your email to email the vendor's support teams to disable SSO and then you have to setup SSO again when it is back up. It sounds like you need to use a better vendor.
sysacc@reddit (OP)
Thankfully these guys can continue to operate their machinery without SSO or most of their systems.
attathomeguy@reddit
Yeah that's great but how do orders get from customer to the machinery? I would imagine all of those systems are on SSO.
Frothyleet@reddit
Would you? I have never seen OT systems tied to SSO and I can't imagine that being a great configuration even if it was possible.
attathomeguy@reddit
I have seen some weird shit in my time doing SSO deployments. I would never assume it was setup correctly
bfodder@reddit
Yeah this is the important part that gets ignored.
Centimane@reddit
Were the 2 non-SSO accounts intentional? If so could chalk that up to they planned for this scenario.
zockie@reddit
And now we introduce you to the industry standard of break glass accounts. Think outside the box!
Unable-Entrance3110@reddit
I actually don't even know my Bitwarden password. I keep that in KeePass.
Yeah, it gets annoying sometimes.
WWGHIAFTC@reddit
I keep my keepassd DB password on a bitlocker thubmdrive. I keep the bitlocker recover codes on an encrypted S3 bucket so it's easily available. I store the S3 keys on ... nevermind.
Total_Job29@reddit
I once was in a real life incident where thieves stole the ‘copper’ cables which were actually fibre cables. This took down a vast area no internet for probably 10 square miles.
Guess where the DR plans were? On the network drive. That network drive was hosted at a remote site.
Queue someone driving 100m round trip to get the plans. Well we sent four people so 1 person could drive and the others could act as a small DR team in the car journey back, phoning other teams and running the plans.
It was hilarious when you look back. The CEOs face when he said ‘so they are on the network we no longer have access to’ and then the silence for 10 seconds while the head of IT and Security tried to think of how to say yes.
Odd_Secret9132@reddit
A 'crash bag' stored in a safe secure location away from the primary site is a good idea. Hard and digital copies of DR plans, key contacts, maybe even copies of important software. Check and update it at least quarterly.
joners02@reddit
I like that fact that the extensive DR plans that we have are all in a project management tool that is locked behind SSO. I have asked, "what happens when sso breaks or that tool is down?" they all have a blank look on their face. I know that there are break glass accounts, but the instructions on how to disable that are all in the same PM tool.
_litz@reddit
That's about as good as the customer who didn't know their VEEAM encryption password when it came time for the DR test ...
(and then, two weeks after we fixed that and they successfully DR tested, they got hacked and their whole environment destroyed ... had they not tested, and discovered the "issue" ... zero recovery ...)
the_doughboy@reddit
It’s in Bitwarden. SSO better be up….