Question on security finding
Posted by Pretend-Weird26@reddit | linuxadmin | View on Reddit | 34 comments
Looking for input on a security question. First thing is I work for a bank and this bank is not one of the top 10, but it is one that has crossed the magic too big to fail line. Our Information security had an audit done, this is just Tuesday, no big deal. These jerks came back with a finding that bash_history had passwords in it. Ok, yeah, mea culpa. It happens during some installs the default password is on the command line, again not a huge deal. The team cleaned it up and did some "set +o history" training. Good? Not even close. Some Windows 2003 MCSE who went into security wants bash_history entirely disabled. It cannot be made so that password CANNOT be "stored in it" so it needs to go. He is serious. He cannot be ignored or made to go away. The audit finding has been put into an immutable table that the Federal Regulators (OCC, FDIC ... ) have reviewed. This must be addressed as it stands. Soft arguments like "so, no text documents", have failed. He means it needs to go. I need a counter argument other than "I need this tool" to use.
Ok, has anyone else hit this? How did you solve it?
A scan tool that can be purchased is an option. What one? Other regulated industries, have you seen this? what was the fix? Is this a thing at DoD?
I don't want to give up bash history! I don't. Especially over something this dumb.
gordonmessmer@reddit
Definitely give up bash_history.
Shell history is only relevant if you are using systems interactively, and I really strongly encourage everyone to simply stop doing that. Systems that allow interactive use are significantly less secure and less reliable. Interactive use should be an emergency-only, break-glass option. For everything else, you should be committing everything that needs to be done to a playbook in a source-code repository, with all commits reviewed and approved by a second individual, run through a system that records results, and run in a test env before they are run in production.
The benefits to auditing, security, and reliability are enormous.
TheLinuxMailman@reddit
This is a good base approach.
An entry can be omitted from bash history by starting the line with a space.
If one forgets, one can also use history -d (preceeded by a space of course) to delete the undesired entry.
This has worked for me.
Pretend-Weird26@reddit (OP)
but on a team with 10 or so members servicing 900 or more servers how do tell what another team member or non-team member did? hos do you diagnose problems that the DBA cause but does not remember what they did?
Pretend-Weird26@reddit (OP)
Audit I see, but the others are a hard sell.
captkirkseviltwin@reddit
What is the wording of the specific security control the auditor is referencing? Do you have the specific code (PCI-DSS? NIST? CIS? Etc.)
It may be he’s misread the control finding and fix text.
gordonmessmer@reddit
Security: If you restrict interactive use (e.g.: store an interactive use password in Vault or some other secure location for emergency use and don't give admins the right to log in without requesting the emergency password), then an attacker has a much harder time stealing useful credentials from your administrative staff and accessing your critical production systems.
Reliability: If you run every playbook in a test environment that is similar to production before you run it in production, you're much less likely to run a command in production that unexpectedly breaks the system. If you're committing these actions to playbooks, you're much less likely to run something other than what you meant to run.
These are not hard sells.
Pretend-Weird26@reddit (OP)
We are starting from zero and have to rewrite procedures and get them accepted by audit. conceptually easy, but a lot of practical problems. DR testing is currently on going. How do you handle fail over to secondary site? We are hot/warm, but the VM's still need to be spun up and tested without impacting operations.
UsedToLikeThisStuff@reddit
Stream your audit logs to a central server. They include sudo commands. Don’t give anyone root’s password or allow ssh in as root.
bearwhiz@reddit
That's why you have a security-rated keylogger that monitors all usage of elevated rights, and why the root password is locked up in a vaulting system that changes it every time someone accesses it and logs the usage. Have a look at software like PowerBroker, which replaces sudo and not only logs what your admins do with their root privileges, and not only limits what commands they can run, but will boot them out of a superuser session if they start typing the wrong command, options, or arguments.
gordonmessmer@reddit
In all cases: It was committed to a source code repo. You can look at the playbook in the source code repo.
It was recorded by the system that ran the playbook (e.g. Jenkins or Ansible AWX (aka Tower)). You can look at that system's records of actions that have been run.
These options are much more likely to have the information you're looking for than shell history, which can be deleted intentionally or accidentally.
Pretend-Weird26@reddit (OP)
This I get. Seems like it would take a rearchitecting of the environment. Our Tower playbooks, around mandatory patching, once a month, all systems is not 100% successful. We usually have 5% or so failure, 20 - 30 systems that need some hand holding. Oracle spacewalk is the worst. Failures would need a session, yes?
DarrenRainey@reddit
Agree on disabling bash_history - Commands can be saved on the client side / SSH client but having that history on the server could be probalmatic if that server was compromised someone could use saved credentials to access other systems or elevate themselves.
Ansible playbooks are good for repeatablity, In terms of helping other staff you should write your own guides / knowledge base rather than telling them to use saved commands from bash history.
In terms of command auditing there are tools like auditd can be configured to log command usage for all users in case you need to check if someone has done something they haven't had and logs can be restricted so only certian users have access.
Pretend-Weird26@reddit (OP)
Does this pass the regulators? do you have a resource to look at? I can see this and it is growing on me, but man that is a steep hill
vogelke@reddit
I've worked in TS/SCI environments with Linux and Solaris, and this has never been a thing. However, those environments were entirely self-contained and allowed no external network connections at all. If you wanted to use the system, you went into the vault. Anything you brought in, like a notebook or an installation DVD, stayed in.
All I can think of is:
suggest all filesystems have encryption turned on, and
the next time you need to install something that requires a passsword on the command line, tell your boss that you are at a work stoppage and CANNOT do the installation due to security concerns.
Pretend_Adeptness781@reddit
You could disable bash history but pipe your shell connection to a log file, in effect storing atleast your own history.
kyleh0@reddit
I've been in IT for over 30 years, and early on I decided o banks, no lawyers, no weapon makers, no healthcare. There are so many jobs where the stakes are so much lower for basically the same pay and job security. Some jobs that are high risk make some kind of sense, but I get no thrills from constantly walking under the kind of audit pressure of a few key industries. Heh
Pretend_Adeptness781@reddit
You could disable bash history but pipe your shell connection to a log file, in effect storing atleast your own history.
HTX-713@reddit
Give up on the bash_history, thats what audit logs are for.
Longjumping_Gap_9325@reddit
Just start typing every command with a leading " " space before it? Then it won't go into bash history. Ugly hack win!
doubled112@reddit
I use this all the time. I don't consider it an ugly hack at all, and always assumed this was the reason the feature existed.
Pretend-Weird26@reddit (OP)
yeah, his thing is "it is still possible to store a password" has made this win/lose. I actually had to explain that the bash_history did not store ALL passwords just one typed as part of a command, like if I jump to another server and am prompted for a password. That is never stored. took 15 min of my life to explain that.
I am looking any third options. Thanks, but this is kind of like the set +o history
Longjumping_Gap_9325@reddit
Yeah, I was just being a smart@$$ haha. I feel your pain though, the asks sometimes are a bit fun
mikeone33@reddit
Do what I did. Transition from Linux Engineering to Security.
NL_Gray-Fox@reddit
First of all the size of the bank doesn't matter that much, probably smaller banks have better security than big ones because security at banks is mostly abysmal anyway.
I've had countless audits and he has ¼ point but he's missing the mark.
Yes passwords should not be in there but you need the history for audit purposes. At my previous company we had them all pushed to elastic search, which was then checked for irregularities manually (if you properly manage your servers there shouldn't be that many interactive logins to begin with).
If a password was found we created a ticket and and the password was changed immediately (using LDAP+Kerberos) so access was lost to the account (also a check was done if the account was nog logged in).
maxlan@reddit
OK. Uninstall bash. Simple.
Move everyone over to zsh.
Or set all shells to /bin/sh with no history. And if you want history you can run bash every time you login.
Whatever you do, it isn't going to stop someone doing "echo password > file". It is usually only the default behaviour they care about. If the default is no history: problem solved.
PS I once "hacked" into a prod server as root because my contact left for the weekend. I looked through the "last" log and found a failed login by dhf5&_u9(: followed by a successful root login a few seconds later.
Try that and then tell them youre going to turn off failed login reporting too, to avoid passwords being saved accidentally in the last log.
Do some curl requests with the password in the URL instead of the auth header and say you need to turn off http request logging. To avoid passwords being stored in http access logs.
Think of a few more places that will totally mess with their "security" brain. See if you can push them into early retirement from having a mental breakdown ;-)
Hotshot55@reddit
lastdoesn't show failed logins andlastbrequires non-standard permissions.mrkurtz@reddit
Umm. What kind of audit is this? I come from finserv, this feels weird to me. Are there controls you can demonstrate? Cuz there should be. Remediation is a thing, controls are a thing, and exceptions are a thing.
Who does this person report to? I’d talk to them. This feels weird.
symcbean@reddit
IMHO the WTF here is that you still seem to be typing in passwords. The bash_history thing is just one symptom of a bigger problem. It raises questions about password complexity, where you are storing the clear text, how the clear text gets from there to the command line.....it also means that the clear text is visible in the auditing tools your security team should ALREADY BE USING AND MONITORING. Maybe this is a just a misdirection attempt by the security guy to cover up what they've been missing for years.
Privileged access tools like CyberArk are one solution. I can think of lots of simpler/cheaper ones but I have no idea how to go about implementing them in a MS-Windows environment.
Rotten_Red@reddit
Is there a CIS requirement to retain shell history?
Pretend-Weird26@reddit (OP)
Have not seem anything one way or the other. That has been my first line of defense. CIS does not address it so why is it a huge risk?
bearwhiz@reddit
Your answer is contained in this statement:
You have no choice. You have to disable bash_history, or you'll fail regulatory audit. Absent a showing of why it's utterly impossible for you to do it, along with evidence of the compensatory controls you've put in place, you're not going to be allowed to keep it enabled. That's the way it goes, working IT in finance. Pick your battles; this isn't one you're going to win.
skibumatbu@reddit
If you know you can't win, then just disable it in /etc/bash_profile and move on. Make sure you document why it was done and have management sign off on the decision against your judgement.
Then, when there is an issue where you wish you had bash history it isn't on you.
As an alternative to bash history, look into jump hosts and only allow logins from those hosts. Your security person will say "ok, one host is better than 50,000 and I can live with that". Then You can then monitor the history for audit and if a password is found in the history an alert is raised for the password to be immediately rotated securely. Now, you're working with them and not against them and you've made it better in that any divulged password is rotated quickly.
Infosec isn't there to be jerks. We want the business to run, but there are definitely better ways to do things. And there may be a policy and regulations we have to enforce. Sadly the problem is in how they are communicating to you and how you are responding to them. It happens. Ask them questions on what they want and how they think it should be done. Ask them why they want it that way. And there is nothing wrong with you saying " but what about this issue?". Maybe you both will learn something.
Hotshot55@reddit
read -s my_passwordinstall_commad --password=${my_password}Problem solved.
Pretend-Weird26@reddit (OP)
this a$$'s point is that bash_history still exists. If I paste a password in as an accident, it is still in the history. I need a way to retain interactive shell, bash history and make it secure.