This exploit is for a combination of fail2ban and `mail`. Reading how it works, it seems much more of a vulnerability for anything using the `mail` command than in fail2ban per se.
That ~! escape is really dangerous. What percentage of sysadmins are even aware of its existence? I can see how it can be useful, but there is a lot of potential for exploit if you aren't extremely careful.
The `mail` command shouldn't so easily accommodate executing arbitrary commands from input. The ~! escape should probably be either removed from `mail` entirely, or enabled only if you pass it a flag. It seems like a vestige from an earlier, more innocent time.
This isn't to absolve sysadmins who fail to sanitize their inputs, but let's not make their job so difficult.
This is a typical old world vs new world vulnerability. Old tools have lots of hacker features that are often not a good idea anymore in this (century's) world. See also cryptographic agility and such.
The Unix philosophy of combining small, specialised tools into a larger whole to accomplish a task is an important concept that really shouldn't be forgotten, but at the same time one should remember not all tools are meant to withstand malicious input from a hostile internet, in particular those that predate the explosion of the internet.
The mail command has been around for a while, and was intended mainly for human use, not scripts.
For example, just run mail alone with no arguments. It'll display the messages in your local mail box, and prompt you which ones to read, keep, delete, reply to, etc.
The mail command is an MUA, just like mutt or Thunderbird, but much older.
For extra fun, there are (or at least were) multiple implementations of the mail command. The arguments were similar enough, but an old (and replaced) system at a previous employer required Heirloom mail/snail, not BSD because it actually intentionally used escape sequences like this.
In particular, it used one to add attachments (by giving the path).
We replaced it with Perl, getting rid of the shell script entirely (the whole stack was Perl).
Shell scripts really ought to use the sendmail command to send mail, but then you have to remember those obscure options to pass and generate the mail headers yourself, so it's understandable why no one does. (And probably handle dot-doubling).
How popular is this feature to send emails from fail2ban? I haven't heard about it until now, and I think I'd have a very noisy inbox if my fail2ban setup sent me mails about every blocked IP.
Feeding the fail2ban log to Papertrail or another logging service should give enough visibility.
I've seen fail2ban block the IPv4 of a company's headquarters because a new member of staff setup their SSH config wrong and made too many attempts to connect to a production system. In this sort of situation, having federated logs on external infrastructure is a real saviour.
Serious question: What do you hope to achieve with all that?
Fail2Ban has caused more problems for me than it has ever solved, and non-standard ports is confusing if you're a large team. Personally I feel like you're better of just requiring ssh keys, don't install fail2ban and stay on port 22.
> non-standard ports is confusing if you're a large team
I think this is the key part. For me, this is for very small teams or for my personal server, which I need to be able to access with a password from possibly anywhere.
For a larger team, I think I agree - emails would be too low of a signal-to-noise ratio. Although I would still use something like fail2ban, but use it just to log to somewhere more appropriate just in case I need it.
Since 0.8.1 Fail2ban uses Sendmail MTA for mailing so this is worrying only if you purposefully reverted to mail or upgraded from an older version. Recent deployments SEEM like they'd be fine out of the box.
The patches just update the action files to add escaping if you happen to be using mail.
Gentle reminder that you do not need fail2ban to block repeat connection attempts. Two IPTables lines will block any IPs connecting over a given rate in a given time. A third line logs it.
For pure ssh, this is true, but note that fail2ban can also automatically unblock after a certain amount of time, which is nice when valid users shut themselves out by mistake over the weekend... For other services, depending on their type, the false-positive rate will often be way too high.
FWIW this can also be done with ipsets. [1] ipset hash tables are quite useful if you are blocking a large number of hosts. The CPU load is much lower than having individual rules for each host blocked. You can set any time limit on a ipset or have no time limit. You can also dynamically insert ip's into ipsets using iptables rules.
You need fail2ban to block repeat failed login attempts. If you just use iptables, you can't distinguish between a failed SSH login and a short-lived successful one, and there's a lot of legitimate reasons you might have several short-lived successful ones in a row.
This is why we enable port knocking (SPA) on all of our Internet-facing systems: it protects against zero-days, keeps the logs pristine, and stops repeat connection attempts. As a matter fact, any failed login of any kind creates a security alert.
it would have to generate an e-mail by default, which means you need mail setup.
on top of this you need to MITM the whois command (which is unencrypted, but still). This is not easily exploitable.
I'm going to guess "noone". This is not the first security hole like this caused by piping to mail. See CVE-2000-0703, a trivial local root via suidperl. Unfortunately backwards compatability often wins over prevention of future security holes.
Interesting. This doesn't sound like that much of a vulnerability itself, but it does help clarify my thinking about SSH security.
Basically, SSH with proper configuration banning password auth is just fine and okay to be exposed to the internet. Extra logs from some failed attempts aren't really a big deal. If you want to make access more secure for it, that's okay, but I'd resist using complex on-server software for that which is likely to be less battle-tested and expose more attack surface. If you must do so, do things that are simple and/or off-server, like run on an alternate port or block network access for control ports at the firewall or security group level from any IP range but the ones you expect to be connecting from.
Looking at the patches, it seems running 'grep -rnw "mail -s" <fail2ban installation folder>, and replacing all found with "mail -E 'set escape' -s" is all that is needed.
Well the "-s" is denoting the subject but yes "-E 'set escape'" is the mitigation fail2ban are employing though it should be noted that this is specific to Mailutils. Many (most?) other mail(1)'s use the -E option to mean don't send messages with an empty body which left me very confused when I first read your comment so had to do some digging to understand it.
We can't ever prove that something is "secure", we can only take precautions. Is your SSH server secure? You can't tell, by definition, in a world where 0 day exploits exist.
If you know where your requests are coming from, you can restrict to those IP addresses. It doesn't mean you think your server is insecure - but it's making the lives of people trying to get to it that much harder. If you don't know, then maybe fail2ban is a worthwhile alternative. It doesn't even necessarily mean you are trying to prevent brute forcing (password logins should always be disabled anyway) - but some exploits require multiple attempts.
In the past I was against measures like changing the default port (although it should still be <1024). But these measures are helpful the moment some new exploit is in the wild that could twart naive scanning scripts. Same thing goes for hiding the banner.
The best thing is still to automate things so that SSH access becomes a 'break the glass' moment. If you are always logging in via SSH(and it's a server, not a workstation that you work on), it means that there's something missing in the automation (or observability) that needs to be fixed.
It's one of the many things designed to make lousy passwords become actually ok-ish.
Specifically for ssh, I've also had my vps eating a surprising amount of cpu just to verify and reject bad logins when some scanner or other spent a couple days on it before moving on.
You can also just use a firewall rule to rate limit syn packets from a source. It won't affect legit traffic, and doesn't require running a userland process to watch the logs.
You can have it block all ports for the attacking host, so it might help other protocols. Aside from just reducing load on the host. By dropping the TCP SYN packet, you also introduce delays for the attacker.
The main benefit for me is protecting password services like HTTP basic auth, ftp, mailserver, etc. It’s pretty pointless for SSH when keys are involved
Is there a flag to disable tilde escapes, or a similar command that doesn't have this feature? The mail command is a really convenient way of sending mail in scripts.
My thinking is that good security configuration makes fail2ban redundant. With password authentication disabled and strong keys, it’s not clear to me what the threat is that fail2ban offers to protect against.
I use it mostly because it simply declutters the logfiles, and it's super easy to set up and has practically no maintenance, so why not?
EDIT: as others have noted here, fail2ban can do much more than just ssh. I also use it for Exim to block all these open-relay-scanners which are polluting the logs.
fail2ban is not only for SSH, but also for HTTP. You have a php website with apache and logs enabled? Fail2ban can ban people if they try to brute force your login page. It's actually quite powerful, but I see it in use less and less.
This. I host a very simple website at home on my Raspberry Pi; maybe 20 regular users. But, I receive a ton of traffic trying to login to PHP admin, nginx scripts, ssh brute force attempts, and on and on... I use fail2ban to ban individual IPs and if I see a lot coming from a particular CIDR range, I'll just block that whole range.
There are a ton of example jails out on GitHub and elsewhere that are easily dropped into your configs.
I mostly see it on smaller systems configured by people or teams who doesn't really manage servers professionally.
Mostly we just don't allow SSH via the internet, you have to be on the office network or on a VPN and AWS instances can be accessed using Amazon Systems Manager. For those few systems that absolutely most be accessible via SSH on the internet: SSH keys and/or multi-factor authentication is required.
This is why we enable port knocking (SPA) on all of our Internet-facing systems: it protects against zero-days and keeps the logs pristine. As a matter fact, any failed login of any kind creates a security alert.
I use it for ssh, ftp, and dovecot. Even with ssh passwords disabled, fail2ban reduces traffic a lot on some servers (since culprits get null-routed) which is always good.
That ~! escape is really dangerous. What percentage of sysadmins are even aware of its existence? I can see how it can be useful, but there is a lot of potential for exploit if you aren't extremely careful.
The `mail` command shouldn't so easily accommodate executing arbitrary commands from input. The ~! escape should probably be either removed from `mail` entirely, or enabled only if you pass it a flag. It seems like a vestige from an earlier, more innocent time.
This isn't to absolve sysadmins who fail to sanitize their inputs, but let's not make their job so difficult.