In today’s current environment it’s not unusual to see log files filled with thousands of entries for failed logins.  Botnets, compromised servers, and even foreign governments are directing their energies toward harvesting valid username/passwords for mail servers, SSH access, and web sites with massive dictionary and brute force attacks.  These attacks can come from several different sources at once and it’s not unheard of to see multiple machines co-ordinating a dictionary attack against a target machine.

To provide the reader with an idea of the scale, we have seen server log files for failed SSH logins grow to 27mb in size and accounted for more 40,000 login failures in a single week.  That was for a small mom and pop web site that had no eCommerce, no forum, or any other obvious attractive targets on it.  Clients with larger, more prominent services that have attracted a botnet’s full attention can see their log files grow into the hundreds of mb in no time. Imagine the number of attacks being directed against banking, social networking, and other high profile entities on the Internet.

There are several strategies for defeating this type of attack, both active and passive.  The most common of the passive counter measures is to employ very strong passwords, or move the attacked service to a different port.  These methods won’t work for a lot of businesses for several reasons.  Moving a web or email service to a non standard port increases support costs and often breaks software.  The general public notoriously picks weak passwords, and even when warned that their passwords are weak, use them anyway. (How to create a secure password)

This common problems has led to a couple of active solutions that work by monitoring log files and then executing commands to block a host based on behavior extracted from logs.  The most prominent is Fail2ban, an open source package written in Python, that can utilize several different methods to block the offending host.  The most common methods for blocking hosts in the Linux/BSD world is to use iptables or ipfw rules.

The Issues:

Fail2ban works very well for the individual server, but most large internet service deployments are comprised of several servers behind a firewall.  The problem we were faced with was not just the large number of physical machines, but the fact that each host had a number of virtual hosts that needed protection as well.   We wanted to block an attacking machine from all protected servers once it has violated the policy on any one of them.

If we fell back on just installing Fail2ban on each server, then a offending host is only blocked from one server at a time while still having access to the others.  In a large virtual environment where one is trying to keep the VM’s as free of extraneous software as possible, having iptables installed on each virtual host chews into memory requirements.  Managing the Fail2ban infrastructure across all machines becomes more complex and more costly as the number of machines increases.

The Solution:

In this case we have a Linux based firewall between the Internet and the servers.  Each machine and VM behind the firewall runs Fail2ban, but instead of creating a local rule, each host sends the IP of the bad host to the firewall.  This firewall will be used to block hosts for the entire network.  We accomplish this by using Fail2ban’s client program and some custom configuration on the protected hosts and firewall.  Below is an outline of how to centralize the blocking of offending host’s IP address using a Linux firewall, iptables, and Fail2ban.

Setting up the firewall:

Install Fail2ban on the firewall (For RHEL and clones, yum install fail2ban, for Debian apt-get install fail2ban).  Edit the /etc/fail2ban/jail.conf.  Adjust ignoreip to include IPs that should not be banned for any reason ever, this usually means management and monitoring networks.  Add a new jail as follows;

[default-iptables]

enabled  = true

filter   = default

action   = iptables[name=default, port=ssh, protocol=all]

sendmail-whois[name=default, dest=someone@geeforce.net, sender=fail2ban@fw.clientdomain.com]

logpath  = /var/log/fail2ban

maxretry = 1

In our case no other jails are relevant on the firewall, so they have all been set to “enabled = false”.  Now we edit the defaults action-iptables.conf file found in /etc/fail2ban/action.d

[Definition]
actionstart = iptables -N fail2ban-<name>
iptables -A fail2ban-<name> -j RETURN
iptables -I FORWARD -p <protocol> -j fail2ban-<name>
actionstop = iptables -D FORWARD -p <protocol> -j fail2ban-<name>
iptables -F fail2ban-<name>
iptables -X fail2ban-<name>
actioncheck = iptables -n -L FORWARD | grep -q fail2ban-<name>
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP
actionunban = iptables -D fail2ban-<name> -s <ip> -j DROP

[Init]
name = default
port = default
protocol = all

Finally we set up a chroot jail for a user on the firewall (Fail2ban) that has sudo permissions to run 2 commands /usr/bin/fail2ban-client and /bin/touch /var/log/fail2ban.  Be sure to touch /var/log/fail2ban and ensure that it is owned by the Fail2ban user.  This is necessary because the Fail2ban client doesn’t work the way you might think, and the user has to be able to touch that file to update the date/timestamp.  Create a ssh key pair to allow logins from remote hosts.

On the hosts, install Fail2ban and edit the /etc/fail2ban/jail.conf file.  Add the following jail;

[ssh-rban]
enabled  = true
filter   = sshd
action   = rban[name=SSH, fwip=IP.OF.FW.HERE]
logpath  = /var/log/secure
maxretry = 2

Create the file /etc/fail2ban/action.d/action-rban.conf that looks like the following

[Definition]
actionstart =
actionstop =
actioncheck = touch /var/log/fail2ban
actionban = /usr/bin/ssh -v -l fail2ban -tt <fwip> ‘sudo /usr/bin/fail2ban-client set default-iptables banip <ip>’ &&
/usr/bin/ssh -v -l fail2ban -tt <fwip> ‘/bin/touch /var/log/fail2ban’;
actionunban =

Test ssh connectivity by logging in to with fail2ban user from your host as root (fail2ban usually runs as root). Once you’ve succeeded in logging in, start Fail2ban on both machines and test.  This example only checks SSH logs for failed logins.  You can modify this setup as a template for any of the checks available for Fail2ban using the same action. This includes denial of service attacks, pop3 login failures, web login failures, and more.

Security Considerations

The methodology employed here took into account several different security and management considerations. One of the more common variations of this theme is to allow the remote Fail2ban server to add an iptables rule on the firewall directly. The method was rejected outright because providing the ability for a host to directly manipulate the firewall rules is extremely problematic from a security standpoint.  In the arrangement outlined here, a protected host could only block other hosts using the Fail2ban client.  Even the ability to unban clients is purposely disallowed.  The firewall itself makes the decisions on when to unban an IP.

An attacker that has gained entry to a protected host cannot override the centralize white list, nor does the firewall user allow an attacker to directly manipulate iptables.  The Fail2ban user on the firewall is specifically setup to be highly unprivileged, and limited, especially with a chroot jail in place.  The firewall essentially doesn’t trust the servers it protects.

This arrangement centralizes the management of all whitelisted IPs.  That makes changes very easy to roll out.  One caveat on the arrangement outline here; this has been specifically setup to block the offending IP from reaching ANY port on ANY machine for the duration of the ban time (defined in /etc/fail2ban/jail.conf).  That was a design choice, but the Fail2ban setup lends itself to banning specific service from offending hosts as opposed to banning all services. A denial of service can result from innocent people tripping the Fail2ban actions.  In this situation, that risk was considered minimal in regards to the amount of protection provided.

Conclusions;

Using Fail2ban to centralize the blocking of hosts has been a success everywhere it’s been deployed.  In one cluster alone, the typical /var/log/secure files went from being 10-20Mb or more every week, to a mere 200K.  Tens of thousands of password fails have been replaced by as little as 15.  By turning on different filters, one is also able to catch dictionary and brute force attacks against SMTP servers, WordPress sites, and webmail installations and protect them all with the same firewall.  This solution isn’t limited to the Linux or even Unix world.  With a centralized log server one can use Fail2ban to protect mixed environments, including Windows and Mac servers and centralize administration even more.   Fail2ban and an active centralized response to attacks just scratches the surface of a fully fleshed out security infrastructure.

Xiotech gets storage

I enjoyed the Xiotech presentation by Peter Selin whose presentation followed mine.  His emphasis on true Total Cost of Ownership (TCO) calculations and understanding how the applications use storage dovetailed very nicely with points I had made earlier.  Xiotech’s presentation went a step further and went into application tuning and how that affects storage performance.  A good part of the presentation was “SSD  facts or fiction”.  There was an enlightening graph on  SSD (Solid State Drives) sustained IOPS  vs Time.  This was nothing new for those of us with SSD server experience, but an eye opener for a lot of people in the room.

If you’re unfamiliar with Xiotech’s concept, then now might be the time to explain what they do.  Xiotech looks at storage like a “black box”.  It doesn’t matter what’s in the box – what matters is the capacity, throughput and reliability of the data storage.  Their solutions utilize Fiber Channel and provide the foundation for a high performance SAN (Storage Area Network).   One of the most unique aspects is that the end user no longer worries about individual drives and data redundancy. Data redundancy  is taken care of by “the box”.  To add more capacity, add another box.  This moves intelligence out of controllers and applications into storage where it belongs.  Just like intelligent networks, having the right intelligence in the right place makes a lot of sense.

I haven’t had a chance to test or use the products but their architecture deserves a very close look when high performance storage is called for.  The company is talking about Fiber Channel over Ethernet in the future and I hope that they also look at Ata Over Ethernet (AoE) as well.

Cisco goes after the datacenter

Cisco’s presentations always pique my interest.  This is a company that spends a lot of time figuring out how to produce a better mouse trap (or buying the company that has) and it shows.

Network and Storage Cisco’s approach is a continuation of the approach that it helped pioneer, network convergence.  Yesterday’s converging voice, video, and data via IP is passé; Cisco is now converging the SAN/LAN (Local Area Network) networks into a unified fabric.  With Fiber Channel over Ethernet the same network is used for SAN and LAN connectivity, simplifying cabling and switches.  With Cisco’s Fiber Channel/Ethernet modules for their Nexus class switches, Cisco is providing a bridge between the current SAN and LAN.  With 10GE (Gigabit Ethernet) networks already here and 40GE just around the corner, the writing is on the wall.  Eventually all LAN & SAN traffic will be carried on the same network.  Robert Metcalfe’s invention lives eternal.

Servers and Virtualization This part of Cisco’s offering is where we see radical innovation. Cisco doesn’t have a history of building servers so their approach is clean sheet  and unique from what I’ve seen from other vendors.  What Cisco did was look at large virtualized environments holistically not just focusing on server, storage, or network individually.  Cisco has tried to converge and unify many components of a large virtulized environment and build management into the entire environment from the get go.  They call their approach the Unified Computing System or UCS.

The UCS structure combines a unified (or should we say converged?) 10GE network fabric with unique super high memory blade servers that can support up to 384 GB DDR3.  The management of the entire structure is built in.  Cisco provides for a virtualized switch within each blade, each virtualized server can be centrally managed in it’s entirety.  Moving a virtual instance from one blade to another becomes simpler because the network moves with the instance and doesn’t require reprogramming the switch.  Cisco’s approach will change the entire management experience for large virtualized environments.

Both presentations have given me a great excuse to deep dive into the vendor’s technology and applications thereof.  Cisco is showing off their C-Series servers in Orlando on March 9th (register for that even t here) and there is some good reading material over a Xiotech.  Keeping up with new technology and it’s application is one of the things I enjoy most about my job.