Rate limiting w/ iptables

About a month ago, my datacenter recieved a 10 Gbps Distributed Denial of Service attack that lasted until now, when i decided to finally, setup per-ip rate limiting using the powerful (and dangerous) tool iptables.

Please note that iptables is extremely dangerous, you can mess up and lose ssh access to your server. I have done that twice and recovered via my Aten iKvm, only do this to your physical servers if you can immediately access the datacenter/have a fallback server connection solution.

That's alot of intro, let's start doing the thing.

Solution #1 (Probably won't work): If the DoS attack is from a single IP(which is usually not the case), you can just block that IP to get rid of the attack. iptables interacts with the kernel's network stack, so please watch your step and don't mess up, it's recommended to make a timer script that reverts the changes after an amount of time to make sure you don't lose access to the server.

First let's figure out how iptables works. iptables is a userland rogram, it manipulates netfilter callback functions. iptables has rules and chains, a rule is basically a logic to match packets, a chain isa bunch of rules for a packet to pass through and decide what to do with it. The packets finally reach a target, which can be ACCEPT or DROP, which is self explanatory.

Now, if you want to block a single IP, for example: 8.8.8.8, you should do:

$ sudo iptables --append INPUT --source 8.8.8.8 --jump DROP

This command basically means appending a rule to the INPUT chain, which means all inbound packets, and if the source IP of the packet is 8.8.8.8, it drops the packet.

If you somehow forgot what you just did or what rule you have in iptables, do iptables --list to list the stuff you have done to your poor kernel.

Now, let's figure out some useful stuff, the conntrack and log module. Basically, conntrack can find all connections, and log can log stuff for iptables, sounds cool? Let's create a rule to log all TCP connections using these two modules.

$ sudo iptables --flush # rebirth, begin and begin again
$ sudo iptables --append INPUT --protocol tcp --match conntrack --ctstate NEW --jump LOG --log-prefix "TCP INBOUND: "

This cool thing clears all the iptables rules and append a new rule to the INPUT chain which logs all inbound tcp connections, cool, I know.

It's not cool tho, if you leave it there on your production server, it will suddenly start to drop connections with the output nf_conntrack: table full, dropping packet.

What? We never specified any rule to drop packets. Well, conntrack has a maximum number of logged connections, you can't just infinitely feed it stuff. If the table is filled up, new connections gets dropped. This can't happen in production, let's tweak the system to avoid this!

$ conn_count=$(sysctl --values net.netfilter.nf_conntrack_count)
$ sysctl --write net.netfilter.nf_conntrack_max=${conn_count}
$ sysctl --write net.netfilter.nf_conntrack_buckets=$((${conn_count}/4))

Now we are a step closer to limiting the rate of new connections by having the ability to log and interact with new connections.

Solution #2 (Probably the best solution): The limit module allow us to perform rate limiting to all packets which goes through this rule. First, we need a new chain, we will name it RATE-LIMITING for now. We will make all inbound packets go through the RATE-LIMITING chain if they are new. Then inside the RATE-LIMITING chain we add the limiting rule.

$ sudo iptables --flush # rebirth, begin and begin again
$ sudo iptables --new-chain RATE-LIMITING
$ sudo iptables --append INPUT --match conntrack --ctstate NEW --jump RATE-LIMITING

Then, create a new rule in the RATE-LIMITING chain. Create a rule which matches no more than 50 packets per second. This will be our limit of connections our server will accept per second. The accepted packets jump to ACCEPT.

$ sudo iptables --append RATE-LIMITING --match limit --limit 50/sec --limit-burst 20 --jump ACCEPT

The rule limits packets by not matching them and making them pass through, so if it's matched, we should drop it.

$ sudo iptables --append RATE-LIMITING --jump DROP

So, this basically works like a timer, it resets every 1 second , if incoming connections uses up all 50 connections before timer reset, they are screwed(aka packets dropped), this mitigates attacks effectively.

Solution #3 (Won't work if the attack is distributed): The module hashlimit basically replaces the limit module, limit limits all connections, hashlimit limits per host connections, this can be cool if you want your services to be usable by regular users during the attack. Let's do this!

$ sudo iptables --flush # rebirth, begin and begin again
$ sudo iptables --new-chain RATE-LIMITING
$ sudo iptables --append RATE-LIMITING \
    --match hashlimit \
    --hashlimit-upto 50/sec \
    --hashlimit-burst 20 \
    --hashlimit-name conn_rate_limit \
    --jump ACCEPT
$ sudo iptables --append RATE-LIMITING --jump DROP

To achieve per ip rate limiting, we need to make hashlimit to group by IP, we do this by setting the hashlimit mode to srcip.

$ sudo iptables --append RATE-LIMITING \
    --match hashlimit \
    --hashlimit-mode srcip \
    --hashlimit-upto 50/sec \
    --hashlimit-burst 20 \
    --hashlimit-name conn_rate_limit \
    --jump ACCEPT
$ sudo iptables --append RATE-LIMITING --jump DROP

What we can also do is to log dropped connections, this prevents legit connections from being dropped and us not knowing that.

$ sudo iptables --append RATE-LIMITING \
    --match hashlimit \
    --hashlimit-mode srcip \
    --hashlimit-upto 50/sec \
    --hashlimit-burst 20 \
    --hashlimit-name conn_rate_limit \
    --jump ACCEPT
$ sudo iptables --append RATE-LIMITING --jump LOG --log-prefix "IPTables-Rejected: "
$ sudo iptables --append RATE-LIMITING --jump REJECT

Now, make our rules permanent since it's stored in RAM, we don't want to set all these again after reboot. First, save all the current rules to a file.

$ sudo iptables-save | tee rules.txt

To restore the rules, use iptables-restore

$ sudo iptables-restore < rules.txt

Done :D Good luck!

{{ message }}

{{ 'Comments are closed.' | trans }}