Trying to stop
4 Sep, 2008. Written by Tom Roggero
1) It is about real attacks.
2) No theory, just practice.
A) Detecting the attack.
1) Using netstat command
netstat -an | grep :80 | sort
netstat -n -p | grep SYN_REC | awk '{print $5}' | awk -F: '{print $1}
netstat -n -p|grep SYN_REC | wc -l
netstat -lpn|grep :80 |awk '{print $5}'|sort
netstat -an | grep :80 | awk '{ print $5 }' | awk -F: '{ print $1 }' | sort | uniq -c | sort -n
E.G of SYN_RECV attack o SYN Flooding to Apache (port 80).
192.168.0.3 is the apache server and 192.168.0.105 attacker ip.
tcp 0 0 192.168.0.3:80 192.168.0.5:60808 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60761 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60876 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60946 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60763 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60955 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60765 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60961 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60923 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61336 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61011 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60911 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60758 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60828 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61114 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61074 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60826 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60959 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60900 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60940 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60920 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60825 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60945 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60913 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61009 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60755 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60904 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61583 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60910 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60915 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60827 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61458 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60908 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61007 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60927 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60951 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60942 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61113 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60909 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60822 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60894 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60952 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60928 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60936 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60906 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61466 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60919 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60914 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60926 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60939 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60931 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60831 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60823 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60954 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60916 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60963 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60947 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61006 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60933 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60950 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60895 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60917 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61480 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60935 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60960 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60767 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60918 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60821 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61077 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60905 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61517 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60893 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60953 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60903 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61439 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61337 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61545 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61299 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61010 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60930 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60744 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60929 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60754 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61008 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61116 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60811 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60807 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60938 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60764 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60873 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60817 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61550 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60748 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60956 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60753 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61115 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60741 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61075 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60948 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60829 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60943 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61338 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60762 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60824 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60830 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61535 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60898 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60815 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60962 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60957 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60944 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60921 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60759 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60897 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61518 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60958 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60922 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60937 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60875 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60766 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60751 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60768 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60743 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:61076 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60912 SYN_RECV tcp 0 0 192.168.0.3:80 192.168.0.5:60816 SYN_RECV
Clear example of SYN Attack to Apache.
2) Looking Apache server-status
If we look apache server-status, we'll see conections on state "Reading" ("R" Reading Request).
The problem is when the Reading conections number fills the Apache MaxClient, so it doesn't accept any more conections despite of it were legal ones.
We can increase the value of MaxClient to not get the request queue limit so it accept any client, being attackers or not. (but it doesn't care, there will be "lag")
To decrease the value of Apache Timeout (so the Reading requests were killed fast, before the MaxClients limit is reached) could be good.
3) Looking mod_evasive logs
Jun 22 18:24:04 lan mod_evasive[3835]: Blacklisting address 82.228.169.50: possible attack. Jun 22 18:24:45 lan mod_evasive[3600]: Blacklisting address 81.206.164.163: possible attack. Jun 22 18:25:46 lan mod_evasive[3589]: Blacklisting address 155.232.250.19: possible attack. Jun 22 18:27:23 lan mod_evasive[3671]: Blacklisting address 83.227.217.2: possible attack. Jun 22 18:28:10 lan mod_evasive[3673]: Blacklisting address 68.187.171.89: possible attack. Jun 22 18:29:57 lan mod_evasive[3605]: Blacklisting address 70.143.2.130: possible attack. Jun 22 18:30:45 lan mod_evasive[3803]: Blacklisting address 69.157.93.88: possible attack. Jun 22 18:31:45 lan mod_evasive[10397]: Blacklisting address 146.64.81.22: possible attack. Jun 22 18:35:01 lan mod_evasive[3794]: Blacklisting address 66.38.192.134: possible attack. Jun 22 18:35:15 lan mod_evasive[3553]: Blacklisting address 81.190.204.64: possible attack. Jun 22 18:40:10 lan mod_evasive[16602]: Blacklisting address 64.231.39.129: possible attack. Jun 22 18:48:04 lan mod_evasive[16479]: Blacklisting address 84.99.195.100: possible attack. Jun 22 18:48:12 lan mod_evasive[16467]: Blacklisting address 201.0.10.142: possible attack. Jun 22 18:52:57 lan mod_evasive[16573]: Blacklisting address 219.95.39.242: possible attack. Jun 22 18:53:07 lan mod_evasive[16534]: Blacklisting address 86.129.3.91: possible attack. Jun 22 18:53:26 lan mod_evasive[16527]: Blacklisting address 62.254.0.32: possible attack. Jun 22 18:54:41 lan mod_evasive[30473]: Blacklisting address 24.196.199.191: possible attack. Jun 22 18:55:17 lan mod_evasive[30520]: Blacklisting address 142.161.157.227: possible attack. Jun 22 18:55:24 lan mod_evasive[30461]: Blacklisting address 65.92.145.133: possible attack. Jun 22 18:55:33 lan mod_evasive[30509]: Blacklisting address 88.111.227.200: possible attack. Jun 22 18:56:13 lan mod_evasive[30473]: Blacklisting address 69.199.94.227: possible attack. Jun 22 18:57:45 lan mod_evasive[30517]: Blacklisting address 86.125.135.212: possible attack. Jun 22 18:57:54 lan mod_evasive[30479]: Blacklisting address 84.192.141.65: possible attack. Jun 22 18:58:46 lan mod_evasive[30527]: Blacklisting address 83.140.97.106: possible attack. Jun 22 18:59:31 lan mod_evasive[30469]: Blacklisting address 82.173.216.196: possible attack. Jun 22 19:00:33 lan mod_evasive[30517]: Blacklisting address 80.176.157.245: possible attack. Jun 22 19:00:38 lan mod_evasive[30470]: Blacklisting address 86.133.102.51: possible attack. Jun 22 19:01:35 lan mod_evasive[30870]: Blacklisting address 24.42.134.253: possible attack. Jun 22 19:01:48 lan mod_evasive[30509]: Blacklisting address 62.254.0.34: possible attack. Jun 22 19:02:57 lan mod_evasive[31009]: Blacklisting address 81.227.219.125: possible attack. Jun 22 19:03:29 lan mod_evasive[31056]: Blacklisting address 172.209.173.153: possible attack. Jun 22 19:05:07 lan mod_evasive[31385]: Blacklisting address 84.6.12.110: possible attack. Jun 22 19:06:52 lan mod_evasive[31008]: Blacklisting address 85.227.144.249: possible attack. Jun 22 19:06:56 lan mod_evasive[31263]: Blacklisting address 213.222.156.222: possible attack. Jun 22 19:07:13 lan mod_evasive[31393]: Blacklisting address 62.163.143.166: possible attack. Jun 22 19:07:37 lan mod_evasive[31021]: Blacklisting address 62.135.101.73: possible attack. Jun 22 19:08:03 lan mod_evasive[31251]: Blacklisting address 82.201.249.69: possible attack. Jun 22 19:08:17 lan mod_evasive[31200]: Blacklisting address 81.62.65.53: possible attack. Jun 22 19:11:04 lan mod_evasive[31263]: Blacklisting address 82.39.148.204: possible attack. Jun 22 19:12:37 lan mod_evasive[31241]: Blacklisting address 213.222.154.13: possible attack. Jun 22 19:13:54 lan mod_evasive[31027]: Blacklisting address 81.51.79.4: possible attack. Jun 22 19:24:04 lan mod_evasive[31041]: Blacklisting address 84.221.118.156: possible attack. Jun 22 19:48:47 lan mod_evasive[3400]: Blacklisting address 62.135.101.192: possible attack. Jun 22 19:53:04 lan mod_evasive[31031]: Blacklisting address 62.30.33.13: possible attack. Jun 22 19:54:32 lan mod_evasive[31016]: Blacklisting address 72.14.194.18: possible attack. Jun 22 19:56:10 lan mod_evasive[31067]: Blacklisting address 198.96.34.58: possible attack. Jun 22 20:03:24 lan mod_evasive[5144]: Blacklisting address 172.213.33.242: possible attack. Jun 22 20:08:31 lan mod_evasive[5137]: Blacklisting address 83.241.11.16: possible attack. Jun 22 20:21:59 lan mod_evasive[6645]: Blacklisting address 201.23.193.20: possible attack. Jun 22 20:32:28 lan mod_evasive[7801]: Blacklisting address 212.38.134.172: possible attack. Jun 22 20:45:46 lan mod_evasive[7836]: Blacklisting address 81.247.11.48: possible attack. Jun 22 20:48:03 lan mod_evasive[7796]: Blacklisting address 70.245.98.186: possible attack. Jun 22 20:49:38 lan mod_evasive[7832]: Blacklisting address 61.8.138.203: possible attack. Jun 22 20:51:21 lan mod_evasive[7801]: Blacklisting address 201.132.197.161: possible attack. Jun 22 20:57:18 lan mod_evasive[10426]: Blacklisting address 82.201.249.67: possible attack. Jun 22 20:57:51 lan mod_evasive[7822]: Blacklisting address 81.77.26.162: possible attack. Jun 22 21:00:25 lan mod_evasive[7817]: Blacklisting address 200.39.202.243: possible attack. Jun 22 21:12:04 lan mod_evasive[7794]: Blacklisting address 84.27.139.25: possible attack. Jun 22 21:22:27 lan mod_evasive[7816]: Blacklisting address 217.208.98.254: possible attack.
If a DDoS is so distributed we will note that lot of differents ip ddos'd the Apache.
4) Looking syslog logs (of kernel)
May 17 13:39:01 lan kernel: possible SYN flooding on port 80. Sending cookies. May 17 13:39:02 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:39:35 lan kernel: NET: 4 messages suppressed. May 17 13:39:35 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:39:38 lan kernel: NET: 1 messages suppressed. May 17 13:39:38 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:39:43 lan kernel: NET: 6 messages suppressed. May 17 13:39:43 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:39:48 lan kernel: NET: 4 messages suppressed. May 17 13:39:48 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:39:52 lan kernel: NET: 9 messages suppressed. May 17 13:39:52 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:39:57 lan kernel: NET: 15 messages suppressed. May 17 13:39:57 lan kernel: ip_conntrack: table full, dropping packet. May 17 13:40:01 lan kernel: possible SYN flooding on port 80. Sending cookies.
To take in care:
possible SYN flooding on port 80. Sending cookies.
"Sending Cookies" if we have it enabled on /etc/sysctl.conf
# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1
Sometimes is better to disable it.
net.ipv4.tcp_syncookies = 0
So we can see the attacker IP
Jul 14 12:46:50 lan kernel: TCP: drop open request from 80.171.45.81/63069 Jul 14 12:46:55 lan kernel: NET: 1401 messages suppressed. Jul 14 12:46:55 lan kernel: TCP: drop open request from 80.103.166.148/4403 Jul 14 12:46:59 lan kernel: NET: 1772 messages suppressed. Jul 14 12:46:59 lan kernel: TCP: drop open request from 200.127.62.215/4019 Jul 14 12:47:05 lan kernel: NET: 2362 messages suppressed. Jul 14 12:47:05 lan kernel: TCP: drop open request from 85.57.169.142/19899 Jul 14 12:47:11 lan kernel: NET: 2618 messages suppressed. Jul 14 12:47:11 lan kernel: TCP: drop open request from 83.19.73.122/2710 Jul 14 12:47:14 lan kernel: NET: 898 messages suppressed. Jul 14 12:47:14 lan kernel: TCP: drop open request from 80.235.39.64/3554 Jul 14 12:47:19 lan kernel: NET: 1120 messages suppressed. Jul 14 12:47:19 lan kernel: TCP: drop open request from 80.171.45.81/62095 Jul 14 12:47:24 lan kernel: NET: 1714 messages suppressed. Jul 14 12:47:24 lan kernel: TCP: drop open request from 84.62.152.44/34014 Jul 14 12:47:29 lan kernel: NET: 2274 messages suppressed. Jul 14 12:47:29 lan kernel: TCP: drop open request from 200.127.62.215/3207 Jul 14 12:47:34 lan kernel: NET: 1552 messages suppressed. Jul 14 12:47:34 lan kernel: TCP: drop open request from 80.103.166.148/4797 Jul 14 12:47:39 lan kernel: NET: 4044 messages suppressed. Jul 14 12:47:39 lan kernel: TCP: drop open request from 80.235.39.64/2678 Jul 14 12:47:44 lan kernel: NET: 4360 messages suppressed. Jul 14 12:47:44 lan kernel: TCP: drop open request from 80.103.166.148/1312 Jul 14 13:04:15 lan kernel: TCP: drop open request from 200.14.237.83/4787 Jul 14 13:04:22 lan kernel: NET: 147 messages suppressed. Jul 14 13:04:22 lan kernel: TCP: drop open request from 81.38.172.161/4892 Jul 14 13:04:30 lan kernel: NET: 6 messages suppressed. Jul 14 13:04:30 lan kernel: TCP: drop open request from 200.14.237.83/4934 Jul 14 13:04:30 lan kernel: TCP: drop open request from 200.14.237.83/4935 Jul 14 13:04:38 lan kernel: NET: 76 messages suppressed. Jul 14 13:04:38 lan kernel: TCP: drop open request from 81.84.212.34/2861 Jul 14 13:04:40 lan kernel: NET: 269 messages suppressed. Jul 14 13:04:40 lan kernel: TCP: drop open request from 200.14.237.83/3070 Jul 14 13:04:45 lan kernel: NET: 287 messages suppressed. Jul 14 13:04:45 lan kernel: TCP: drop open request from 81.203.228.102/4400 Jul 14 13:04:50 lan kernel: NET: 98 messages suppressed. Jul 14 13:04:50 lan kernel: TCP: drop open request from 81.84.212.34/3961 Jul 14 13:04:54 lan kernel: NET: 245 messages suppressed. Jul 14 13:04:54 lan kernel: TCP: drop open request from 200.84.169.200/1183 Jul 14 13:05:00 lan kernel: NET: 1787 messages suppressed. Jul 14 13:05:00 lan kernel: TCP: drop open request from 81.203.228.102/2050 Jul 14 13:05:04 lan kernel: NET: 3208 messages suppressed. Jul 14 13:05:04 lan kernel: TCP: drop open request from 86.212.167.27/4720 Jul 14 13:05:09 lan kernel: NET: 2031 messages suppressed. Jul 14 13:05:09 lan kernel: TCP: drop open request from 81.203.228.102/1794 Jul 14 13:05:14 lan kernel: NET: 2221 messages suppressed. Jul 14 13:05:14 lan kernel: TCP: drop open request from 81.38.172.161/4908 Jul 14 13:05:21 lan kernel: NET: 730 messages suppressed. Jul 14 13:05:21 lan kernel: TCP: drop open request from 81.203.228.102/1430 Jul 14 13:05:25 lan kernel: NET: 234 messages suppressed. Jul 14 13:05:25 lan kernel: TCP: drop open request from 81.203.228.102/2939 Jul 14 13:05:30 lan kernel: NET: 1594 messages suppressed. Jul 14 13:05:30 lan kernel: TCP: drop open request from 200.14.237.83/3876 Jul 14 13:05:36 lan kernel: NET: 633 messages suppressed. Jul 14 13:05:36 lan kernel: TCP: drop open request from 86.212.167.27/1116 Jul 14 13:05:39 lan kernel: NET: 970 messages suppressed. Jul 14 13:05:39 lan kernel: TCP: drop open request from 81.38.172.161/3040 Jul 14 13:05:45 lan kernel: NET: 548 messages suppressed. Jul 14 13:05:45 lan kernel: TCP: drop open request from 81.203.228.102/2119 Jul 14 13:05:50 lan kernel: NET: 421 messages suppressed. Jul 14 13:05:50 lan kernel: TCP: drop open request from 81.203.228.102/2478 Jul 14 13:05:56 lan kernel: NET: 379 messages suppressed. Jul 14 13:05:56 lan kernel: TCP: drop open request from 81.203.228.102/4005 Jul 14 13:05:59 lan kernel: NET: 891 messages suppressed. Jul 14 13:05:59 lan kernel: TCP: drop open request from 81.38.172.161/3568 Jul 14 13:06:04 lan kernel: NET: 2221 messages suppressed. Jul 14 13:06:04 lan kernel: TCP: drop open request from 81.203.228.102/4532 Jul 14 13:06:09 lan kernel: NET: 243 messages suppressed. Jul 14 13:06:09 lan kernel: TCP: drop open request from 81.203.228.102/1939 Jul 14 13:06:14 lan kernel: NET: 2166 messages suppressed. Jul 14 13:06:14 lan kernel: TCP: drop open request from 81.38.172.161/2137 Jul 14 13:06:19 lan kernel: NET: 2071 messages suppressed. Jul 14 13:06:19 lan kernel: TCP: drop open request from 81.38.172.161/3136 Jul 14 13:06:24 lan kernel: NET: 2069 messages suppressed. Jul 14 13:06:24 lan kernel: TCP: drop open request from 81.84.212.34/4600 Jul 14 13:06:29 lan kernel: NET: 1797 messages suppressed. Jul 14 13:06:29 lan kernel: TCP: drop open request from 86.212.167.27/3171 Jul 14 13:06:35 lan kernel: NET: 1292 messages suppressed. Jul 14 13:06:35 lan kernel: TCP: drop open request from 81.203.228.102/1394 Jul 14 13:06:39 lan kernel: NET: 715 messages suppressed. May 17 14:13:24 lan kernel: ip_conntrack: table full, dropping packet.
Table FULL. We've a problem because we won't admit any more connections even if they are valid ones (real users).
We can increase the value of the table is we have a nice net.
Directly:
echo "65535" > /proc/sys/net/ipv4/ip_conntrack_max
To save the value and don't lost at rebooting, we've to add in the sysctl.conf
net.ipv4.ip_conntrack_max = 65535
REMEMBER reboot the net to apply the changes in the /proc (service network restart).
Martian packages:
Aug 31 12:41:29 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 12:45:07 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 12:52:57 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 12:58:55 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:08:12 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:12:03 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:34:38 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:37:38 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:52:42 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:56:18 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 13:59:54 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 14:13:32 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 14:38:08 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 14:43:42 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 14:50:05 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 14:51:05 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 14:57:58 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 15:05:27 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 15:06:14 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0 Aug 31 15:09:08 lan kernel: martian source 192.168.0.10 from 0.0.0.0, on dev eth0
They are unwaited packages that comes from a way which they can't come. It is a courage problem (cracker).
Using packages like these, can be attacked remote vulnerabilities on TCP/IP stacks
5) Looking MRTG graphics, RRDtool
If you see inbound traffic raises to 100mbps is DDoS
B) TRYING TO STOP THE ATTACK
1) mod_evasive
Official Web:
http://www.nuclearelephant.com/projects/mod_evasive/
We consider 50 connections per second to 2 pages is enough to block the ip:
<IfModule mod_evasive.c>
DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 900
</IfModule>
Like the last but 50 conn/sec to only 1 page:
<IfModule mod_evasive.c> DOSHashTableSize 3097 DOSPageCount 1 DOSSiteCount 50 DOSPageInterval 1 DOSSiteInterval 1 </IfModule>
If we want to block flooding ip, we could use iptables:
DOSSystemCommand
"sudo -u root -c '/sbin/iptables -A INPUT -s %s -j DROP"
Remember to see syslog, posibilities of positive falses (non flooding ip's).
To evit them (p.falses):
><IfModule mod_evasive.c> # add these lines (google bots range) DOSWhitelist 66.249.65.* DOSWhitelist 66.249.66.* </IfModule>
Important:
To have mod_evasive working right, you have to modify:
MaxRequestsPerChild 0
To write a high value but never ilimited (0).
MaxRequestsPerChild 10000
E.G: http://www.eth0.us/mod_evasive
2) mod_security
The unique problem of mod_security is that we need at least one argument to detect the attack.
In the example we use http_referer and the User Agent to detect DDoS:
Blocking an Iframe attack (spanish) http://foro.elhacker.net/index.php/topic,127481.0.html
3) tcplimit, ipdrop, ipblock
Using dynamic firewalls.
4) Optimizing and securing the net with sysctl.conf
cat /proc/sys/net/ipv4/tcp_syncookies net.ipv4.conf.all.rp_filter = 1 # Enable IP spoofing protection, turn on Source Address Verification net.ipv4.icmp_echo_ignore_broadcasts = 1 # Enable TCP SYN Cookie Protection net.ipv4.tcp_syncookies = 1 # Enable ignoring broadcasts request
1). Activate SynCookies protection
It works by sending out 'syncookies' when the
syn backlog queue of a socket overflows.
=> echo 1 >/proc/sys/net/ipv4/tcp_syncookies
or
=> /sbin/sysctl -w net.ipv4.tcp_syncookies=1
2). Disable source routing
=> for f in /proc/sys/net/ipv4/conf/*/accept_source_route; do
echo 0 > $f
done
or
=> /sbin/sysctl -w net.ipv4.conf.all.accept_source_route=0
3). Reverse Path Filtering
Reject incoming packets if their source address doesn't match
the network interface that they're arriving on
=> for f in /proc/sys/net/ipv4/conf/*/rp_filter; do
echo 1 > $f
done
or
=> /sbin/systcl -w net.ipv4.conf.all.rp_filter=1
4). Log RP filter dropped packets (martians)
=> for f in /proc/sys/net/ipv4/conf/*/log_martians; do
echo 1 > $f
done
or
=> /sbin/sysctl -w net.ipv4.conf.all.log_martians=1
5). Maximal number of remembered connection requests
=> /sbin/sysctl -w net.ipv4.tcp_max_syn_backlog=256
6). How may times to retry before killing TCP connection
(default 7 on most systems)
=> /sbin/sysctl -w net.ipv4.tcp_orphan_retries=4
7). Number of SYN packets the kernel will send before giving up
=> /sbin/sysctl -w net.ipv4.tcp_syn_retries=5
8). Disable broadcast icmp reply
=> /sbin/sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=1
9). Ignore Bogus icmp packets
=> /sbin/sysctl -w net.ipv4.icmp_ignore_bogus_error_responses=1
10). Disable ICMP redirect
=> echo 0 >/proc/sys/net/ipv4/conf/all/accept_redirects
=> echo 0 >/proc/sys/net/ipv4/conf/all/send_redirects
or
=> /sbin/sysctl -w net.ipv4.conf.all.accept_redirects=0
=> /sbin/sysctl -w net.ipv4.conf.all.send_redirects=0
11). Disable timestamps
=> echo 0 >/proc/sys/net/ipv4/tcp_timestamps
or
=> /sbin/sysctl -w net.ipv4.tcp_timestamps=0
12). Reduce DOS ability by reducing timeouts
=> echo 30 >/proc/sys/net/ipv4/tcp_fin_timeout
=> echo 1800 >/proc/sys/net/ipv4/tcp_keepalive_time
=> echo 0 >/proc/sys/net/ipv4/tcp_window_scaling
=> echo 0 >/proc/sys/net/ipv4/tcp_sack
or
=> /sbin/sysctl -w net.ipv4.tcp_fin_timeout=30
=> /sbin/sysctl -w net.ipv4.tcp_keepalive_time=1800
=> /sbin/sysctl -w net.ipv4.tcp_window_scaling=0
=> /sbin/sysctl -w net.ipv4.tcp_sack=0
- TCP variables list:
http://ipsysctl-tutorial.frozentux.net/chunkyhtml/tcpvariables.html
- /proc/sys/net/ipv4/* variables (with defaults and explanations)
http://ipsysctl-tutorial.frozentux.net/other/ip-sysctl.txt
More examples of complete configuration of sysctl.conf in the references.
5) APF Firewall with anti-ddos module
wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz tar xvzf apf-current.tar.gz cd apf-0.9.6-1/ ./install.sh service apf start /usr/local/sbin/apf -s
Configuration file:
/etc/apf/conf.apf
After testing, let:
DEVEL_MODE="0"
If we get a similar error like this:
apf(9413): unable to load iptables module (ip_tables), aborting.
Chage this:
SET_MONOKERN="1"
Wanted open ports (inbound)
IG_TCP_CPORTS="21,22,25,53,80,110"
If we want to block all the outgoing traffic we put it on 1 (outbound)
EGF="0"
If we want to use the antddos module:
USE_AD="0"
Log: /var/log/apf_log
To see the dropped packages:
LOG_DROP="1"
It will be save on syslog, eg:
Proto= Protocol
SRC= source ip
SPT= Source Port
DST= Destination Port
Oct 20 13:59:27 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=18779 PROTO=TCP SPT=11629 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:16 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20376 PROTO=TCP SPT=27734 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:17 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20382 PROTO=TCP SPT=25943 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:17 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20387 PROTO=TCP SPT=19026 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:17 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20397 PROTO=TCP SPT=2155 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:17 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20407 PROTO=TCP SPT=9294 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:22 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20687 PROTO=TCP SPT=9269 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:22 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20694 PROTO=TCP SPT=27223 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:23 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=20830 PROTO=TCP SPT=30938 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:25 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=21038 PROTO=TCP SPT=5377 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:27 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=21219 PROTO=TCP SPT=13341 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:00:42 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=21990 PROTO=TCP SPT=22960 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0 Oct 20 14:02:32 ns2 kernel: ** SANITY ** IN=eth0 OUT= SRC=213.27.201.254 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=26386 PROTO=TCP SPT=2826 DPT=80 WINDOW=0 RES=0x00 RST FIN URGP=0
Remember to use the antidos you should add the cron job:
*/8 * * * * root /etc/apf/ad/antidos -a >> /dev/null 2>&1
http://www.r-fx.org/apf/README.antidos
KISS My Firewall is an alternative: http://www.prism-hosting.com/AntiDoS
6) Stop the botnet
ZmbScap - Zombie Scapper - Stoopt DDoS Programs
http://www.metaeye.org/projects/zmbscap/
Tracking Botnets - Bot-Commands
http://www.honeynet.org/papers/bots/botnet-commands.html
Tracking Botnets
http://www.honeynet.org/papers/bots/
Tracking Botnets - DDoS-attacks
http://www.honeynet.org/papers/bots/botnet-ddos.html
Phatbot Trojan Analysis
http://www.lurhq.com/phatbot.html
F-Bot by f-secure- Elimina el Agobot y todas sus variantes
http://www.f-secure.com/tools/f-bot.zip
Nepenthes - Autoinfecarse sin peligro para analizar
http://nepenthes.mwcollect.org/
honeytrap – trap attacks against tcp services
http://honeytrap.sourceforge.net/
7) Using iptables rules
# all syn traffic -P INPUT DROP -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p tcp ! --syn -j REJECT --reject-with tcp-reset -A INPUT -m state --state INVALID -j DROP -P OUTPUT DROP -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A OUTPUT -p tcp ! --syn -j REJECT --reject-with tcp-reset -A OUTPUT -m state --state INVALID -j DROP -P FORWARD DROP -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT -A FORWARD -p tcp ! --syn -j REJECT --reject-with tcp-reset -A FORWARD -m state --state INVALID -j DROP -A INPUT -i lo -j ACCEPT -A OUTPUT -o lo -j ACCEPT -A FORWARD -i lo -o lo -j ACCEPT # increases the loading but get good signal -A INPUT -p tcp --syn -j REJECT --reject-with icmp-port-unreachable # the best -N syn-flood -A syn-flood -m limit --limit 100/second --limit-burst 150 -j RETURN -A syn-flood -j LOG --log-prefix "SYN flood: " -A syn-flood -j DROP # like the last but animal -N syn-flood -A INPUT -i eth0:2 -p tcp --syn -j syn-flood -A syn-flood -m limit --limit 1/s --limit-burst 4 -j RETURN -A syn-flood -j DROP -A INPUT -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -m limit --limit 1/sec -j ACCEPT -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 1/sec -j ACCEPT # no so effective -A INPUT -s 0/0 -p tcp --syn --source-port 1000:5000 --destination-port 80 -j DROP # no so effective -A INPUT -p tcp -m tcp --dport 80 --sport 1000:5000 --tcp-flags SYN SYN -j DROP # Discard some packages wrong formed -N PKT_FAKE -A PKT_FAKE -m state --state INVALID -j DROP -A PKT_FAKE -p tcp --dport 80 --tcp-flags ALL ACK,RST,SYN,FIN -j DROP -A PKT_FAKE -p tcp --dport 80 --tcp-flags SYN,FIN SYN,FIN -j DROP -A PKT_FAKE -p tcp --dport 80 --tcp-flags SYN,RST SYN,RST -j DROP -A PKT_FAKE -p tcp --dport 80 ! --syn -m state --state NEW -j DROP -A PKT_FAKE -f -j DROP -A PKT_FAKE -j RETURN # syn-flood -N syn-flood -A INPUT -i eth+ -p tcp --tcp-flags SYN,ACK,FIN,RST RST -j syn-flood -A FORWARD -i eth+ -p tcp --tcp-flags SYN,ACK,FIN,RST RST -j syn-flood -A syn-flood -m limit --limit 4/s --limit-burst 16 -j RETURN -A syn-flood -m limit --limit 75/s --limit-burst 100 -j RETURN -A syn-flood -j LOG --log-prefix "SYN FLOOD " --log-tcp-sequence --log-tcp-options --log-ip-options -m limit --limit 1/second -A syn-flood -j DROP # By pepel. Requires limit module -A INPUT -p tcp --dport 80 -m state --state NEW -m recent --set -A INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 -j DROP # explanation: # It adds every IP that connect to recent table # By each IP on the recent table, drop if it have more than X hits in X secs
Using mod_throttle
http://www.snert.com/Software/mod_throttle/
Others: Mod_Throttle, mod_bandwidth, mod_iplimit, mod_tsunami, mod_limitipconn.c
For Apache 2: mod_cband
cd /usr/src wget http://www.snert.com/Software/mod_throttle/mod_throttle312.tgz tar zxvf mod_throttle312.tgz cd mod_throttle-3.1.2 pico Makefile Then edit the line that reads: APXS=apxs And change it to read: APXS=/usr/local/apache/bin/apxs make make install service httpd restart <IfModule mod_throttle.c> ThrottlePolicy Volume 10G 30d </IfModule> <Location /throttle-me> SetHandler throttle-me </Location>
http://www.webhostgear.com/160.html
C) References
- Security options on linux through /proc (I) and (II) [spanish]
http://www.elhacker.net/opciones-seguridad-linux-proc.html
- Syctl.conf Hardening
http://www.eth0.us/sysctl
- Ipsysctl tutorial 1.0.4
http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html
- Hardening the TCP/IP stack to SYN attacks
http://www.securityfocus.com/infocus/1729
- DDOS and SYN_Recv Attacks And some SOlutions
http://www.vbulletin.com/forum/showthread.php?t=126699
- Distributed Reflection Denial of Service
http://www.grc.com/dos/drdos.htm
- Dynamic iptables firewalls
http://www-128.ibm.com/developerworks/library/l-fw/
- Preventing DDoS Attacks
http://www.linuxsecurity.com/content/view/121960/49/
- Distributed Denial of Service (DDoS) Attacks/tools
http://staff.washington.edu/dittrich/misc/ddos/
- Linux firewall rulesets and snippets of rule sets
http://www.gotroot.com/tiki-index.php?page=Linux%20Firewall%20rules