Hello,
I have a IPVS-NAT setup on RHEL6.5 64 bit (2.6.32-431.20.3.el6.x86_64).
The system has 4 GB of memory (a Dell R200). This setup primarily
services HTTP to real-servers and peaks around 10,000 http req/sec.
Anyway, I am finding that I need to restart iptables every few days to
prevent system CPU usage from going to 100%. When this happens, the
cluster starts getting super sluggish and things grind to a halt. Here's
my current iptables
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Table: nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 SNAT all -- 192.168.0.0/24 0.0.0.0/0
to:MY.PUBLIC.IP.HERE
2 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Here's an example of 'sar' output when I reached near 100% system usage
04:20:01 PM CPU %user %nice %system %iowait %steal
%idle
04:00:04 PM all 0.36 0.00 98.50 0.02 0.00
1.12
04:10:01 PM all 0.71 0.00 98.87 0.03 0.00
0.39
04:20:01 PM all 0.38 0.00 95.10 0.09 0.00
4.43
04:30:01 PM all 0.38 0.00 51.68 0.14 0.00
47.79
04:40:01 PM all 0.37 0.00 47.86 0.12 0.00
51.65
04:50:01 PM all 0.34 0.00 48.89 0.12 0.00
50.65
So I restarted iptables around 4:20 PM and you can see the %system usage
drop in half and things returned to 'normal' when I did that.
I only have two setting changed in /etc/sysctl.conf
net.netfilter.nf_conntrack_max = 768288
net.ipv4.ip_forward = 1
I've noticed from 'iptables -L' output, that my inactive connections has
reached around 1.3 million during peak usage.
As I write this email, my system usage percentage is about 30%, for 6,000
http requests per second, 100 Mb/s bandwidth through the NAT and
/proc/sys/net/netfilter/nf_conntrack_count reports 278515
Perhaps I am exhausting nf_conntrack_max ? Googling around, perhaps I
should configure iptables to NOTRACK the http(s) traffic? Anybody with
thoughts on what I am doing wrong and how I could support 1 Gb/s worth of
http traffic (~ 20,000 req/sec) through this setup?
Maybe I should give up on NAT and figure out IPVS-DR? :) NAT is just so
convenient to use! :)
Thank you!
daryl
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
I have a IPVS-NAT setup on RHEL6.5 64 bit (2.6.32-431.20.3.el6.x86_64).
The system has 4 GB of memory (a Dell R200). This setup primarily
services HTTP to real-servers and peaks around 10,000 http req/sec.
Anyway, I am finding that I need to restart iptables every few days to
prevent system CPU usage from going to 100%. When this happens, the
cluster starts getting super sluggish and things grind to a halt. Here's
my current iptables
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Table: nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 SNAT all -- 192.168.0.0/24 0.0.0.0/0
to:MY.PUBLIC.IP.HERE
2 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Here's an example of 'sar' output when I reached near 100% system usage
04:20:01 PM CPU %user %nice %system %iowait %steal
%idle
04:00:04 PM all 0.36 0.00 98.50 0.02 0.00
1.12
04:10:01 PM all 0.71 0.00 98.87 0.03 0.00
0.39
04:20:01 PM all 0.38 0.00 95.10 0.09 0.00
4.43
04:30:01 PM all 0.38 0.00 51.68 0.14 0.00
47.79
04:40:01 PM all 0.37 0.00 47.86 0.12 0.00
51.65
04:50:01 PM all 0.34 0.00 48.89 0.12 0.00
50.65
So I restarted iptables around 4:20 PM and you can see the %system usage
drop in half and things returned to 'normal' when I did that.
I only have two setting changed in /etc/sysctl.conf
net.netfilter.nf_conntrack_max = 768288
net.ipv4.ip_forward = 1
I've noticed from 'iptables -L' output, that my inactive connections has
reached around 1.3 million during peak usage.
As I write this email, my system usage percentage is about 30%, for 6,000
http requests per second, 100 Mb/s bandwidth through the NAT and
/proc/sys/net/netfilter/nf_conntrack_count reports 278515
Perhaps I am exhausting nf_conntrack_max ? Googling around, perhaps I
should configure iptables to NOTRACK the http(s) traffic? Anybody with
thoughts on what I am doing wrong and how I could support 1 Gb/s worth of
http traffic (~ 20,000 req/sec) through this setup?
Maybe I should give up on NAT and figure out IPVS-DR? :) NAT is just so
convenient to use! :)
Thank you!
daryl
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users