Mailing List Archive

On Fri, 2007-07-27 at 23:30 +0200, Július Bemš wrote:
> Hi,
>
> When I need to add som rules (in my case 100) to some chain it takes various
> time, depending on the number of existing rules in the chain. So when I add
> 100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
> chain, it takes 4.5sec and if there is 10000 rules it takes 21sec.
>
> My problem is, that i need to add this rules to chain which contains 20000
> rules in short time. I think, that the way of adding rules into chain is
> very innefective.
>
> Could someone tell me how the adding works internally? What data structures
> are used? Because I need solve this and find the way how to add new rules
> quickly.
>
> Thanks for replies
>
>
We face the same problem on the ISCS project
(http://iscs.sourceforge.net). There, do to micro-perimeter network
security, we frequently generate thousands or tens of thousands of rules
with the click of a mouse.

We handle it by adding rules via iptables-restore rather than iptables.
The load time difference is remarkable. You write your rules into files
with very similar syntax to iptables and then direct them into
iptables-restore, e.g.,

iptables-restore < /etc/PEP/rules.txt

or, if you do not want to overwrite existing rules,

iptables-restore -n < /etc/PEP/runtimerules.txt

Hope this helps - John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com

John A. Sullivan III wrote:

>
> We handle it by adding rules via iptables-restore rather than iptables.
> The load time difference is remarkable. You write your rules into files
> with very similar syntax to iptables and then direct them into
> iptables-restore, e.g.,

The new Perl-based Shorewall rules compiler has adopted this same strategy.
As John says, the difference in performance is remarkable.

-Tom
--
Tom Eastep \ Nothing is foolproof to a sufficiently talented fool
Shoreline, \ http://shorewall.net
Washington USA \ teastep@shorewall.net
PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key

Thank you very much ... good idea.
But do you know why is iptables so slow? Because I think about writing my own utility for adding new rules.

-----Original Message-----
From: netfilter-bounces@lists.netfilter.org [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of John A. Sullivan III
Sent: Friday, July 27, 2007 11:38 PM
To: Július Bemš
Cc: netfilter@lists.netfilter.org
Subject: Re: adding rules - slow

On Fri, 2007-07-27 at 23:30 +0200, Július Bemš wrote:
> Hi,
>
> When I need to add som rules (in my case 100) to some chain it takes various
> time, depending on the number of existing rules in the chain. So when I add
> 100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
> chain, it takes 4.5sec and if there is 10000 rules it takes 21sec.
>
> My problem is, that i need to add this rules to chain which contains 20000
> rules in short time. I think, that the way of adding rules into chain is
> very innefective.
>
> Could someone tell me how the adding works internally? What data structures
> are used? Because I need solve this and find the way how to add new rules
> quickly.
>
> Thanks for replies
>
>
We face the same problem on the ISCS project
(http://iscs.sourceforge.net). There, do to micro-perimeter network
security, we frequently generate thousands or tens of thousands of rules
with the click of a mouse.

We handle it by adding rules via iptables-restore rather than iptables.
The load time difference is remarkable. You write your rules into files
with very similar syntax to iptables and then direct them into
iptables-restore, e.g.,

iptables-restore < /etc/PEP/rules.txt

or, if you do not want to overwrite existing rules,

iptables-restore -n < /etc/PEP/runtimerules.txt

Hope this helps - John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com

Hi,

Le mercredi 01 août 2007 à 09:24 +0200, Július Bemš a écrit :
> Thank you very much ... good idea.
> But do you know why is iptables so slow? Because I think about writing my own utility for adding new rules.

It's a design problem: when you add a new rule, the whole table id
dumped in userspace, modified and sent back to kernel space.

And this take long when there is a large number of rules !

BR,
--
Éric Leblond, eleblond@inl.fr
Téléphone : 01 44 89 46 39, Fax : 01 44 89 45 01
INL, http://www.inl.fr

В Срд, 01/08/2007 в 09:24 +0200, Július Bemš пишет:
> Thank you very much ... good idea.
> But do you know why is iptables so slow? Because I think about writing my own utility for adding new rules.

date; I=0; while [ $I -lt 1000 ]; do I=`expr $I + 1`; iptables
>/dev/null 2>&1; done; date
Wed Aug 1 10:36:32 EEST 2007
Wed Aug 1 10:36:39 EEST 2007

As you can see just invoking iptables 1000 time from a shell takes 7
seconds on my machine. Shell is slow. Moreover when there are many rules
each iptables invocation does same checks that many times.
iptables-restore do much of the work once for all rules.

> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of John A. Sullivan III
> Sent: Friday, July 27, 2007 11:38 PM
> To: Július Bemš
> Cc: netfilter@lists.netfilter.org
> Subject: Re: adding rules - slow
>
> On Fri, 2007-07-27 at 23:30 +0200, Július Bemš wrote:
> > Hi,
> >
> > When I need to add som rules (in my case 100) to some chain it takes various
> > time, depending on the number of existing rules in the chain. So when I add
> > 100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
> > chain, it takes 4.5sec and if there is 10000 rules it takes 21sec.
> >
> > My problem is, that i need to add this rules to chain which contains 20000
> > rules in short time. I think, that the way of adding rules into chain is
> > very innefective.
> >
> > Could someone tell me how the adding works internally? What data structures
> > are used? Because I need solve this and find the way how to add new rules
> > quickly.
> >
> > Thanks for replies
> >
> >
> We face the same problem on the ISCS project
> (http://iscs.sourceforge.net). There, do to micro-perimeter network
> security, we frequently generate thousands or tens of thousands of rules
> with the click of a mouse.
>
> We handle it by adding rules via iptables-restore rather than iptables.
> The load time difference is remarkable. You write your rules into files
> with very similar syntax to iptables and then direct them into
> iptables-restore, e.g.,
>
> iptables-restore < /etc/PEP/rules.txt
>
> or, if you do not want to overwrite existing rules,
>
> iptables-restore -n < /etc/PEP/runtimerules.txt
>
> Hope this helps - John
--
Покотиленко Костик <casper@meteor.dp.ua>