Hi,

A few months ago I sent an email to these lists describing a problem I was
having.  I was putting an awful lot of "count" rules into a NetBSD PC-hardware
based router, and running into OS performance problems.  Because of this
problem I had to greatly reduce the number of rules I was using with IPFILTER
on a production system, which resulted in me having less accuracy in data
reporting than I would have liked.

Since then, I have done some testing with the help of some people at work in a
semi-controlled environment to help better clarify the problem.  I've created
a table summarizing what we observed.

The following table has four columns;  

# of Rules:   the number of 'count' rules used in the test

latency(1):   min/avg/max ping times on 300 "ping" packets across the router,
              with negligible traffic being forwarded (in milliseconds)

throughput:   Kbytes/s as measured by ttcp.c

latency(2):   as above latency(1) but while a ttcp stream was saturating the
              router (measured in milliseconds)

------------------------------------------------------
Effects of filter rule count on latency and throughput
------+-----------------+-----+-----------------------
Rules | latency(1)      | thpt| latency(2)
------+-----------------+-----+-----------------------
20000 | 24.3/37.8/488.0 |  -- |      --
18000 |     --          |  -- | 139.6/305.3/557.1
17500 | 24.5/40.4/373.1 |  -- |      --
16000 | 24.6/40.8/630.4 |  55 | 110.7/243.1/367.4
15000 | 16.8/19.5/83.5  |  -- |      --
14000 | 23.0/25.1/142.9 |  65 | 112.3/218.9/373.0
12000 | 19.8/20.6/46.6  |  70 | 111.0/197.8/287.2
10000 | 11.0/11.9/82.8  |  90 | 85.5/180.8/382.4
 7500 | 12.5/13.3/25.1  | 110 | 80.6/150.2/241.3
 5000 | 5.9/6.4/24.2    | 180 | 58.5/95.2/129.7
 4000 | 7.1/7.3/12.1    | 230 | 54.8/73.7/90.4
 3000 | 5.5/5.7/16.5    | 300 | 49.4/55.9/67.3
 2000 | 2.7/2.9/6.7     | 445 | 22.6/38.1/43.6
 1500 | 3.0/3.1/9.1     | 550 | 14.5/29.3/39.0
 1200 | 2.5/2.6/4.2     | 680 | 21.8/26.0/28.2
 1100 | 2.4/2.4/3.4     | 740 | 13.4/23.6/26.7
 1000 | 2.2/2.3/11.8    | 800 | 18.8/21.5/25.3
  800 | 1.9/2.0/3.4     | 830 | 2.4/14.3/42.9
  600 | 1.6/1.6/2.5     | 840 | 1.6/11.6/26.6
  400 | 1.3/1.3/4.7     | 795 | 1.3/10.9/39.4
  300 | 1.2/1.2/2.0     | 795 | 1.2/11.6/26.4
  200 | 1.0/1.1/6.9     | 840 | 1.1/8.6/23.2
  100 | 0.9/0.9/4.2     | 790 | 0.9/12.0/31.8
    2 |                 | 815 | 0.7/7.8/19.8


Other observations:

- interactive shell response time was normal on the console and on telnet
  connections for all rule counts when forwarding levels were negligible,
  however response time went from "boggy" (seconds) to "unusable" (minutes)
  at around 2000 rules when forwarding traffic was heavy
- test router was a Pentium 166 with 32MB ram, 3 3C905B cards
- Stock NetBSD 1.4 system with stock IPFILTER
- kernel config I built myself
- testing was performed on a NetBSD-1.4 workstation and a Linux workstation
  on either side of the router box
- filter load time (eg 'time ipf -Fa -f /etc/ipf.conf') seemed to increase 
  exponentially with number of rules being loaded
- kernel would panic at somewhere slightly higher than 20000 rules or more
  being loaded
- packet loss was generally 0% for all tests
- the test environment was not as 'perfect' as I would have liked ie. there
  was some data going through the network not related to the above tests
- no 'quick' keywords were used
- machine load average was less than 0.1 even when the machine was "locking
  up" because of traffic forwarding
- one of the workstations had a 10Mbit card.  Everything else was 100Mbit.

Interpretations:

1.  The ability of the router to forward packets quickly diminishes as a large
    number of IPF count rules are loaded

2.  latency times through the router while under heavy load are significantly
    higher than latency times through the router while under minimal load

3.  Using NetBSD/i386 in combination with IPFILTER to count a lot of
    fine-grained data going through a router doesn't work very well, at least
    with the versions of the softare that I tested.


Disclaimer:

Please do not interpret this email as an attack or a complaint against
IPFILTER or any of the operating systems that use it. I understand that my
tests were very limited in terms of implementation, hardware, OS choice, and
IPFILTER revision, and may not even be relevant to some.  I'm simply
presenting what I've observed, because I don't personally have the ability to
do anything about it.  I do believe there are a lot of highly skilled people
out there who might be interested.  So here it is.

If anybody wants to ask me questions on this topic or cut me up or mailbomb
me, feel free to send me an email at 'blackbox@openface.ca'.  I'd be happy to
clarify anything from this email or otherwise discuss the subject further.

Erik Rungi