Mailing List Archive: packet replication

Hi,

I have a couple of questions regarding netfilter capability.

I am building a network management system (a set of servers and
software on a large network) and encountering a number of challenges.
I would like to be able to send all NMS traffic (snmp, syslog, netflow
etc) to a single linux-based server per location, which then forwards
the traffic to an associated grouping of servers that are 'behind'
this server or on the same subnet.

However I have multiple servers that want to receive the same (udp)
traffic stream. This is partly to achieve redundancy, partly because
there are limitation in how many destinations can be set on the device
sending the traffic.

SO, iptables can handle the forwarding and NAT, but can I forward a
udp stream to more than one device?

Secondly - and this is a big ask but I am mainly looking for ideas
about how I might achieve this - I would like the status or
reachability of the end device to be tracked by the packet forwarder,
such that if the device becomes unreachable the traffic is disk cached
on the gateway until it becomes reachable again, or storage limits are
reached.

To illustrate this, imagine I have a router sending syslog to the "NMS
gateway" - the server running iptables. The gateway translates the dst
address in the ip header to that of a logging server - the final
destination of the syslog traffic. After a while it determines the
logging server is unreachable, and forwarded traffic is probably not
getting through. It starts caching all traffic on disk. A short time
later the logging server becomes reachable and the cached traffic is
forwarded. One way to achieve this would be by converting the udp
stream to individual tcp streams.

Is there an application/feature/tool that can handle this?

Would appreciate any info or pointers in the right direction.

cheers,

Ben

On 08/17/07 00:14, Ben Carbery wrote:
> However I have multiple servers that want to receive the same (udp)
> traffic stream. This is partly to achieve redundancy, partly because
> there are limitation in how many destinations can be set on the
> device sending the traffic.

(See below.)

> SO, iptables can handle the forwarding and NAT, but can I forward a
> udp stream to more than one device?

To my knowledge, IPTables its self does not have this capability.

> Secondly - and this is a big ask but I am mainly looking for ideas
> about how I might achieve this - I would like the status or
> reachability of the end device to be tracked by the packet forwarder,
> such that if the device becomes unreachable the traffic is disk
> cached on the gateway until it becomes reachable again, or storage
> limits are reached.

(See below.)

> Is there an application/feature/tool that can handle this?

Not per say, at least not as far as I'm aware of. There is nothing that
I'm aware of that will duplicate traffic to multiple destinations like
you are wanting to nor is there any thing that will queue your traffic
in the event that the endpoint can not be reached.

However there are a couple of different options you may want to
investigate. Namely, the ULOG target and the method that allows you to
send packets from kernel space to user space (and then back if so
desired) to decide how to process the packets. The intent being to
intercept packets in their path, hand them off to a user space process
that will do the duplication and queuing that you are talking about. If
it were me, I'd queue everything and then have another process that
empties the queue by transferring the traffic to your end point
system(s). This will make the process that diverts the traffic simpler.

Have the kernel divert the traffic, have a user space process log the
traffic, have one or more user space processes send the traffic to an
end point, have a final user space process remove the previously
transmitted traffic from the queue. This type of structure will allow
you to send the traffic to as many places as you would like also with
out the concern of the kernel or the logging process having to know any
thing about the end points. This system would be very modular and
probably fairly scalable. The only potential problem that I see is that
as stated above, is that if there are multiple end points over the same
WAN link you may end up transferring the same data across the WAN link
multiple times. However if the logging process speaks the same (to be
defined) protocol as the sending process you could in theory put another
logging process at the remote end and transfer the data one time from
one log to another log and then duplicate it from there. Hence saleability.

> Would appreciate any info or pointers in the right direction.

If you would like to discuss details drop me a line.

Grant. . . .