Mailing List Archive

Suggestions for per user bandwidth accounting over a router device?
This is almost certainly the wrong place to ask, but have any clever
folks here got some ideas for doing per user (and eventually per
user/per protocol) accounting for data crossing a router box (running
gentoo)?

The situation is something like: users connect to the router, are
authenticated and then the router forwards data through one of several
WAN connections (wireless, 3g and dialup). The goal is to track usage
per user across each *WAN connection* in order to bill appropriately
(because it costs more to use dialup than to use wireless, but sometimes
dialup is all that is available - the hotspot is mobile...)

For various reasons we can't assume that the bytes in/out from the LAN
to the router are the same as the bytes sent over the WAN, eg in the
simplest case lets add a squid proxy to the router and now the billable
WAN data may be much lower than the LAN data. Additionally the router
will host a simple CMS/file share that won't cause data to go over the WAN

I guess the pieces are:

1) User authentication on the wired/wireless (LAN) input side (802.1x +
perhaps a captive portal which also auths via radius?)

2) Gateway which allows authenticated users to route through the various
WANs

3) Packet accounting that collects info on the data used over each WAN


I'm somewhat unsure what my options are though to pass through the
per-user auth info collected in step 1 to other pieces of the puzzle? I
could build a mapping of users to IP addresses and then IP becomes a
synonym for "user". Are there other packet marking techniques I could
use that will scale to hundreds of users? (eg vlans appear not to be a
sensible option?)

I think 3) will often require support from the applications running on
the router itself, eg a web connection passing through squid is hard to
account for across the WAN because it all looks like a mass of data from
the squid process. Any thoughts on a scalable way to account for the
data from each app and log it? (Radius / DB / some library which someone
already wrote?)

Ideally I would like to be able to show quite granular statistics for
each user, eg connection at 8pm for 10 mins, 200KB of email, 5KB web,
2KB DNS. Can radius be (mis) used to track accounting to this low level?

The WAN connections in this scenario are quite expensive and we have a
requirement to track quite granularly...


Thanks for any pointers to techniques I could use to solve any of the
pieces above? Note: I'm currently pulling apart wifidog and coova to
get some ideas about how some of the captive portals implement the
gateway part, but their bandwidth accounting is all measured at the user
side and I need to measure mine at the WAN side...

Thanks

Ed W
Re: Suggestions for per user bandwidth accounting over a router device? [ In reply to ]
On 03/11/11 07:41, Ed W wrote:
> This is almost certainly the wrong place to ask, but have any clever
> folks here got some ideas for doing per user (and eventually per
> user/per protocol) accounting for data crossing a router box (running
> gentoo)?


Architecture
Via an iptables router or transparent bridge?
Too many issues for advise. What every you do,
you'll re-engineer it over and over again.
This is why carriers have hundreds of coders
and thousands of folks working on their code.
It's a task, that is NEVER COMPLETE!

(There is a reason the carriers of the world
abandoned ATM, as it had this sort of granularity
built in....caveat emptor!


Method 1
http://conntrack-tools.netfilter.org/

Your going to overwhelm an embedded system with all of this
accounting and database, so split it across several
systems. Choose the processors wisely and make sure
they support massive amounts of ram. Choose your final database wisely
and only track what you need. BLOAT will kill the project.


Method 2
Adapt an excellent high end NMS (Network Management System)
Such as Nagios or JFFNMS to your needs. JFFNMS
does quiet a lot with cisco and other routers,
pumps data into mysql or postgresql-9; Postgresql-9
has many new features and is my choice, but
it's a BEAST to get your arms around. Unlimited
scaling and lots of SeLinux support, at least
in Gentoo. JFFNMS also supports TACAS and
TACAS+, which, if it encompasses what
you need, would be your best route to avoid
a monstrous amount of coding on your own.


I'm sure there are other methods. ANY method you choose
will be a gargantuan task to implement and maintain.
Many go down this path (bit counting) most fail
miserably!

I would go with JFFNMS, TACAS+ and Postgresql
as the backend database (and sql-ledger). If there are
features you require that TACAS+ does not provide
build a transparent bridge and use ConnTrak
in IPtables, to shove that additional data not
provided by TACAS+, into your postgresql database.
MySQL might be a better choice for you database
depending on how big and what you (your staff) know.

For billing purposes, use this software
SQL-LEDGER for double entry accounting
(it works great with postgresql!)
It's been around for a long time and very
mature open source. Not sure of the license
any more, read up on that.....

http://sql-ledger.com/

See you in a year....
TTFN(ta ta for now).
James
Re: Suggestions for per user bandwidth accounting over a router device? [ In reply to ]
Hi

Thanks for the reply!

> Method 1
> http://conntrack-tools.netfilter.org/

Super - actually I just discovered ulogd which is I guess the preferred
userspace logger now. I think I'm a bit out of date on iptables because
that appears to be able to do even per connection statistics... Only
skimming the docs at present, anyone got any experience using this in anger?

>
> Your going to overwhelm an embedded system with all of this
> accounting and database, so split it across several
> systems.

Actually, although not stated, the WAN connections will be generally
quite slow and expensive (satellite), and the number of users normally
small. So I'm not expecting a ton of traffic to log in general


>
> Method 2
> Adapt an excellent high end NMS (Network Management System)
> Such as Nagios or JFFNMS to your needs

I hadn't come across JFFNMS before - very cool

I presume you have seen that Nagios has very firmly forked to become Icinga?


> in Gentoo. JFFNMS also supports TACAS and
> TACAS+, which, if it encompasses what
> you need, would be your best route to avoid
> a monstrous amount of coding on your own.

I don't see that TACAS+ offers the accounting side? From a quick google
it appears to handle the authentication side only?

My requirements for authentication are going to be fairly
straightforward, largely just yes/no.


From a few mins reading up my initial design is looking a little like:

- FreeRadius on sqlite (perhaps mysql)
- HostAPD
- IPTables to limit access (with daemon to talk to DHCP server)
- ulogd to log most of the traffic. Custom app loggers to add
granularity where needed

It's the accounting side and the use of iptables to limit access which
is still looking rather hairy. If anyone has any experience of fiddling
with this stuff then please let me know? Also any other features of
iptables that I might have not noticed would be useful? (I see packet
marking, vlans, mac matching, conntrack based accounting - anything else?)

Thanks for the hints

Ed W