Mailing List Archive

Andrew Fort(afort at choqolat.org)@2002.09.02 22:16:20 +0000:
> How do you cope with these regular changes operationally with RANCID?
> Architectural staff would like to avoid seeing the ACL/pfx.list changes that
> occur when customer registers a route with your IRR, but you'd still like to
> know when ops folk accidentally smoke some important part of the router
> config.

I've the same issue, save it is with static route entries for secondary IPs. Also,
I don't care about saved configs. I asked a similar question, but was basically told that
the current release doesn't support it.

> 1. (messy) munge the rancid output for the filter changes when you do a
Yes, messy should be the first and last word for this method.

> 2. run RANCID 'quietly', immediately before and after each router's successful
I think there is too much risk here, defeating a prime benefit of RANCID.

> 3. hack up your own version of do-diffs/control_rancid to perform 2. without
I've got this on my tuit list. As soon as I'm done hacking on a DoS det. app.

> Other approaches?
One could stay off "do-diffs" and just use clogin with your own commands and some
easy shell scripts to parse the data--basically a shortcut to 3.

> Also, how many devices do people have RANCID scaling to? I'm looking at a
Yours it about equal to the largest I've heard of for a full implementation. Personally,
I only use it for a subset of our devices for monitoring changes, mainly due to the
issues you described above. However, for changing snmp strings or local passwords
I'll use it across the board.
--
> -amf

Let me know if you opt on number 3, possibly we could QA each others work or swap some ideas.
My time schedule puts it a few weeks out yet.

--
sig=$header

On Mon, 2 Sep 2002 23:46, jnull wrote:

> I've the same issue, save it is with static route entries for
> secondary IPs. Also, I don't care about saved configs. I asked a similar
> question, but was basically told that the current release doesn't support
> it.

Yep, I remember your earlier post. Like I'd suspect most others, we want to
use the saved configs for long term problem tracking across the whole
network. I imagine you're interested in diffs of specific devices, minus one
or two regularly changing bits. We'd like to ignore all changes made at a
particular time in the day, as we run our major filter update only once
daily.

I'm guessing some NSPs that use RANCID deal with the matter operationally by
having some person review the diffs; expected ACL changes, warts and all, and
then bring/forward out-of-spec stuff to the architects. I'd like a
meat-free approach to this....

> > 2. run RANCID 'quietly', immediately before and after each router's
> > successful
>
> I think there is too much risk here, defeating a prime benefit of
> RANCID.

I think this is only an issue because you don't care about saved
configurations. My thoughts go something along the lines of:

1. run a "regular" do-diffs immediately before you run your nightly routing
maintenance job (that builds router configs and spits em out).

2. run your maintenance, updating router configs.

3. run a "quiet" do-diffs, so that configs are still in CVS, but the usual
aliases aren't mailed with the diff output which will consist of the
maintenance changes (which we dont care to see, but we'd like a record of
incase they fail).

You may have rogue operatives attempting to sneak config changes under your
nose during this quiet diff, but 1. you still have the diffs in your CVS
tree, and 2. you've got bigger problems to deal with if this is happening :).

> > 3. hack up your own version of do-diffs/control_rancid to perform 2.
> > without
>
> I've got this on my tuit list. As soon as I'm done hacking on a DoS
> det. app.

OT: If you publish your work/findings, drop me a line, working with
netflow/etc data on attack analysis was an a challenging and enjoyable part
of my work in my previous life (running a colo/hosting farm similar to
rackspace) and I'm interested in all efforts and research in this area.

> However,
> for changing snmp strings or local passwords I'll use it across the board.

I've been using a combination of scripts to do this, including pancho, but
find rancid *login useful for its cross platform capabilty and scripting
flexibility.

> Let me know if you opt on number 3, possibly we could QA each others work or
> swap some ideas.
> My time schedule puts it a few weeks out yet.

Right now, I'm learning towards a silent diff after the routing update due to
my requirements (i.e., I'm looking filtering alot of stuff out of the diff,
rather than just a little). The difference between exclusive and inclusive
route filtering, I suppose, and likely just as religious an argument :-)

-amf

Mon, Sep 02, 2002 at 10:16:20PM +1000, Andrew Fort:
> An architectural question has come up in a relatively large deployment of
> RANCID I'm working on..
>
> brief: You operate an SP network, and you have software (RtConfig in our case)
> pushing out regular configuration changes to your routers, primarily
> consisting of prefix lists/ACL and route-map/policy-statement changes for
> your BGP routers.
>
> How do you cope with these regular changes operationally with RANCID?
> Architectural staff would like to avoid seeing the ACL/pfx.list changes that
> occur when customer registers a route with your IRR, but you'd still like to
> know when ops folk accidentally smoke some important part of the router
> config.
>
> The thoughts which come to mind for me are:
>
> 1. (messy) munge the rancid output for the filter changes when you do a
> nightly RANCID run after the RtConfig changes, grepping out ACLs and prefix
> lists. you still get a diff with a whole bunch of 'empty' diff section
> headers, however.
>
> 2. run RANCID 'quietly', immediately before and after each router's successful
> run. using 2.2.2, you could create a mail alias that goes to /dev/null and
> do
>
> do-diffs -r box -m nullmail
>
> for each box.
>
> 3. hack up your own version of do-diffs/control_rancid to perform 2. without
> mailing anyone/creating diffs in the first place.
>
> Other approaches?

when someone bitched about nvram changes, i cooked up the following procmail
recipe for them. perhaps you can adapt it for your needs.

:0 H
* ^Subject: .* config diffs
* !^X-RLOOP: rancid
| sed -e '/private-config/n' -e '/\!Flash: nvram:/d' | formail -A "X-RLOOP: rancid" -s procmail

# /dev/null rancid msgs which have no diffs other than nvram:
:0 HWi b
* Subject: .* router config diffs$
| awk 'BEGIN{n=0;}{if(/...Flash: nvram:/)next; if(/^[-+]/){n++;}next;}END{exit n;}'

i like things like these; it allows an individual see what they wish and
preserves the whole of what rancid collects.

> Also, how many devices do people have RANCID scaling to? I'm looking at a
> daily run of around 350 devices, with about 40 BGP routers that need to be
> treated using one of the methods above. Sofar it's scaling very well and the
> overall runtime (~ 1 hour) beats our existing config collection system by
> about 80% (snmp triggered tftp dumps).

i've seen ~750 devices in < 1 hour, almost all bgp speakers.

On Tue, 3 Sep 2002 15:51, john heasley wrote:

> when someone bitched about nvram changes, i cooked up the following
> procmail recipe for them. perhaps you can adapt it for your needs.
>
> i like things like these; it allows an individual see what they wish and
> preserves the whole of what rancid collects.

lovely, this type of approach fits our requirements quite nicely. thanks.
-amf