Mailing List Archive

Cisco devices and calculating percent free
I recently updated to v3.13, I do not recall exactly what version I
was on but it was >1 year old, and could've been HEAD based on some
previous debugging of other issues.

Anyhow, we have many IOS devices that calculate percent free and it oscillates:

- !Flash: bootflash: 1 GB total (19% free)
+ !Flash: bootflash: 1 GB total (18% free)

Without manually patching ios.pm (or wherever this takes place) is
there any type of config setting to just turn off showing any info
about how much disk space is free? We get this info properly from snmp
trending systems.

Just trying to cut down on router diffs where there were no actual
config changes.

Cheers,
Chris

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
something analogous to

--- rancid.types.base.ORIG 2020-03-19 01:11:40.000000000 +0000
+++ rancid.types.base 2020-03-21 19:41:36.522622000 +0000
@@ -221,7 +221,7 @@
cisco-nx;command;nxos::ShowEnvPower;show environment power
cisco-nx;command;nxos::ShowBoot;show boot
cisco-nx;command;nxos::DirSlotN;dir bootflash:
-cisco-nx;command;nxos::DirSlotN;dir debug:
+## cisco-nx;command;nxos::DirSlotN;dir debug:
cisco-nx;command;nxos::DirSlotN;dir logflash:
cisco-nx;command;nxos::DirSlotN;dir slot0:
cisco-nx;command;nxos::DirSlotN;dir usb1:
@@ -598,7 +598,7 @@
junos;module;junos
junos;inloop;junos::inloop
junos;command;junos::ShowChassisClocks;show chassis clocks
-junos;command;junos::ShowChassisEnvironment;show chassis environment
+## junos;command;junos::ShowChassisEnvironment;show chassis environment
junos;command;junos::ShowChassisFirmware;show chassis firmware local
junos;command;junos::ShowChassisFirmware;show chassis firmware

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
Thu, Nov 12, 2020 at 07:19:36AM -0600, Chris Wopat:
> I recently updated to v3.13, I do not recall exactly what version I
> was on but it was >1 year old, and could've been HEAD based on some
> previous debugging of other issues.
>
> Anyhow, we have many IOS devices that calculate percent free and it oscillates:
>
> - !Flash: bootflash: 1 GB total (19% free)
> + !Flash: bootflash: 1 GB total (18% free)
>
> Without manually patching ios.pm (or wherever this takes place) is
> there any type of config setting to just turn off showing any info
> about how much disk space is free? We get this info properly from snmp
> trending systems.
>
> Just trying to cut down on router diffs where there were no actual
> config changes.

Except that something to writing to your flash...maybe you expect that.

anyway, private devices types without the dir commands could be
created in rancid.types.conf(5).

or, i have almost finished an option for a user-defined per-device type
postscript filter. eg:

+# example use of the postscript directive. This will replace control
+# characters in the output with an ASCII representation.
+arbor-ctrl;script;rancid -t arbor-ctrl
+arbor-ctrl;login;anlogin
+arbor-ctrl;module;arbor
+arbor-ctrl;inloop;arbor::inloop
+arbor-ctrl;postscript;/bin/cat -v
+arbor-ctrl;command;arbor::ShowVersion;system version
+arbor-ctrl;command;arbor::ShowVersion;system file show
+arbor-ctrl;command;arbor::ShowHardware;system hardware
+arbor-ctrl;command;arbor::ShowHardware;system show
+arbor-ctrl;command;arbor::ShowFlash;system disk show
+arbor-ctrl;command;arbor::ShowConfig;config show

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
On Thu, Nov 12, 2020 at 12:17 PM Randy Bush <randy@psg.com> wrote:
>
> something analogous to
>
> --- rancid.types.base.ORIG 2020-03-19 01:11:40.000000000 +0000
> +++ rancid.types.base 2020-03-21 19:41:36.522622000 +0000
> @@ -221,7 +221,7 @@
> cisco-nx;command;nxos::ShowEnvPower;show environment power
> cisco-nx;command;nxos::ShowBoot;show boot
> cisco-nx;command;nxos::DirSlotN;dir bootflash:
> -cisco-nx;command;nxos::DirSlotN;dir debug:
> +## cisco-nx;command;nxos::DirSlotN;dir debug:
> cisco-nx;command;nxos::DirSlotN;dir logflash:
> cisco-nx;command;nxos::DirSlotN;dir slot0:
> cisco-nx;command;nxos::DirSlotN;dir usb1:
> @@ -598,7 +598,7 @@
> junos;module;junos
> junos;inloop;junos::inloop
> junos;command;junos::ShowChassisClocks;show chassis clocks
> -junos;command;junos::ShowChassisEnvironment;show chassis environment
> +## junos;command;junos::ShowChassisEnvironment;show chassis environment
> junos;command;junos::ShowChassisFirmware;show chassis firmware local
> junos;command;junos::ShowChassisFirmware;show chassis firmware

I had not yet tested this out, but it looks like it may remove the
full directory listing from whichever disc it's churning on.

I should have noted that I do have value in knowing the contents of
flash within RANCID, just not the amount of free space, which I get
via SNMP.

I'll test above or heasley's suggestion when it comes out to confirm.
Hell maybe I'll even spend a few minutes checking the cause of the
issue (ie what's happening to fluctuate disk utilization)

Cheers,
--Chris

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
If I recall, end users should alter rancid.types.conf, not .base - or better yet, add a custom device type to rancid.types.conf and leave the defaults alone.
weylin

?-----Original Message-----
From: Randy Bush <randy@psg.com>
Date: Thursday, November 12, 2020 at 1:17 PM
To: Chris Wopat <me@falz.net>
Cc: <rancid-discuss@www.shrubbery.net>
Subject: Re: [rancid] Cisco devices and calculating percent free

something analogous to

--- rancid.types.base.ORIG 2020-03-19 01:11:40.000000000 +0000
+++ rancid.types.base 2020-03-21 19:41:36.522622000 +0000
@@ -221,7 +221,7 @@
cisco-nx;command;nxos::ShowEnvPower;show environment power
cisco-nx;command;nxos::ShowBoot;show boot
cisco-nx;command;nxos::DirSlotN;dir bootflash:
-cisco-nx;command;nxos::DirSlotN;dir debug:
+## cisco-nx;command;nxos::DirSlotN;dir debug:
cisco-nx;command;nxos::DirSlotN;dir logflash:
cisco-nx;command;nxos::DirSlotN;dir slot0:
cisco-nx;command;nxos::DirSlotN;dir usb1:
@@ -598,7 +598,7 @@
junos;module;junos
junos;inloop;junos::inloop
junos;command;junos::ShowChassisClocks;show chassis clocks
-junos;command;junos::ShowChassisEnvironment;show chassis environment
+## junos;command;junos::ShowChassisEnvironment;show chassis environment
junos;command;junos::ShowChassisFirmware;show chassis firmware local
junos;command;junos::ShowChassisFirmware;show chassis firmware



_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
I’m expecting the postscript filter because there more and more files in flash. I need per-device per-tag postscript filter.

from Affa Pan

> john heasley <heas@shrubbery.net> ? 2020?11?13? ??2:32 ???
>
> ?Thu, Nov 12, 2020 at 07:19:36AM -0600, Chris Wopat:
>> I recently updated to v3.13, I do not recall exactly what version I
>> was on but it was >1 year old, and could've been HEAD based on some
>> previous debugging of other issues.
>>
>> Anyhow, we have many IOS devices that calculate percent free and it oscillates:
>>
>> - !Flash: bootflash: 1 GB total (19% free)
>> + !Flash: bootflash: 1 GB total (18% free)
>>
>> Without manually patching ios.pm (or wherever this takes place) is
>> there any type of config setting to just turn off showing any info
>> about how much disk space is free? We get this info properly from snmp
>> trending systems.
>>
>> Just trying to cut down on router diffs where there were no actual
>> config changes.
>
> Except that something to writing to your flash...maybe you expect that.
>
> anyway, private devices types without the dir commands could be
> created in rancid.types.conf(5).
>
> or, i have almost finished an option for a user-defined per-device type
> postscript filter. eg:
>
> +# example use of the postscript directive. This will replace control
> +# characters in the output with an ASCII representation.
> +arbor-ctrl;script;rancid -t arbor-ctrl
> +arbor-ctrl;login;anlogin
> +arbor-ctrl;module;arbor
> +arbor-ctrl;inloop;arbor::inloop
> +arbor-ctrl;postscript;/bin/cat -v
> +arbor-ctrl;command;arbor::ShowVersion;system version
> +arbor-ctrl;command;arbor::ShowVersion;system file show
> +arbor-ctrl;command;arbor::ShowHardware;system hardware
> +arbor-ctrl;command;arbor::ShowHardware;system show
> +arbor-ctrl;command;arbor::ShowFlash;system disk show
> +arbor-ctrl;command;arbor::ShowConfig;config show
>
> _______________________________________________
> Rancid-discuss mailing list
> Rancid-discuss@www.shrubbery.net
> https://www.shrubbery.net/mailman/listinfo/rancid-discuss
_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
Looked into this a little further.

The disk space churn is being caused by this device type - Cisco
ASR920, with churn of whatever in bootflash:/tracelogs . IOS-XE seems
to be doing its own thing in there as files tend to be <1 day old
while no device changes are happening. The contents of this directory
are properly ignored and aren't in diffs, but this slight change in
disk space is causing the issue.

If I do comment this from rancid.types.conf or rancid.types.base, as
expected it removes the directory contents, which does contain
valuable info I like to have (ie which IOS images are chilling there):

#cisco;command;ios::DirSlotN;dir /all bootflash:

So, the only way for me to currently workaround this is to make a
change directly to rancid.pm, in "sub diskszsummary", such as, but not
limited to:

#return("$total total ($pcnt% free)");
return("$total total");

Certainly dirty but works. Curious if there's any thoughts on a
'clean' way to do this? I know there's an existing 'FILTER_OSC' flag
that I believe is intended to reduce churn in general, my best guess
for a 'clean' fix, if it makes sense to people, would be to check that
here and only return "$pcnt% free" if that isn't set.

My $.02

If people are opposed to such a thing, I'll likely add this to my
'manually patch these couple of things' list that already exists
post-upgrades.

Cheers
--Chris

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
> ASR920, with churn of whatever in bootflash:/tracelogs

I've been through the ringer on this one (about this time last year, in fact).
If you dig even deeper, it seems to be attached to the cylon_mgr process, which presumably ties to the chipset.
In our environment, busier boxes have greater churn (in the 100's of MBytes/day).
Our patch around this (pre RANCID v3.10) was not to ignore the disk space, but rather impose an upper ceiling for our ASR920's.
Eg;
- !Flash: bootflash: 1339412480 bytes total (470 MB free)
+ !Flash: bootflash: 1339412480 bytes total (>300 MB free)

This way, we still get insight into run-away processes that suddenly burn through the box' flash:, and it adequately stops the noise.
Since upgrading 3.10, it has been less frequent, so we never re-patched.

But can confirm, out of all our devices, it's only the ASR920's that exhibit this behavior in our environment.
ASR901's, 1000's seem fine.

Food for further thought.

-----Original Message-----
From: Rancid-discuss [mailto:rancid-discuss-bounces@www.shrubbery.net] On Behalf Of Chris Wopat
Sent: Friday, November 13, 2020 6:31 AM
To: Randy Bush
Cc: rancid-discuss@www.shrubbery.net
Subject: Re: [rancid] Cisco devices and calculating percent free

Looked into this a little further.

The disk space churn is being caused by this device type - Cisco
ASR920, with churn of whatever in bootflash:/tracelogs . IOS-XE seems
to be doing its own thing in there as files tend to be <1 day old
while no device changes are happening. The contents of this directory
are properly ignored and aren't in diffs, but this slight change in
disk space is causing the issue.

If I do comment this from rancid.types.conf or rancid.types.base, as
expected it removes the directory contents, which does contain
valuable info I like to have (ie which IOS images are chilling there):

#cisco;command;ios::DirSlotN;dir /all bootflash:

So, the only way for me to currently workaround this is to make a
change directly to rancid.pm, in "sub diskszsummary", such as, but not
limited to:

#return("$total total ($pcnt% free)");
return("$total total");

Certainly dirty but works. Curious if there's any thoughts on a
'clean' way to do this? I know there's an existing 'FILTER_OSC' flag
that I believe is intended to reduce churn in general, my best guess
for a 'clean' fix, if it makes sense to people, would be to check that
here and only return "$pcnt% free" if that isn't set.

My $.02

If people are opposed to such a thing, I'll likely add this to my
'manually patch these couple of things' list that already exists
post-upgrades.

Cheers
--Chris

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
On Fri, Nov 13, 2020 at 8:31 AM Chris Wopat <me@falz.net> wrote:
>
> So, the only way for me to currently workaround this is to make a
> change directly to rancid.pm, in "sub diskszsummary", such as, but not
> limited to:
>
> #return("$total total ($pcnt% free)");
> return("$total total");
>
> Certainly dirty but works. Curious if there's any thoughts on a
> 'clean' way to do this? I know there's an existing 'FILTER_OSC' flag
> that I believe is intended to reduce churn in general, my best guess
> for a 'clean' fix, if it makes sense to people, would be to check that
> here and only return "$pcnt% free" if that isn't set.

We've been running this way for a bit and it's solved our churn issue.
I'll just manually patch this during upgrade for now.

If a similar fix does get included in the FILTER_OSC variable, then
all the better but no biggie.

--Chris

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
Fri, Nov 13, 2020 at 04:27:02PM +0000, Emille Blanc:
> Our patch around this (pre RANCID v3.10) was not to ignore the disk space, but rather impose an upper ceiling for our ASR920's.
> Eg;
> - !Flash: bootflash: 1339412480 bytes total (470 MB free)
> + !Flash: bootflash: 1339412480 bytes total (>300 MB free)
>
> This way, we still get insight into run-away processes that suddenly burn through the box' flash:, and it adequately stops the noise.
> Since upgrading 3.10, it has been less frequent, so we never re-patched.
>
> But can confirm, out of all our devices, it's only the ASR920's that exhibit this behavior in our environment.
> ASR901's, 1000's seem fine.

Still seems like it should not do that; that it is something that mistakingly
escaped DE.

anyway, i'm open to your solution or more suggestions. The ios script
could do this just for the asr900 series, but even then I am not sure that
one size fits all.

Prost

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Cisco devices and calculating percent free [ In reply to ]
Fri, Nov 13, 2020 at 01:03:54PM +0000, Pan Affa:
> I’m expecting the postscript filter because there more and more files in flash. I need per-device per-tag postscript filter.
>
> from Affa Pan

> > or, i have almost finished an option for a user-defined per-device type
> > postscript filter. eg:

You can try ftp://ftp.shrubbery.net/pub/rancid/alpha/rancid-3.99.99.tar.gz

I've done minimal testing. This will become 4.0 and does require
python >= 3.5, which is not exactly used yet and use that at your own risk.
If you do not deviate from the rancid 3.x features + this postscript feature,
there should be no ill effect.

> > +# example use of the postscript directive. This will replace control
> > +# characters in the output with an ASCII representation.
> > +arbor-ctrl;script;rancid -t arbor-ctrl
> > +arbor-ctrl;login;anlogin
> > +arbor-ctrl;module;arbor
> > +arbor-ctrl;inloop;arbor::inloop
> > +arbor-ctrl;postscript;/bin/cat -v
> > +arbor-ctrl;command;arbor::ShowVersion;system version
> > +arbor-ctrl;command;arbor::ShowVersion;system file show
> > +arbor-ctrl;command;arbor::ShowHardware;system hardware
> > +arbor-ctrl;command;arbor::ShowHardware;system show
> > +arbor-ctrl;command;arbor::ShowFlash;system disk show
> > +arbor-ctrl;command;arbor::ShowConfig;config show

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@www.shrubbery.net
https://www.shrubbery.net/mailman/listinfo/rancid-discuss