Mailing List Archive

Watchguard xml file
Hi,

OK, so I can get into the firewall and pull the config with "export config to console". However, the config file is a very large xml file, this one is about 2MB in size. However, it seems like it only recorded the first 388KB of data. Is there a size limit on what rancid can process, or maybe there was a character in the xml that rancid didn't like and it just aborted processing it? How would I go about troubleshooting this?

Thanks,
Wayne


_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Watchguard xml file [ In reply to ]
Sat, Jun 29, 2019 at 11:46:23AM +0000, Wayne Eisenberg:
> Hi,
>
> OK, so I can get into the firewall and pull the config with "export config to console". However, the config file is a very large xml file, this one is about 2MB in size. However, it seems like it only recorded the first 388KB of data. Is there a size limit on what rancid can process, or maybe there was a character in the xml that rancid didn't like and it just aborted processing it? How would I go about troubleshooting this?
>

there is no such limit. I would suspect a PAGER is involved, causing the
output to cease.

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Watchguard xml file [ In reply to ]
If I run the export command manually, it just dumps the whole thing to the screen without any breaks or requests to 'hit space to continue' or things like that, so I don't *think* it's a page length type setting?

Actually, I just did another review and I'm thinking that it has something to do with the prompt definition. Just so we're looking at the same thing, the files are here: https://github.com/hillscott/rancid-watchguard. Forked from https://bitbucket.org/aquerubin/rancid-vyatta.

In the xtmlogin file, it sets the prompt (line 436) to something I don't see. In this original state, xtmlogin never recognized it finished the login. When I changed that line to
set prompt ">>|#"
then xtmlogin completes successfully. (The prompt for this watchguard firewall is "WG#")

-----------
foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"

# device timeout
set timeout [find timeout $router]
if { [llength $timeout] == 0 } {
set timeout $timeoutdflt
}

set prompt ">>"

# Figure out username
if {[info exists username]} {
-----------

However, in the xtm.pm module, line 102 defines it again.
-----------
while (/\s*($cmds_regexp)\s*$/) {
$cmd = $1;
$prompt = ">>";

if (!defined($prompt)) {
$prompt = ($_ =~ /^([^>]+>)/)[0];
$prompt =~ s/([][}{)(\\])/\\$1/g;
print STDERR ("PROMPT MATCH: $prompt\n") if ($debug);
}
print STDERR ("HIT COMMAND:$_") if ($debug);
if (! defined($commands{$cmd})) {
print STDERR "$host: found unexpected command - \"$cmd\"\n";
$clean_run = 0;
last TOP;
}
$rval = &{$commands{$cmd}}($INPUT, $OUTPUT, $cmd);
delete($commands{$cmd});
if ($rval == -1) {
$clean_run = 0;
last TOP;
}
}
-----------
Once you get to the sub ShowConfiguration section, on line 199 if it sees the prompt, end. Guess what? The "#" character is inside the config (there is some html code in one of the xml sections) and that is where the config ends.

-----------
sub ShowConfiguration {
my($INPUT, $OUTPUT, $cmd) = @_;
my($lines) = 0;
my($snmp) = 0;
print STDERR " In ShowConfiguration: $_" if ($debug);
# We don't care about password filtering as passwords are hashed
# So don't use this if you need it (or develop the functionality).
if ($filter_pwds >= 1){
print STDERR "WARNING: Password filtering isn't implemented yet!\n";
print STDERR "Either disable password filtering in rancid.conf";
print STDERR " or don't use this plugin.\n";
}
s/^[a-z]+@//;
ProcessHistory("","","","# $_");
while (<$INPUT>) {
tr/\015//d;
next if (/^\s*$/);
# end of config - hopefully.
# end-of-config tag. appears to end with "\nPROMPT:~$".
if (/$prompt/) {
$found_end++;
last;
}
-----------

So I'm thinking if I can figure out a different way to define the prompt to be more than just the # sign (at least in the xtm.pm), that should do the trick? Can you do something like $prompt = "#$" ?

Wayne



-----Original Message-----
From: john heasley <heas@shrubbery.net>
Sent: Tuesday, July 02, 2019 7:48 PM
To: Wayne Eisenberg <Wayne.Eisenberg@CarolinasIT.com>
Cc: 'rancid-discuss@shrubbery.net' <rancid-discuss@shrubbery.net>
Subject: Re: [rancid] Watchguard xml file

Sat, Jun 29, 2019 at 11:46:23AM +0000, Wayne Eisenberg:
> Hi,
>
> OK, so I can get into the firewall and pull the config with "export config to console". However, the config file is a very large xml file, this one is about 2MB in size. However, it seems like it only recorded the first 388KB of data. Is there a size limit on what rancid can process, or maybe there was a character in the xml that rancid didn't like and it just aborted processing it? How would I go about troubleshooting this?
>

there is no such limit. I would suspect a PAGER is involved, causing the output to cease.


_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Watchguard xml file [ In reply to ]
Wed, Jul 03, 2019 at 04:18:25PM +0000, Wayne Eisenberg:
> If I run the export command manually, it just dumps the whole thing to the screen without any breaks or requests to 'hit space to continue' or things like that, so I don't *think* it's a page length type setting?
>
> Actually, I just did another review and I'm thinking that it has something to do with the prompt definition. Just so we're looking at the same thing, the files are here: https://github.com/hillscott/rancid-watchguard. Forked from https://bitbucket.org/aquerubin/rancid-vyatta.
>
> In the xtmlogin file, it sets the prompt (line 436) to something I don't see. In this original state, xtmlogin never recognized it finished the login. When I changed that line to
> set prompt ">>|#"
> then xtmlogin completes successfully. (The prompt for this watchguard firewall is "WG#")
>

> However, in the xtm.pm module, line 102 defines it again.

i'm not familiar with this device, but redefining (or refining) the
prompt is normal. the filter functions and login scripts begin with
something loose, and once it sees the prompt, it can be refined to be
more precise, and may later further refine it (eg: in run_commands) to
match the prompt when/if it changes in config or other modes that are
platform dependent.

> -----------
> while (/\s*($cmds_regexp)\s*$/) {
> $cmd = $1;
> $prompt = ">>";
^^^^^^^^^^^^ this is probably a mistake; should be part of
the while() regex. I suspect it might be here because the author could
not make the regex below match correctly.

> if (!defined($prompt)) {
> $prompt = ($_ =~ /^([^>]+>)/)[0];
> $prompt =~ s/([][}{)(\\])/\\$1/g;
> print STDERR ("PROMPT MATCH: $prompt\n") if ($debug);
> }
> print STDERR ("HIT COMMAND:$_") if ($debug);
> if (! defined($commands{$cmd})) {
> print STDERR "$host: found unexpected command - \"$cmd\"\n";
> $clean_run = 0;
> last TOP;
> }
> $rval = &{$commands{$cmd}}($INPUT, $OUTPUT, $cmd);
> delete($commands{$cmd});
> if ($rval == -1) {
> $clean_run = 0;
> last TOP;
> }
> }
> -----------
> Once you get to the sub ShowConfiguration section, on line 199 if it sees the prompt, end. Guess what? The "#" character is inside the config (there is some html code in one of the xml sections) and that is where the config ends.

seems that the prompt is ">>".

> -----------
> sub ShowConfiguration {
> my($INPUT, $OUTPUT, $cmd) = @_;
> my($lines) = 0;
> my($snmp) = 0;
> print STDERR " In ShowConfiguration: $_" if ($debug);
> # We don't care about password filtering as passwords are hashed
> # So don't use this if you need it (or develop the functionality).
> if ($filter_pwds >= 1){
> print STDERR "WARNING: Password filtering isn't implemented yet!\n";
> print STDERR "Either disable password filtering in rancid.conf";
> print STDERR " or don't use this plugin.\n";
> }
> s/^[a-z]+@//;
> ProcessHistory("","","","# $_");
> while (<$INPUT>) {
> tr/\015//d;
> next if (/^\s*$/);
> # end of config - hopefully.
> # end-of-config tag. appears to end with "\nPROMPT:~$".
> if (/$prompt/) {
> $found_end++;
> last;
> }
> -----------
>
> So I'm thinking if I can figure out a different way to define the prompt to be more than just the # sign (at least in the xtm.pm), that should do the trick? Can you do something like $prompt = "#$" ?

its better to anchor it and have it be as complete as reasonable. eg:
not #
not hostname#
but ^hostname#

look at ios.pm.
.
> Wayne
>
>
>
> -----Original Message-----
> From: john heasley <heas@shrubbery.net>
> Sent: Tuesday, July 02, 2019 7:48 PM
> To: Wayne Eisenberg <Wayne.Eisenberg@CarolinasIT.com>
> Cc: 'rancid-discuss@shrubbery.net' <rancid-discuss@shrubbery.net>
> Subject: Re: [rancid] Watchguard xml file
>
> Sat, Jun 29, 2019 at 11:46:23AM +0000, Wayne Eisenberg:
> > Hi,
> >
> > OK, so I can get into the firewall and pull the config with "export config to console". However, the config file is a very large xml file, this one is about 2MB in size. However, it seems like it only recorded the first 388KB of data. Is there a size limit on what rancid can process, or maybe there was a character in the xml that rancid didn't like and it just aborted processing it? How would I go about troubleshooting this?
> >
>
> there is no such limit. I would suspect a PAGER is involved, causing the output to cease.
>

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Watchguard xml file [ In reply to ]
-----Original Message-----
From: 'john heasley' <heas@shrubbery.net>
Sent: Wednesday, July 03, 2019 1:41 PM
To: Wayne Eisenberg <Wayne.Eisenberg@CarolinasIT.com>
Cc: 'john heasley' <heas@shrubbery.net>; 'rancid-discuss@shrubbery.net' <rancid-discuss@shrubbery.net>
Subject: Re: [rancid] Watchguard xml file


>> However, in the xtm.pm module, line 102 defines it again.

>i'm not familiar with this device, but redefining (or refining) the prompt is normal. the filter functions and login scripts begin with something loose, and once it sees the prompt, it can be refined to be more precise, and >may later further refine it (eg: in run_commands) to match the prompt when/if it changes in config or other modes that are platform dependent.

Ah, if I only had that skill.

>> -----------
>> while (/\s*($cmds_regexp)\s*$/) {
>> $cmd = $1;
>> $prompt = ">>";
^^^^^^^^^^^^ this is probably a mistake; should be part of the while() regex. I suspect it might be here because the author could not make the regex below match correctly.

>> if (!defined($prompt)) {
>> $prompt = ($_ =~ /^([^>]+>)/)[0];
>> $prompt =~ s/([][}{)(\\])/\\$1/g;
>> print STDERR ("PROMPT MATCH: $prompt\n") if ($debug);
>> }
>> -----------
>> Once you get to the sub ShowConfiguration section, on line 199 if it sees the prompt, end. Guess what? The "#" character is inside the config (there is some html code in one of the xml sections) and that is where the config ends.

>seems that the prompt is ">>".

Yes, in this example. I wanted to show the original file, not something that I modded. In my current version, the line is
$prompt = ">>|#"
which works, but causes the problem of the config getting truncated because it sees "#" as the prompt. The $prompt should either be the entire thing or some string that ends in #.

>> -----------
>> sub ShowConfiguration {
>> my($INPUT, $OUTPUT, $cmd) = @_;
>> my($lines) = 0;
>> my($snmp) = 0;
>> print STDERR " In ShowConfiguration: $_" if ($debug);
>> # We don't care about password filtering as passwords are hashed
>> # So don't use this if you need it (or develop the functionality).
>> if ($filter_pwds >= 1){
>> print STDERR "WARNING: Password filtering isn't implemented yet!\n";
>> print STDERR "Either disable password filtering in rancid.conf";
>> print STDERR " or don't use this plugin.\n";
>> }
>> s/^[a-z]+@//;
>> ProcessHistory("","","","# $_");
>> while (<$INPUT>) {
>> tr/\015//d;
>> next if (/^\s*$/);
>> # end of config - hopefully.
>> # end-of-config tag. appears to end with "\nPROMPT:~$".
>> if (/$prompt/) {
>> $found_end++;
>> last;
>> }
>> -----------
>>
>> So I'm thinking if I can figure out a different way to define the prompt to be more than just the # sign (at least in the xtm.pm), that should do the trick? Can you do something like $prompt = "#$" ?

>its better to anchor it and have it be as complete as reasonable. eg:
>not #
>not hostname#
>but ^hostname#

>look at ios.pm.

Looking, but I don't see anywhere that it defines the prompt. It uses it a lot, but doesn't define it.

Thanks,
Wayne


_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
Re: Watchguard xml file [ In reply to ]
Wed, Jul 03, 2019 at 06:49:20PM +0000, Wayne Eisenberg:
> -----Original Message-----
> From: 'john heasley' <heas@shrubbery.net>
> Sent: Wednesday, July 03, 2019 1:41 PM
> To: Wayne Eisenberg <Wayne.Eisenberg@CarolinasIT.com>
> Cc: 'john heasley' <heas@shrubbery.net>; 'rancid-discuss@shrubbery.net' <rancid-discuss@shrubbery.net>
> Subject: Re: [rancid] Watchguard xml file
>
>
> >> However, in the xtm.pm module, line 102 defines it again.
>
> >i'm not familiar with this device, but redefining (or refining) the prompt is normal. the filter functions and login scripts begin with something loose, and once it sees the prompt, it can be refined to be more precise, and >may later further refine it (eg: in run_commands) to match the prompt when/if it changes in config or other modes that are platform dependent.
>
> Ah, if I only had that skill.
>
> >> -----------
> >> while (/\s*($cmds_regexp)\s*$/) {
> >> $cmd = $1;
> >> $prompt = ">>";
> ^^^^^^^^^^^^ this is probably a mistake; should be part of the while() regex. I suspect it might be here because the author could not make the regex below match correctly.
>
> >> if (!defined($prompt)) {
> >> $prompt = ($_ =~ /^([^>]+>)/)[0];
> >> $prompt =~ s/([][}{)(\\])/\\$1/g;
> >> print STDERR ("PROMPT MATCH: $prompt\n") if ($debug);
> >> }
> >> -----------
> >> Once you get to the sub ShowConfiguration section, on line 199 if it sees the prompt, end. Guess what? The "#" character is inside the config (there is some html code in one of the xml sections) and that is where the config ends.
>
> >seems that the prompt is ">>".
>
> Yes, in this example. I wanted to show the original file, not something that I modded. In my current version, the line is
> $prompt = ">>|#"
> which works, but causes the problem of the config getting truncated because it sees "#" as the prompt. The $prompt should either be the entire thing or some string that ends in #.

yes, this is why it refines the prompt match to be the complete thing, but
it has to see one before it can extract it. and your inloop set is at the
top of the loop, so it never refines it to be the whole prompt.

> >> -----------
> >> sub ShowConfiguration {
> >> my($INPUT, $OUTPUT, $cmd) = @_;
> >> my($lines) = 0;
> >> my($snmp) = 0;
> >> print STDERR " In ShowConfiguration: $_" if ($debug);
> >> # We don't care about password filtering as passwords are hashed
> >> # So don't use this if you need it (or develop the functionality).
> >> if ($filter_pwds >= 1){
> >> print STDERR "WARNING: Password filtering isn't implemented yet!\n";
> >> print STDERR "Either disable password filtering in rancid.conf";
> >> print STDERR " or don't use this plugin.\n";
> >> }
> >> s/^[a-z]+@//;
> >> ProcessHistory("","","","# $_");
> >> while (<$INPUT>) {
> >> tr/\015//d;
> >> next if (/^\s*$/);
> >> # end of config - hopefully.
> >> # end-of-config tag. appears to end with "\nPROMPT:~$".
> >> if (/$prompt/) {
> >> $found_end++;
> >> last;
> >> }
> >> -----------
> >>
> >> So I'm thinking if I can figure out a different way to define the prompt to be more than just the # sign (at least in the xtm.pm), that should do the trick? Can you do something like $prompt = "#$" ?

it has to be as a set (regex or glob), like; [#$]. but that is a single
atom; if your prompt is or may be ">>", then you likely need to use a
group atom, like (>>|#).

> >its better to anchor it and have it be as complete as reasonable. eg:
> >not #
> >not hostname#
> >but ^hostname#
>
> >look at ios.pm.
>
> Looking, but I don't see anywhere that it defines the prompt. It uses it a lot, but doesn't define it.

its starts with [>#] in the while() (and exit match); then refines it to be
a match the entire prompt with regex atoms escaped in the
if(!defined($prompt)). after that, it anchors the prompt match when
appropiate; /^$prompt/.

you should do similarly for this watchguard device. I suspect that you can
just steal the ios.pm inloop() and modify the initial prompt matching. It
could be kinkier, but it is a good starting point.

i think i;ve answered everything.

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss@shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss