Mailing List Archive

I totally agree. Anything that will help make the parsing easier would
be appreciated.

Bill Petersen, CISSP
Senior Information Security Analyst
Alcatel North America Information Security
Bill.Petersen@alcatel.com
Voice: 972-519-4249
Fax: 972-477-5300



Matt Van Mater wrote:

>I was updating a nbe file parser I wrote in perl and noticed a few
>plugins do not output the various fields in the 'normal' order. For
>example, there are some plugins where the "Risk factor" is printed
>before the "Solution" (nasl #10399), when in most other plugin output
>the solution is printed before the risk factor. I can give specific
>examples of other plugins that exhibit similar behavior, and can work
>with Renaud/whoever to do that as a 'short term' fix
>
>However, as a long term fix, I'd love to see a truly well formed way
>of parsing all possible fields from a NBE file. It could be as simple
>as extending the pipe | delimiter to the nasl output fields as well,
>or something more complicated.
>
>I realize that this might result in output lines with empty fields
>that look like:
>results|W.X.Y|W.X.Y.Z|rdp (3389/tcp)||||||||||
>or something more verbose like
>results|W.X.Y|W.X.Y.Z|rdp (3389/tcp)|Synopsis: |Description: |foo: |bar: |baz:
>
>But I personally am very willing to eat a little extra disk space in
>my report files as a tradeoff for having a strictly ordered, well
>formed nasl output. While my parser can handle a few of the
>inconsistent output formats, it won't be able to handle all of them
>without getting ugly, and might not work properly if/when new nasls
>are released.
>
>I admit that i'm not the worlds best expert on parsing/regexps but I
>do think I'm fairly good and I find parsing NBE cumbersome due to
>these inconsistencies. Am I totally off base here?
>
>Matt
>_______________________________________________
>Nessus-devel mailing list
>Nessus-devel@list.nessus.org
>http://mail.nessus.org/mailman/listinfo/nessus-devel
>
>

On Wed, Dec 07, 2005 at 01:13:30PM -0500, Matt Van Mater wrote:

> I was updating a nbe file parser I wrote in perl and noticed a few
> plugins do not output the various fields in the 'normal' order. For
> example, there are some plugins where the "Risk factor" is printed
> before the "Solution" (nasl #10399), when in most other plugin output
> the solution is printed before the risk factor.

What about iterating over the description and pulling out key/value
pairs matching, like this:

# $line is a line of input from an NBE file.
@fields = split(/\|/, $line);
if ($fields[0] eq 'results') {
$desc = $fields[6];
$keypat = qr/(Synopsis|Description|See also|Solution|Risk
factor|Plugin output|BID|CVE|Other references)/i;

while ($desc =~ /($keypat)\s*:\s*(.+?)(?=($keypat|$))/gm) {
print $1, " = ", $3, "\n\n";
}
}

[.NB: Perl mongers can probably suggest another alternative involving
some module from CPAN.] You'll probably need to tweak $keypat somewhat
more as the descriptions are, as you note, currently unstandardized.

> However, as a long term fix, I'd love to see a truly well formed way
> of parsing all possible fields from a NBE file. It could be as simple
> as extending the pipe | delimiter to the nasl output fields as well,
> or something more complicated.

We've been using a standard format for all new plugins since October or
so; eg, take a look at mailenable_imap_rename_dos.nasl. This will almost
certainly, though, be a long-term rather than short-term solution given
the number of existing plugins that must be revised.


George
--
theall@tenablesecurity.com
_______________________________________________
Nessus-devel mailing list
Nessus-devel@list.nessus.org
http://mail.nessus.org/mailman/listinfo/nessus-devel

>What about iterating over the description and pulling out key/value
>pairs matching, like this:

> # $line is a line of input from an NBE file.
> @fields = split(/\|/, $line);
> if ($fields[0] eq 'results') {
> $desc = $fields[6];
> $keypat = qr/(Synopsis|Description|See also|Solution|Risk
>factor|Plugin output|BID|CVE|Other references)/i;

> while ($desc =~ /($keypat)\s*:\s*(.+?)(?=($keypat|$))/gm) {
> print $1, " = ", $3, "\n\n";
> }
> }

I think you meant to write:
print $1, " = ", $2, "\n\n";

But that aside, this is very similar to what I was thinking about
changing my script to but I think it would make your parsing routine a
somewhat slower.

Also, my end goal is to store each matched pattern to a specific
variable (ie the synopsis test is stored in $synopsis, etc) which I
will then use to insert the value into a database. This means foreach
$keypat matched I will have to do a 'lookup' on $1 to see what
corresponding variable I want to store the $2 match in. While this is
certainly doable, it would require two nested while loops to find a
match and then store it appropriately. Of course if any new fields
are added to the $desc body, your time spent parsing will grow
exponentially.

In the real world with our wonderful high powered machines this
shouldn't be a big problem, but I think it's clear that a static,
unchanging format would be both simpler to understand and somewhat
faster to parse.

>[.NB: Perl mongers can probably suggest another alternative involving
>some module from CPAN.] You'll probably need to tweak $keypat somewhat
>more as the descriptions are, as you note, currently unstandardized.

I checked out some of the CPAN modules, and in my very short overview
it didn't look like they had the granular level of parsing that we're
talking about.

>We've been using a standard format for all new plugins since October or
>so; eg, take a look at mailenable_imap_rename_dos.nasl. This will almost
>certainly, though, be a long-term rather than short-term solution given
>the number of existing plugins that must be revised.

I agree. I was trying to think of a writing a script to parse the
nasl files and find the ones that have "non standard" output format.
No matter how you look at it, I think they will be somewhat unpleasant
to fix and may require hands on attention to fix. I suppose you could
make the script find the output irregularities and then actually
automatically rewrite the nasl
in the correct format, but that doesn't sound like a whole lot of fun
to me and may not be feasible.

>George

Thanks for your comments. I think I may rewrite my parser along the
lines of what you suggested above and then compare the performance for
sheer geekiness. It might be interesting.


Matt Van Mater
_______________________________________________
Nessus-devel mailing list
Nessus-devel@list.nessus.org
http://mail.nessus.org/mailman/listinfo/nessus-devel

On Fri, Dec 09, 2005 at 01:04:46PM -0500, Matt Van Mater wrote:

> I think you meant to write:
> print $1, " = ", $2, "\n\n";

I bet you didn't try it that way - it needs to be $3 to account for the
parens in $keypat.

> But that aside, this is very similar to what I was thinking about
> changing my script to but I think it would make your parsing routine a
> somewhat slower.

Short-term, the descriptions are unlikely to be cleaned up. So you're
faced with making mods to your parsing routine to handle the "messy"
data. Yes, that's going to be slower, but I suspect it's likely to be
one that at least works.

> I agree. I was trying to think of a writing a script to parse the
> nasl files and find the ones that have "non standard" output format.
> No matter how you look at it, I think they will be somewhat unpleasant
> to fix and may require hands on attention to fix.

Perhaps "improve" would be a better word than "fix"... After all, the
description's are not "broken", are they?

> I suppose you could
> make the script find the output irregularities and then actually
> automatically rewrite the nasl
> in the correct format, but that doesn't sound like a whole lot of fun
> to me and may not be feasible.

Given how non-standard the descriptions currently are, I don't think an
automated solution would be a good idea.


George
--
theall@tenablesecurity.com
_______________________________________________
Nessus-devel mailing list
Nessus-devel@list.nessus.org
http://mail.nessus.org/mailman/listinfo/nessus-devel