Mailing List Archive

I think the schema will be the sticking point for most people towards
gaining concensus. What I would push for is a generic format that is
mostly a CSV that get stuck directly into a SQL database. From there,
it would be pretty easy to mangle it around into whatever a specific
user wants. At the very least, I think this would be a good starting
point for working out the database issues and getting us close fast.
After that, we could have a second output format (sqlbasic and sqladv)
that had the "advanced" features that was normalized and more geared
towards writing new frontends for. Let's face it, there's a lot of us
that could write frontends or integrate nessus into existing projects if
there was just a simple way to output SQL.

Joe

On Mon, 2003-01-13 at 09:16, Javier Fernandez-Sanguino wrote:
> First of all, please excuse me if this is a beaten horse and developers
> (Renaud?) already have an idea on how this is going to evolve and
> wether or not it is intented for 1.3 (it is in the ROADMAP_TO_1_2.txt file)
>
> I have been finding my way through the nessus (user's and developers)
> mailing list for information on this issue. It seems there are
> currently, at least, three tools that can use a database as backend for
> Nessus: Inprotect's [1] NessQuick [2] and Nessus-php [3]. With one
> patch [4] provided a long time ago to provide this into the Nessus
> server itself and changes in the NessusWx client [5] to provide this in
> there (rather than in server side).
>
> I am going to volunteer to try to provide something that functionality
> in the near term for Nessus. However, I think it's proper to discuss first:
>
> - the schema to be used for the database (none of the mentioned
> proposals have a common schema, however, having a common definition
> would allow Nessus to provide one for people who want to write
> PHP/whatever other language frontends that way users could jump from one
> to the other without much problems)
>
> - the location of the backend: it could be either the server (which
> could use the database for session/knowledgebase saving/recoverty), the
> client (through an 'export to SQL' dialog) and/or a separate tool (which
> would process session/XML/NBE/NSR output and insert it into the database)
>
> Now, if no one objects or says that this is already being implemented by
> someone I will send out a proposal for schema based on what I've looked
> of the different implementations, my opinion and my reviewal of what
> other tools seem to do (aka Internet Scanner).
>
> Anyone?
>
> Javi
>
> PS: I'm not going to give my arguments on why this is necessary, since I
> believe everybody agrees it is. Feel free to discuss it, in any case.
>
>
> [1]
> http://www.inprotect.com/modules.php?op=modload&name=Downloads&file=index&req=viewdownload&cid=1
> [2] www.atriskonline/projects/nessQuick.html
> [3] http://enterprise.bidmc.harvard.edu/pub/nessus-php/
> [4] http://msgs.securepoint.com/cgi-bin/get/nessus-0101/53.html
> [5] http://www.securityprojects.org/nessuswx/sql_tables.html

I think having a database backend will help further development of many
Nessus addons and tools.

> First of all, please excuse me if this is a beaten horse and developers
> (Renaud?) already have an idea on how this is going to evolve and
> wether or not it is intented for 1.3 (it is in the ROADMAP_TO_1_2.txt
> file)

It'll be nice to know what main Nessus developers think about it.

> I am going to volunteer to try to provide something that functionality
> in the near term for Nessus. However, I think it's proper to discuss
> first:

Let me know if you'll need any help.
I'm very interested in the backend/frontend development too.

> - the location of the backend: it could be either the server (which
> could use the database for session/knowledgebase saving/recoverty), the
> client (through an 'export to SQL' dialog) and/or a separate tool (which
> would process session/XML/NBE/NSR output and insert it into the
> database)

I agree, if it'll be a standard Nessus backend database, lot's of things
have to be defined.

Alex Zimin
alex@inprotect.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Please use standard SQL92 or later rather than a specific sql server.
It's a lot easier to write the intermediary abstraction interface than
it is to [re]write every sql() function for each new DB that is supported :)

And just for the record, I somewhat strongly oppose using Berkely DB.
They seem to love rewriting APIs with every release.

- -d

- --
I may have the information you need and I may choose only HTML. It's up to you. Disclaimer: I am not responsible for any email that you send me nor am I bound to any obligation to deal with any received email in any given fashion. If you send me spam or a virus, I may in whole or part send you 50,000 return copies of it. I may also publically announce any and all emails and post them to message boards, news sites, and even parody sites. I may also mark them up, cut and paste, print, and staple them to telephone poles for the enjoyment of people without internet access. This is not a confidential medium and your assumption that your email can or will be handled confidentially is akin to baring your backside, burying your head in the ground, and thinking nobody can see you butt nekkid and in plain view for miles away. Don't be a cluebert, buy one from K-mart today.

When it absolutely, positively, has to be destroyed overnight.
AIR FORCE

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE+I3Xz74cGT/9uvgsRAp6dAKCs5BWyAP9d5SIRUUl78lirvGri0ACg2GU9
k16mW9A1p/hYsdlbzHXgjYQ=
=eX8k
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I love PostgreSQL :)

I don't want to start a zealot war so I'm not going to compare them.

David

Alex Zimin wrote:

>I think MySQL or Postgres would be best choices.
>These databases should be SQL92 compliant, free and very popular among
>Internet community.
>
>Alex.
>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE+I30C74cGT/9uvgsRAiHdAJ92Mvo4YfAEogDOOSPhGXcAxmT9pwCfSoz5
fnKNoL9VnR9uf4Pogrv5dPc=
=huQf
-----END PGP SIGNATURE-----

I think MySQL or Postgres would be best choices.
These databases should be SQL92 compliant, free and very popular among
Internet community.

Alex.


> Please use standard SQL92 or later rather than a specific sql server.
> It's a lot easier to write the intermediary abstraction interface than
> it is to [re]write every sql() function for each new DB that is
> supported :)
>
> And just for the record, I somewhat strongly oppose using Berkely DB.
> They seem to love rewriting APIs with every release.
>
> - -d

> -----Original Message-----
> From: Michael Boman [mailto:michael.boman@securecirt.com]
> Sent: Monday, January 13, 2003 9:55 PM
> To: nessus-devel@list.nessus.org
> Subject: Re: Using a database as backend for Nessus (current state and
> volunteering for development)
>
>
> On Mon, Jan 13, 2003 at 09:29:04PM -0500, David Ford wrote:
> > Please use standard SQL92 or later rather than a specific sql server.
>
> > It's a lot easier to write the intermediary abstraction interface than
>
> > it is to [re]write every sql() function for each new DB that is
> supported :)
> >
> > And just for the record, I somewhat strongly oppose using Berkely DB.
>
> > They seem to love rewriting APIs with every release.
> >
> > - -d
>
> What about just writing the INSERT statments to a textfile the user can
> use a (provided*) shell script to upload it to the database. I think
> this
> is much better then supplying username/password/host/db to the nessus
> process. If it is done in this way the machine that runs nessus and the
> machine that runs the upload script doesn't need to be the same box,
> as the report.sql (or what you want to name it) could be transfered
> to another machine in the terms of shared network filesystem, scp,
> ftp <choose your poison>.
>
> I think the DB scheme should be modeled up after the newer XML output
> plugin as AFAIK it contain more data then any other format - and data
> is good to have because you shouldn't assume that everyone wants to do
> the report in the same way..
>
> I'll take a look at the XML format and start sketchign on a DB scheme.
>
> Best regards
> Michael Boman
>
> (*)
> For extra security measures it could make sure that it ONLY contains
> INSERT statements, so that a hacker on a compromised nessus box
> doesn't
> do anything in the style of 'drop database mysql' (example only). I
> know
> for an fact that there are people that are giving out root access to
> normal applications, either of laziness or just be un-informed about
> the penalties.

One thing to think about here is that a database backend should be a
datastore for the server portion so that say, you could have an interrupted
scan restart with 100% integrity, or you could have multiple scanners
working against the same database and not step on each other's toes etc...

After the backend is a database, we get normalization, comparative reporting
etc...

I think reading the data into a database after the fact is

On Monday 13 January 2003 20:30, Alex Zimin wrote:

> > I am going to volunteer to try to provide something that functionality
> > in the near term for Nessus. However, I think it's proper to discuss
> > first:
>
> Let me know if you'll need any help.
> I'm very interested in the backend/frontend development too.
I too would like to contribute, though I confess my C/C++ is weak. I do have
experience writing apps that use databases on the backend, however, and have
have some thoughts to SQL DB design for a Nessus backend database.

> > - the location of the backend: it could be either the server (which
> > could use the database for session/knowledgebase saving/recoverty), the
> > client (through an 'export to SQL' dialog) and/or a separate tool (which
> > would process session/XML/NBE/NSR output and insert it into the
> > database)
I for one would favor the sever-side database. NessusWX already has a
client-side MySQL database defined and implemented. It may be worth
consulting what they've done.

I use Nessus on a farily regular basis for work, and have often wanted an SQL
database backend to draw results from. I would very much like to contribute
in anyway I can.


Eric I. Arnoth, CISSP (http://www.isc2.org)
earnoth@comcast.net
http://mywebpages.comcast.net/earnoth/index.html
¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø
The radical of one century is the conservative of the next. The radical
invents the views. When he has worn them out, the conservative adopts them.
- Mark Twain, 1898.

On Mon, 2003-01-13 at 10:16, Javier Fernandez-Sanguino wrote:
> First of all, please excuse me if this is a beaten horse and developers
> (Renaud?) already have an idea on how this is going to evolve and
> wether or not it is intented for 1.3 (it is in the ROADMAP_TO_1_2.txt file)

> [..snip..]



> I am going to volunteer to try to provide something that functionality
> in the near term for Nessus. However, I think it's proper to discuss first:
>

> - the schema to be used for the database (none of the mentioned
> proposals have a common schema, however, having a common definition
> would allow Nessus to provide one for people who want to write
> PHP/whatever other language frontends that way users could jump from one
> to the other without much problems)
>
Maybe its time we start playing with developing a schema to call the
standard. I have attached the sql that creates(in Postgres) the table
structure that I am using to place XML output into Postgres. The
structure is fairly standard, the only "non-standard" sql type that I
have used is the serial type. This should be fairly portable to any
other DBMS out there. Also i'm not sold on my "naming convention".



> - the location of the backend: it could be either the server (which
> could use the database for session/knowledgebase saving/recoverty), the
> client (through an 'export to SQL' dialog) and/or a separate tool (which
> would process session/XML/NBE/NSR output and insert it into the database)
>
I have a python app that I am using to perform the dumps. This lacks the
sex appeal of dumping to a database from the server while scans are
running but it provides our team the neccessary functionality.


> Now, if no one objects or says that this is already being implemented by
> someone I will send out a proposal for schema based on what I've looked
> of the different implementations, my opinion and my reviewal of what
> other tools seem to do (aka Internet Scanner).

>
> Anyone?
>
> Javi
>
> PS: I'm not going to give my arguments on why this is necessary, since I
> believe everybody agrees it is. Feel free to discuss it, in any case.
>
>
> [1]
> http://www.inprotect.com/modules.php?op=modload&name=Downloads&file=index&req=viewdownload&cid=1
> [2] www.atriskonline/projects/nessQuick.html
> [3] http://enterprise.bidmc.harvard.edu/pub/nessus-php/
> [4] http://msgs.securepoint.com/cgi-bin/get/nessus-0101/53.html
> [5] http://www.securityprojects.org/nessuswx/sql_tables.html
--
Matt Woodyard <matt@mattwoodyard.com>

On Mon, Jan 13, 2003 at 09:29:04PM -0500, David Ford wrote:
> Please use standard SQL92 or later rather than a specific sql server.
> It's a lot easier to write the intermediary abstraction interface than
> it is to [re]write every sql() function for each new DB that is supported :)
>
> And just for the record, I somewhat strongly oppose using Berkely DB.
> They seem to love rewriting APIs with every release.
>
> - -d

What about just writing the INSERT statments to a textfile the user can
use a (provided*) shell script to upload it to the database. I think this
is much better then supplying username/password/host/db to the nessus
process. If it is done in this way the machine that runs nessus and the
machine that runs the upload script doesn't need to be the same box,
as the report.sql (or what you want to name it) could be transfered
to another machine in the terms of shared network filesystem, scp,
ftp <choose your poison>.

I think the DB scheme should be modeled up after the newer XML output
plugin as AFAIK it contain more data then any other format - and data
is good to have because you shouldn't assume that everyone wants to do
the report in the same way..

I'll take a look at the XML format and start sketchign on a DB scheme.

Best regards
Michael Boman

(*)
For extra security measures it could make sure that it ONLY contains
INSERT statements, so that a hacker on a compromised nessus box doesn't
do anything in the style of 'drop database mysql' (example only). I know
for an fact that there are people that are giving out root access to
normal applications, either of laziness or just be un-informed about
the penalties.

--
Michael Boman
Security Architect, SecureCiRT (A SBU of Z-Vance Pte Ltd)
http://www.securecirt.com

Currently, I am adding some functionality to the 1.2.7 release of Nessus:
.New Reports - which will be customizeable (limited) if the Summary/Risk
level/Fix is separated and the vulnerability/risk levels, consolitated (or
I get around to parsing through the current description)
.Timestamping
.More host info - IP/MAC/DNS
.New outputs - PDF? Better HTML Graphs
.Plugin reports - plugins run
+more...

Where is this going? Well, since I have already familiarized myself with
the Nessus source, I would be glad to assist the effort. I am not very
familiar with DB, as most of my experience is in C/C++. However I could
easily customize the output of the client or save the server information
into any necessary format.

I think it is best to output the report as a module that can be exported
to a DB. It would be faster to create the module in Nessus rather than
parsing an output report, but I do NOT want to create any dependencies
within Nessus that are not absolutely necessary.

----------------------
William Heinbockel
Information Security Incident Response Assistant
Co-op Risk & Safety Management
Rochester Institute of Technology
E-mail: wjh3710@rit.edu

On Mon, 13 Jan 2003, Eric Arnoth wrote:

> On Monday 13 January 2003 20:30, Alex Zimin wrote:
>
> > > I am going to volunteer to try to provide something that functionality
> > > in the near term for Nessus. However, I think it's proper to discuss
> > > first:
> >
> > Let me know if you'll need any help.
> > I'm very interested in the backend/frontend development too.
> I too would like to contribute, though I confess my C/C++ is weak. I do have
> experience writing apps that use databases on the backend, however, and have
> have some thoughts to SQL DB design for a Nessus backend database.
>
> > > - the location of the backend: it could be either the server (which
> > > could use the database for session/knowledgebase saving/recoverty), the
> > > client (through an 'export to SQL' dialog) and/or a separate tool (which
> > > would process session/XML/NBE/NSR output and insert it into the
> > > database)
> I for one would favor the sever-side database. NessusWX already has a
> client-side MySQL database defined and implemented. It may be worth
> consulting what they've done.
>
> I use Nessus on a farily regular basis for work, and have often wanted an SQL
> database backend to draw results from. I would very much like to contribute
> in anyway I can.
>
>
> Eric I. Arnoth, CISSP (http://www.isc2.org)
> earnoth@comcast.net
> http://mywebpages.comcast.net/earnoth/index.html
> ¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø
> The radical of one century is the conservative of the next. The radical
> invents the views. When he has worn them out, the conservative adopts them.
> - Mark Twain, 1898.
>
>

On Mon, Jan 13, 2003 at 04:16:37PM +0100, Javier Fernandez-Sanguino wrote:
> I am going to volunteer to try to provide something that functionality
> in the near term for Nessus.

Thanks !

> - the schema to be used for the database (none of the mentioned
> proposals have a common schema, however, having a common definition
> would allow Nessus to provide one for people who want to write
> PHP/whatever other language frontends that way users could jump from one
> to the other without much problems)

You'll have to take a decision - everyone has its own idea about what
the best schema is (see the archives of the list). I've never tried
NessusWX's SQL output, but it would make sense to have the same.


> - the location of the backend: it could be either the server (which
> could use the database for session/knowledgebase saving/recoverty), the
> client (through an 'export to SQL' dialog) and/or a separate tool (which
> would process session/XML/NBE/NSR output and insert it into the database)

I think it would make more sense to have it on the server side. This
way, one could start queries before the scan even finishes, which is
definitely interesting.

The most interesting hooks are probably in
nessus-core/nessusd/save_kb.c. As the KB contains the text of the report
sent to the client (SentData/<pluginid>=ReportText) it would make sense
to save it there. Plus, saving the KB saves additional data about the
scan, which would allow to perform SQL queries to perform extremely fast
scans :

ie: Let's say a new worm is hitting Solaris's UUCP, then you could
select all the potentially vulnerable hosts by doing :


SELECT host_ip_address FROM some_table WHERE kb_entry = 'Host/OS' AND
kb_value = '%Solaris%'
SELECT host_ip_address FROM some_table WHERE kb_entry = 'Port/TCP/540'
AND kb_value != '0'

Then the intersection of the two subsets would give the list of
potentially vulnerable hosts.

(mhhh, that's looks awkward, but I have no nice schema in mind, and I'm
not an SQL expert either...)


So, have a look at nessus-core/nessusd/save_kb.c. Rewriting all the
functions to add SQL support (#ifdef/#endif is probably the best). You
definitely want to modify :

save_kb_fname()
save_kb_write()
save_kb_new()
save_kb_close()
save_kb_exists()
save_kb_restore_backup()
save_kb_backup()
save_kb_load_kb()



You'll want to add a small parser in save_kb_write() to specially
process the plugins output (SentData/<NUM>), but the whole thing should
hopefully be nearly painless. Let me know what I can do to help.


-- Renaud

Renaud Deraison wrote:

>On Mon, Jan 13, 2003 at 04:16:37PM +0100, Javier Fernandez-Sanguino wrote:
>
(...)

>>- the schema to be used for the database (none of the mentioned
>>proposals have a common schema, however, having a common definition
>>would allow Nessus to provide one for people who want to write
>>PHP/whatever other language frontends that way users could jump from one
>>to the other without much problems)
>>
>>
>
>You'll have to take a decision - everyone has its own idea about what
>the best schema is (see the archives of the list). I've never tried
>NessusWX's SQL output, but it would make sense to have the same.
>
>
Yes, I know. I belive that the discussion around the schema issue might
bring some new ideas on which kind of stuff the people would like to do
with the database (what kind of reporting, trend analysis or whatever).
My current schema draft includes some of the proposals brought in the
list (and will submit as soon as our e-mail server behaves properly).

The problem with NessusWX SQL output is that information is not
separated in such a way that it would make it easy to do trend analysis.
Also, the result information is redundant with the information already
available in the plugins (includes CVE id, description, risk). However,
I intend to use some of the good stuff (the false-positive boolean
should be pretty useful to remove vulnerabilities from reports).

>>- the location of the backend: it could be either the server (which
>>could use the database for session/knowledgebase saving/recoverty), the
>>client (through an 'export to SQL' dialog) and/or a separate tool (which
>>would process session/XML/NBE/NSR output and insert it into the database)
>>
>>
>
>I think it would make more sense to have it on the server side. This
>way, one could start queries before the scan even finishes, which is
>definitely interesting.
>
I agree for this in the long term. However, for prototype testing and to
allow other kind of database management I think it would be pretty
useful to have a generic NBE/XML parser that would translate the reports
into database INSERT/UPDATES. That way people who have been using Nessus
for a while could load all their reports to a database (which at least I
have to do) before starting using the SQL functionalities.

One of the problems with doing it in the server side (i.e. having the
server populate the database as it goes) is that database support is
probably going to become a compilation issue. I.e. if SQL calls are
going to be done while the nessus server does it work it's going to need
proper calls for
MySQL/PostgresSQL/Oracle/whatever_you_want_as_a_database inlined in the
code. Also, it would be necessary to add information to the Nessus
configuration file so that admins can provide the appropiate database
information:

- location of the database server
- user/password
- database to use in the server

Which, obviously, should not be part of the code itself. Otherwise it
would be difficult to provide compiled packages with database server
support.

>The most interesting hooks are probably in
>nessus-core/nessusd/save_kb.c. As the KB contains the text of the report
>sent to the client (SentData/<pluginid>=ReportText) it would make sense
>to save it there. Plus, saving the KB saves additional data about the
>scan, which would allow to perform SQL queries to perform extremely fast
>scans :
>
>
I'll have to think on how to include the knowledgebase into the SQL
database too. That's something I had not though of and it's definitely
worth a shot. However, there is information provided by the plugins
which is not available in the KB and that should be included too.
(...)

>So, have a look at nessus-core/nessusd/save_kb.c. Rewriting all the
>functions to add SQL support (#ifdef/#endif is probably the best). You
>definitely want to modify :
>
(...)

Will do.

>You'll want to add a small parser in save_kb_write() to specially
>process the plugins output (SentData/<NUM>), but the whole thing should
>hopefully be nearly painless. Let me know what I can do to help.
>
>
>
Ok. Thanks.

Javi

PS: Reanud, you did not include the nessus-extract patch I submited a
while back. Did you get it? I didn't want to file a bug report since for
this, after all, it's not part of the nessus distribution :)

Joe Francis wrote:

>I think the schema will be the sticking point for most people towards
>gaining concensus. What I would push for is a generic format that is
>mostly a CSV that get stuck directly into a SQL database.
>
Sorry I don't agree with you here. You can get this kind of CSV output
just by using the NSR/NBE reports. Just:

$ cat nessus-report.nbe | sed -e 's/^/"/; s/$/"/; s/|/","/g'

That's not a database.

From there,
>it would be pretty easy to mangle it around into whatever a specific
>user wants. At the very least, I think this would be a good starting
>point for working out the database issues and getting us close fast.
>
IMHO if Nessus does not provide an effective database everyone will use
their own hand-made instead of resorting to it.

>After that, we could have a second output format (sqlbasic and sqladv)
>that had the "advanced" features that was normalized and more geared
>towards writing new frontends for. Let's face it, there's a lot of us
>that could write frontends or integrate nessus into existing projects if
>there was just a simple way to output SQL.
>
>
Let's face it: knowledgeable users will make their own databases,
unknowledgeable users will not. It's not a matter of outputting SQL. SQL
does not make any sense if the schema is not useful. If we can provide a
schema we can determine which SQL to output to populate the database.
Without a layout of what tables/information to populate the SQL output
does not make much sense. Does it?

Regards

Javi

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>
>
>I think it would make more sense to have it on the server side. This
>way, one could start queries before the scan even finishes, which is
>definitely interesting.
>

- -Very- much agreed. One of the best selling points of "some other"
scanner is the real time report information.

>SELECT host_ip_address FROM some_table WHERE kb_entry = 'Host/OS' AND
>kb_value = '%Solaris%'
>SELECT host_ip_address FROM some_table WHERE kb_entry = 'Port/TCP/540'
>AND kb_value != '0'
>

May I suggest that the better form of doing this in a RDBMS is to split
of that information so it's not 'Port/TCP/540', but the TCP and 540 are
split. Leverage the power of RDBMS to our advantage. Same goes for all
the KB and other information. Each component should be in it's own
column to better use the power of the DB. Otherwise we're just
attempting to use the DB as a networked file system.

>Then the intersection of the two subsets would give the list of
>potentially vulnerable hosts.
>
>(mhhh, that's looks awkward, but I have no nice schema in mind, and I'm
>not an SQL expert either...)
>

Time will help us derive schemas and queries that are powerful and
present the best information.

David

- --
I may have the information you need and I may choose only HTML. It's up to you. Disclaimer: I am not responsible for any email that you send me nor am I bound to any obligation to deal with any received email in any given fashion. If you send me spam or a virus, I may in whole or part send you 50,000 return copies of it. I may also publically announce any and all emails and post them to message boards, news sites, and even parody sites. I may also mark them up, cut and paste, print, and staple them to telephone poles for the enjoyment of people without internet access. This is not a confidential medium and your assumption that your email can or will be handled confidentially is akin to baring your backside, burying your head in the ground, and thinking nobody can see you butt nekkid and in plain view for miles away. Don't be a cluebert, buy one from K-mart today.

When it absolutely, positively, has to be destroyed overnight.
AIR FORCE

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE+JEha74cGT/9uvgsRAmrRAJ0SOy4b2kw6NeWiMaiwQaTReWs9LwCgt65K
QU0o/lOqKuxvUNWEa7CSGas=
=YCyA
-----END PGP SIGNATURE-----

Not to post a "me too" post, but really I like this idea. I, for one,
have systems running nessusd that cannot connect to our database
directly, the only way is that I download the reports via ssh and then
use a script on the main database to import them all. Poking another
firewall hole to send mysql through or forwarding ports via ssh is also
not an option for us.

Joe

On Mon, 2003-01-13 at 21:54, Michael Boman wrote:
> On Mon, Jan 13, 2003 at 09:29:04PM -0500, David Ford wrote:
> > Please use standard SQL92 or later rather than a specific sql server.
> > It's a lot easier to write the intermediary abstraction interface than
> > it is to [re]write every sql() function for each new DB that is supported :)
> >
> > And just for the record, I somewhat strongly oppose using Berkely DB.
> > They seem to love rewriting APIs with every release.
> >
> > - -d
>
> What about just writing the INSERT statments to a textfile the user can
> use a (provided*) shell script to upload it to the database. I think this
> is much better then supplying username/password/host/db to the nessus
> process. If it is done in this way the machine that runs nessus and the
> machine that runs the upload script doesn't need to be the same box,
> as the report.sql (or what you want to name it) could be transfered
> to another machine in the terms of shared network filesystem, scp,
> ftp <choose your poison>.
>
> I think the DB scheme should be modeled up after the newer XML output
> plugin as AFAIK it contain more data then any other format - and data
> is good to have because you shouldn't assume that everyone wants to do
> the report in the same way..
>
> I'll take a look at the XML format and start sketchign on a DB scheme.
>
> Best regards
> Michael Boman
>
> (*)
> For extra security measures it could make sure that it ONLY contains
> INSERT statements, so that a hacker on a compromised nessus box doesn't
> do anything in the style of 'drop database mysql' (example only). I know
> for an fact that there are people that are giving out root access to
> normal applications, either of laziness or just be un-informed about
> the penalties.
>
> --
> Michael Boman
> Security Architect, SecureCiRT (A SBU of Z-Vance Pte Ltd)
> http://www.securecirt.com

On Tue, Jan 14, 2003 at 10:55:33AM +0100, Javier Fernandez-Sanguino wrote:
> >I think it would make more sense to have it on the server side. This
> >way, one could start queries before the scan even finishes, which is
> >definitely interesting.
> >
> I agree for this in the long term. However, for prototype testing and to
> allow other kind of database management I think it would be pretty
> useful to have a generic NBE/XML parser that would translate the reports
> into database INSERT/UPDATES.

I think this leads us in the wrong path - .nsr reports are nowhere as
complete as what is on the server side and what can be filled in
real time (list of plugins selected, time of the day, and so on...)


[...]
> One of the problems with doing it in the server side (i.e. having the
> server populate the database as it goes) is that database support is
> probably going to become a compilation issue. I.e. if SQL calls are
> going to be done while the nessus server does it work it's going to need
> proper calls for
> MySQL/PostgresSQL/Oracle/whatever_you_want_as_a_database inlined in the
> code.

autoconf is here for this reason. You can easily create
nessus-core/nessusd/db_{mysql,oracle,postgres}.c and have them compile
if and only if the relevant libraries are installed on the system.
The difficulty will be to create some kind of clean abstraction layer
for all of these, but we're not doing anything too complicated either,
so that should not be a problem.


> Also, it would be necessary to add information to the Nessus
> configuration file so that admins can provide the appropiate database
> information:
[....]
> Which, obviously, should not be part of the code itself. Otherwise it
> would be difficult to provide compiled packages with database server
> support.

nessusd has a very extandable preference system. Most of the code is
here already - just add the options to the nessusd.conf file, and you
can read them within nessusd with no change at all, through
calls to arg_get_value(). This really is not the worse part.

[...]
> However, there is information provided by the plugins
> which is not available in the KB and that should be included too.

I don't think so. Once you're in nessusd and know that plugin#1234 is
issuing an alert, you can have all the data you want (but CVE/BIDs are
the only thing we'd be missing, and those should be indexed in a
"plugin" table).

> PS: Reanud, you did not include the nessus-extract patch I submited a
> while back. Did you get it? I didn't want to file a bug report since for
> this, after all, it's not part of the nessus distribution :)


Please fill a bug report marked as an "enhancement". This allows (and
everyone) to keep track of it, and bugzilla hammers me automatically if
I don't treat it with due care.


-- Renaud

While a db back-end is a great idea, keep in mind one
_very_ important design issue. If you have a db, and
you issue a new release of Nessus, you don't have nearly
the same flexibility changing the db design as you do
changing nessus functionality.

People will have lots of scans in the db, and if you go
and change the schema, you're now forced to do a db conversion.
It's not impossible. Not even terribly difficult. But it
certainly can be prone to problems.

You'll need to do things like marking the schema, and
when a new release of Nessus tries to access an old schema
oriented db, deny the transactions and indicate a conversion
needs to be done.

Not only that, but you'll be faced with possibly multiple
conversions. At minimum, each schema change will require
a conversion script. At worst, you'll need to accomodate
several different types of conversions as people who skip
releases of several different schema changes want to bump
themselves up to the latest and greatest without necessarily
going through the conversion from all the previous releases
(unless you can make that relatively painless for them).

There's also the minor issue of permissions - e.g. with
postgres, afaik, one cannot be root and access the db
(or at least its HIGHLY discouraged). Since Nessus needs
to run as root, you have a minor problem now that you need
to solve if the server is the one accessing the DB. From
that perspective, it may be worthwhile to completely
separate out the DB management actions from the Nessus
daemon.

Cheers, Thomas

----------------------
William Heinbockel
Information Security Incident Response Assistant
Co-op Risk & Safety Management
Rochester Institute of Technology
E-mail: wjh3710@rit.edu

On Tue, 14 Jan 2003, Thomas Reinke wrote:

> While a db back-end is a great idea, keep in mind one
> _very_ important design issue. If you have a db, and
> you issue a new release of Nessus, you don't have nearly
> the same flexibility changing the db design as you do
> changing nessus functionality.

I have thoroughly examined a majority of the Nessus code.
Renaud has done an excellent job of make Nessus very flexible,
while still maintaining functionality.
The only case that I would believe your statement to hold true is
if Renaud et al. would do a rewrite of the entire Nessus code.
Currently, the way Nessus is set-up I could rewrite NTP and change some
of the backend and structs, and as long as the DB code was integrated into
the code (nessus/nessusd or both), instead of a hack on top of it, you
shouldn't have any major problems (if any at all) with porting it along
with the Nessus updates.

>
> People will have lots of scans in the db, and if you go
> and change the schema, you're now forced to do a db conversion.
> It's not impossible. Not even terribly difficult. But it
> certainly can be prone to problems.

This will exist for any database. This is being designed as
a tool to assist users/admins. I'm sure many avid Nessus users
have already created a script to parse through Nessus reports.
If the database was implemented as an extension of nessusd (the server),
you could easily dump the information. Nessus stores a lot of info,
but only some of it gets transfered to the client for the reports :)

>
> You'll need to do things like marking the schema, and
> when a new release of Nessus tries to access an old schema
> oriented db, deny the transactions and indicate a conversion
> needs to be done.

I don't think that needs addressed at this moment.
Porting old databases to new databases is just the task of translating
the old protocol to the new. If we make the DB complete the first time,
there should be no need to convert it to a new schema.

>
> Not only that, but you'll be faced with possibly multiple
> conversions. At minimum, each schema change will require
> a conversion script. At worst, you'll need to accomodate
> several different types of conversions as people who skip
> releases of several different schema changes want to bump
> themselves up to the latest and greatest without necessarily
> going through the conversion from all the previous releases
> (unless you can make that relatively painless for them).
>
> There's also the minor issue of permissions - e.g. with
> postgres, afaik, one cannot be root and access the db
> (or at least its HIGHLY discouraged). Since Nessus needs
> to run as root, you have a minor problem now that you need
> to solve if the server is the one accessing the DB. From
> that perspective, it may be worthwhile to completely
> separate out the DB management actions from the Nessus
> daemon.
>
> Cheers, Thomas
>
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>
>
>>There's also the minor issue of permissions - e.g. with
>>postgres, afaik, one cannot be root and access the db
>>(or at least its HIGHLY discouraged). Since Nessus needs
>>to run as root, you have a minor problem now that you need
>>to solve if the server is the one accessing the DB. From
>>that perspective, it may be worthwhile to completely
>>separate out the DB management actions from the Nessus
>>daemon.
>>
>>Cheers, Thomas
>>
>>

Eh. Not so :)

You can acces the database as any user at all. UNIX user does not equal
postgres user. The highly discouraged "root" user is the use of the
postgres user to addess databases since the postgres user (is the
default owner of/has access to) everything from files on disks to data
within the DB. The use of this user (postgres user) is highly discouraged.

root# psql -U some_postgres_username <some command>

Root can connect to postgres all you like. Simply specify the postgres
username to do operations as. This is what is important.

David

- --
I may have the information you need and I may choose only HTML. It's up to you. Disclaimer: I am not responsible for any email that you send me nor am I bound to any obligation to deal with any received email in any given fashion. If you send me spam or a virus, I may in whole or part send you 50,000 return copies of it. I may also publically announce any and all emails and post them to message boards, news sites, and even parody sites. I may also mark them up, cut and paste, print, and staple them to telephone poles for the enjoyment of people without internet access. This is not a confidential medium and your assumption that your email can or will be handled confidentially is akin to baring your backside, burying your head in the ground, and thinking nobody can see you butt nekkid and in plain view for miles away. Don't be a cluebert, buy one from K-mart today.

When it absolutely, positively, has to be destroyed overnight.
AIR FORCE

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE+JOVU74cGT/9uvgsRAlusAJsEcrF/441DpVmIPwm2/o0+8C/LDgCgzs1+
oYYBfsGGDD1Zw8L7ZAnL59c=
=OBcu
-----END PGP SIGNATURE-----

> While a db back-end is a great idea, keep in mind one
> _very_ important design issue. If you have a db, and
> you issue a new release of Nessus, you don't have nearly
> the same flexibility changing the db design as you do
> changing nessus functionality.
>
> People will have lots of scans in the db, and if you go
> and change the schema, you're now forced to do a db conversion.
> It's not impossible. Not even terribly difficult. But it
> certainly can be prone to problems.

Having a standard Nessus backend database will bring lots of benefits to
the Nessus users, which in my opinion will overweight concerns about
possible schema or data migration down the road.

Alex Zimin
alex@inprotect.com

> I fear you have been the victim of FUD. "root" means nothing to
> postgres data other than the fact that "root" can do pretty much what
> root wants on a filesystem.
>
> Postgres does not use unix username/password mapping thus "root" is not
> root within postgres. The postgres database owner is the all powerful
> user..again, inside the database. There is no correlation to
> /etc/password and the pg_users table unless you distinctly and
> intentionally create it.

Nah...am aware of the difference between postgres superuser and
real root. But obviously I seem to remember things wrong. I
seemed to have recalled that I was unable to do a "createuser root"
as postgres user to set up root for db access. But when I tried
it now, it worked like a charm. So obviously I had my facts
wrong on that one.

Let's stop _talking_ and get down to action, shall we? I have not yet
taken a look at the code itself, but this is a quick draft based on what
XML_NG can do, with what people (might) want to be able to do..:

This is psudo code!!!

TABLE hosts (
scan_id, # A auto incrimental value for each scan we do
hostname, # name of host
ip, # address of host
scan_start, # when did we start scanning this host?
scan_end, # when did we end scanning this host?
nessus_version, # What version of nessus was used?
request_user, # Who requested the scan (username), good if
# you want to limit what a user can see
request_host, # Where did this user login from? For audit purpose.. IP number here.
);

TABLE reports (
scan_id, # Refer to hosts.scan_id
plugin_start, # When did we start this scan?
plugin_end, # When did this plugin end?
plugin_id, # Plugin number
plugin_version, # (is there such a thing? if not, something
# for NASL2?)
port_number, # What port is it we scanned?
port_protocol, # and what protocol is it using?
serverity, # Number refering to serverity.serverity_id
completed, # did the plugin actually finish or did nessusd
# kill it?

#ifndef If people want to store plugin output in several languages
lang # I don't know how relivant this is, but some
# ppl might want to store what lang the message
# is written in. Do we want to store several languages at once?
data # plugin output
#endif

)

#ifdef If people want to store plugin output in several languages
TABLE data (
scan_id # refer to hosts.scan_id
lang # language
data # plugin output
)
#endif

TABLE services (
port_number, # port number
port_protocol, # protocol
service_name # human readable name
)

TABLE serverity (
serverity_id # Number
serverity_name # Human readable serverity
)

TABLE references (
plugin_id, # Plugin ID
refrence_type, # reference_types.reference_type value
reference_data # CVE-1999-0191 (example)
)

TABLE reference_types (
reference_type # Autoinc value
reference_name # CVE, CAN, BID and so on...
reference_url # http://cve.mitre.org/cgi-bin/cvename.cgi?name=
# (example). The base URL to get to the page of
# interest
)

--
Michael Boman
Security Architect, SecureCiRT (A SBU of Z-Vance Pte Ltd)
http://www.securecirt.com

Thomas Reinke wrote:

>
>
>> I fear you have been the victim of FUD. "root" means nothing to
>> postgres data other than the fact that "root" can do pretty much what
>> root wants on a filesystem.
>>
>> Postgres does not use unix username/password mapping thus "root" is
>> not root within postgres. The postgres database owner is the all
>> powerful user..again, inside the database. There is no correlation
>> to /etc/password and the pg_users table unless you distinctly and
>> intentionally create it.
>
>
> Nah...am aware of the difference between postgres superuser and
> real root. But obviously I seem to remember things wrong. I
> seemed to have recalled that I was unable to do a "createuser root"
> as postgres user to set up root for db access. But when I tried
> it now, it worked like a charm. So obviously I had my facts
> wrong on that one.
>
That's not an issue. Regardless of what user nessusd run as you can
still connect to the database using any other user. If the proper
configuration (user+location of the database+password) can be done
through nessus.conf it's up to the admin/user to make user that the
database will allow the authentication of the user Nessus claim to be
(in Postgres, for example, you have different authentication mechanisms
and you can configure, as needed, either a 'trust users in this host'
or a 'require password to access this database', it's pretty flexible).

Regards

Javi

Michael Boman wrote:

>Let's stop _talking_ and get down to action, shall we? I have not yet
>taken a look at the code itself, but this is a quick draft based on what
>XML_NG can do, with what people (might) want to be able to do..:
>
>This is psudo code!!!
>
>
>
Ok. Before going in to the information that is introduced into each of
the tables... why not talk about designing the overall schema? How about
this one (adjointed, in raw form). Main idea is :

1- users run vulnerability sessions (it's important to keep track of who
made what).
2.- sessions run plugins against a set of hosts (information available
should include: start time/end time, session configuration: specified
plugins, parameters...)
3.- vulnerabilities are associated to a given ExecutedPluginId (_not_ a
plugin, see below) and might, or might not, discover information
associated with it (which is kept in the plugin table, not separated).
4.- services are also detected by specific port-scan plugins (but might
not be vulnerable in hosts in a given session).

The idea is that some administrators might want to keep track of how
sessions have been executed: which plugins have been executed (the
ExecutedPlugin entity has to hold also the version of the plugin), which
hosts have been detected and which services are deemed vulnerable. This
is why it's useful to separate plugin information (which BTW you can
extract with 'nessus -qSp' or with the script I sent to the list), you
don't need to duplicate plugin descriptions whenever a vulnerability is
discovered. This also permits users to provide new translations for
plugins by modifying the plugin table directly (or by introducing
multiple language description cells)

Also, a given plugin might detect more than one vulnerability and these
might/might not include information notes (banners et al). Separating
also service discovery from other vulnerability information is done
because, IMHO, this is one of the most important (and homogeneus)
information a given plugin (portscanners) will provide. It's also useful
to do an inventory of open services in the network.

Some other things you could do with this schema:

- keep track of which plugins (and specific revisions/versions) were
executed against a given host. This allows admins to automatically rerun
them whenever new plugins are added or a plugin has been updated. Also,
since plugins are continously being added to Nessus it might be
appropiate to differentiate when a vulnerability has been discovered
when it was not previously there (but tested) and when it was discovered
because a new plugin was added (or modified) that detected it.
- keep track of vulnerabilities associated with detected services.
- separate detected services (which might not be viewed as vulnerable
since a plugin might not exploit them at the time of the run) from
vulnerabilities. This allows for reports based on "seen services" in
different sessions. Something on the lines of: 'which new services (open
ports) have been detected on system X since the last run'.

Note that reports are generated _based_ on the information in the
database. As such, they do not really need to be stored in the database
itself (IMHO)

Regards

Javi


PS: I might have missed some of the discussion due to a mail server
problem. Sorry.