Mailing List Archive

I am interested in contributing whatever I can to this. There are
probably going to be some tough decisions to be made in the process
of moving nessus over to a database backend.

I'll snag a copy of your code and start looking through it tonight.
I think Renaud actually gave Javi some hints in a previous email.
I need to go back and look for that.

-----Original Message-----
From: William Heinbockel [mailto:wjh3710@osfmail.isc.rit.edu]
Sent: Thursday, February 06, 2003 12:23 PM
To: nessus-devel@list.nessus.org
Subject: mySQL database for nessusd (code!)


Just an update on the database backend for nessusd...

I've started coding a mysql interface/library for nessusd.
It is written in C and uses mysql.h (the mysql C API).

The code can be found at:
<a href="http://www.rit.edu/~wjh3710/pub/save_mysql.c">save_mysql.c</a>
and
<a href="http://www.rit.edu/~wjh3710/pub/save_mysql.h">save_mysql.h</a>

This is by no means complete. I will try to update it as much as
possible. But if others want to assist with the development, it
might be best to set up a CVS server (anyone interested?).

I'm working with the database schema that Javi posted awhile back,
but due to my lack of SQL knowledge and the way nessusd is set-up,
I don't think that some of the tables can be setup in that manner.

I think the easiest way to incorporate the non-session specific data
(i.e. the Nessus-plugin table and Services table and the like...) is
to write a simple [perl?] script to create an sql modules directly
from the plugins' directory and hard code a script to deal with the
services.

----------------------
William Heinbockel
Information Security Incident Response Assistant
Co-op Risk & Safety Management
Rochester Institute of Technology
E-mail: wjh3710@rit.edu

On Thu, 6 Feb 2003, Minderhout, Doug wrote:

> I am interested in contributing whatever I can to this. There are
> probably going to be some tough decisions to be made in the process
> of moving nessus over to a database backend.
>
> I'll snag a copy of your code and start looking through it tonight.
> I think Renaud actually gave Javi some hints in a previous email.
> I need to go back and look for that.
>

Currently, I'm developing just a library of commands to be integrated into
nessusd. I just started learning mySQL on Tues ;) so I'm not familiar
with it in any way. Also, the aim of the project is not to replace the
existing datastructures in Nessus. It is being developed in addition to
so that large amounts of scan data can be easily stored and admin can
easily draw out information and perform trend analysis.

Renaud's suggestions from before where more target to use the database
in the same fashion as the knowledge base - to store the RESULT of the
scans. Javi and myself agree, it would be better to get realtime
information from Nessus.

The code that I have is relatively simplistic... it pretty much
translates method calls into mySQL commands. Right now, I am figuring
out if the information in the database is easily obtainable, and
where the best place would be to save the information to the database.


> -----Original Message-----
> From: William Heinbockel [mailto:wjh3710@osfmail.isc.rit.edu]
> Sent: Thursday, February 06, 2003 12:23 PM
> To: nessus-devel@list.nessus.org
> Subject: mySQL database for nessusd (code!)
>
>
> Just an update on the database backend for nessusd...
>
> I've started coding a mysql interface/library for nessusd.
> It is written in C and uses mysql.h (the mysql C API).
>
> The code can be found at:
> save_mysql.c
> and
> save_mysql.h
>
> This is by no means complete. I will try to update it as much as
> possible. But if others want to assist with the development, it
> might be best to set up a CVS server (anyone interested?).
>
> I'm working with the database schema that Javi posted awhile back,
> but due to my lack of SQL knowledge and the way nessusd is set-up,
> I don't think that some of the tables can be setup in that manner.
>
> I think the easiest way to incorporate the non-session specific data
> (i.e. the Nessus-plugin table and Services table and the like...) is
> to write a simple [perl?] script to create an sql modules directly
> from the plugins' directory and hard code a script to deal with the
> services.
>
> ----------------------
> William Heinbockel
> Information Security Incident Response Assistant
> Co-op Risk & Safety Management
> Rochester Institute of Technology
> E-mail: wjh3710@rit.edu
>

----------------------
William Heinbockel
Information Security Incident Response Assistant
Co-op Risk & Safety Management
Rochester Institute of Technology
E-mail: wjh3710@rit.edu

Could you post to the list which database scheme you're using? I got lost in
all the threads, my inbox is overflowing from neglect because of the time I
spent on the SQL worm last week, so I'm not sure what the final cut was.

I'm wondering if both PostgreSQL and MySQL could be supported. Perhaps
abstracting the database transaction layer? I apologize if this is digging
up something that was already settled.

Also, has any decision been made with regards to using transactions? Since a
single scanner might be running multiple scans for multiple clients, it might
be a good idea to make use of.

--
Eric I. Arnoth CISSP (http://www.isc2.org)
earnoth@comcast.net
http://mywebpages.comcast.net/earnoth
¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø¤ø,¸¸,ø¤º°*°º¤ø,¸¸,ø
On Thursday 06 February 2003 15:21, William Heinbockel wrote:
> On Thu, 6 Feb 2003, Minderhout, Doug wrote:
> > I am interested in contributing whatever I can to this. There are
> > probably going to be some tough decisions to be made in the process
> > of moving nessus over to a database backend.
> >
> > I'll snag a copy of your code and start looking through it tonight.
> > I think Renaud actually gave Javi some hints in a previous email.
> > I need to go back and look for that.
>
> Currently, I'm developing just a library of commands to be integrated into
> nessusd. I just started learning mySQL on Tues ;) so I'm not familiar
> with it in any way. Also, the aim of the project is not to replace the
> existing datastructures in Nessus. It is being developed in addition to
> so that large amounts of scan data can be easily stored and admin can
> easily draw out information and perform trend analysis.
>
> Renaud's suggestions from before where more target to use the database
> in the same fashion as the knowledge base - to store the RESULT of the
> scans. Javi and myself agree, it would be better to get realtime
> information from Nessus.
>
> The code that I have is relatively simplistic... it pretty much
> translates method calls into mySQL commands. Right now, I am figuring
> out if the information in the database is easily obtainable, and
> where the best place would be to save the information to the database.
>
> > -----Original Message-----
> > From: William Heinbockel [mailto:wjh3710@osfmail.isc.rit.edu]
> > Sent: Thursday, February 06, 2003 12:23 PM
> > To: nessus-devel@list.nessus.org
> > Subject: mySQL database for nessusd (code!)
> >
> >
> > Just an update on the database backend for nessusd...
> >
> > I've started coding a mysql interface/library for nessusd.
> > It is written in C and uses mysql.h (the mysql C API).
> >
> > The code can be found at:
> > save_mysql.c
> > and
> > save_mysql.h
> >
> > This is by no means complete. I will try to update it as much as
> > possible. But if others want to assist with the development, it
> > might be best to set up a CVS server (anyone interested?).
> >
> > I'm working with the database schema that Javi posted awhile back,
> > but due to my lack of SQL knowledge and the way nessusd is set-up,
> > I don't think that some of the tables can be setup in that manner.
> >
> > I think the easiest way to incorporate the non-session specific data
> > (i.e. the Nessus-plugin table and Services table and the like...) is
> > to write a simple [perl?] script to create an sql modules directly
> > from the plugins' directory and hard code a script to deal with the
> > services.
> >
> > ----------------------
> > William Heinbockel
> > Information Security Incident Response Assistant
> > Co-op Risk & Safety Management
> > Rochester Institute of Technology
> > E-mail: wjh3710@rit.edu
>
> ----------------------
> William Heinbockel
> Information Security Incident Response Assistant
> Co-op Risk & Safety Management
> Rochester Institute of Technology
> E-mail: wjh3710@rit.edu

William Heinbockel wrote:
> Just an update on the database backend for nessusd...
>
> I've started coding a mysql interface/library for nessusd.
> It is written in C and uses mysql.h (the mysql C API).

That's great!

> I'm working with the database schema that Javi posted awhile back,
> but due to my lack of SQL knowledge and the way nessusd is set-up,
> I don't think that some of the tables can be setup in that manner.

Which ones specifically?

>
> I think the easiest way to incorporate the non-session specific data
> (i.e. the Nessus-plugin table and Services table and the like...) is
> to write a simple [perl?] script to create an sql modules directly
> from the plugins' directory and hard code a script to deal with the
> services.

For the plugin information you already have a way to do it. Check the
nessus-extract tool:
http://cvs.nessus.org/cgi-bin/cvsweb.cgi/nessus-tools/nessus-extract/nessus-extract.pl?rev=1.4&content-type=text/x-cvsweb-markup

It does not (yet) output in the schema that I proposed a while back for
plugins (CVE/Bugtraq references should be held in a separate table to
accomodate better with having multiple references per plugin)


As for the service information it should be pretty easy to make a
script to take all services (from /etc/services, the nmap services file
or whatever) and add them to a table. However, since services are not
referenced by port you need to first look into the service table (and
extract the ServiceId before inserting into the DetectedService table.
Regards

Javi

PS: Starting next week I hope I will have more time available to work on
this.

On Fri, 7 Feb 2003, Javier Fernandez-Sanguino wrote:

> William Heinbockel wrote:
> > Just an update on the database backend for nessusd...
> >
> > I've started coding a mysql interface/library for nessusd.
> > It is written in C and uses mysql.h (the mysql C API).
>
> That's great!
>
> > I'm working with the database schema that Javi posted awhile back,
> > but due to my lack of SQL knowledge and the way nessusd is set-up,
> > I don't think that some of the tables can be setup in that manner.
>
> Which ones specifically?
>
All of the ID's...
Currently I'm using the timestamp for the SessionID, but have no
idea what to do with the UserID, ExecutionID, ProtocolID...
This is probably most due to my lack of knowledge with databases.
And what is UserSessions::Location and User::Hash?

It might also end up being a problem to store these ID's and pass
them in the program.

> >
> > I think the easiest way to incorporate the non-session specific data
> > (i.e. the Nessus-plugin table and Services table and the like...) is
> > to write a simple [perl?] script to create an sql modules directly
> > from the plugins' directory and hard code a script to deal with the
> > services.
>
> For the plugin information you already have a way to do it. Check the
> nessus-extract tool:
> http://cvs.nessus.org/cgi-bin/cvsweb.cgi/nessus-tools/nessus-extract/nessus-extract.pl?rev=1.4&content-type=text/x-cvsweb-markup
>
> It does not (yet) output in the schema that I proposed a while back for
> plugins (CVE/Bugtraq references should be held in a separate table to
> accomodate better with having multiple references per plugin)
>

Right now, I am mainly trying to focus on where the information for the
database can be best obtained, for some stuff with the plugins, such
as RevisionNumber, you might have to integrate some code into NASL and
such. I don't know for certain though.

>
> As for the service information it should be pretty easy to make a
> script to take all services (from /etc/services, the nmap services file
> or whatever) and add them to a table. However, since services are not
> referenced by port you need to first look into the service table (and
> extract the ServiceId before inserting into the DetectedService table.
> Regards
>
> Javi
>
> PS: Starting next week I hope I will have more time available to work on
> this.
>
>

----------------------
William Heinbockel
Information Security Incident Response Assistant
Co-op Risk & Safety Management
Rochester Institute of Technology
E-mail: wjh3710@rit.edu

> -----Original Message-----
> From: William Heinbockel [mailto:wjh3710@osfmail.isc.rit.edu]
> Sent: Friday, February 07, 2003 8:53 AM
> To: Javier Fernandez-Sanguino
> Cc: nessus-devel@list.nessus.org
> Subject: Re: mySQL database for nessusd (code!)
>
>
> On Fri, 7 Feb 2003, Javier Fernandez-Sanguino wrote:
>
> > William Heinbockel wrote:
> > > Just an update on the database backend for nessusd...
> > >
> > > I've started coding a mysql interface/library for nessusd.
> > > It is written in C and uses mysql.h (the mysql C API).
> >
> > That's great!
> >
> > > I'm working with the database schema that Javi posted awhile back,
> > > but due to my lack of SQL knowledge and the way nessusd is set-up,
> > > I don't think that some of the tables can be setup in that manner.
> >
> > Which ones specifically?
> >
> All of the ID's...
> Currently I'm using the timestamp for the SessionID, but have no
> idea what to do with the UserID, ExecutionID, ProtocolID...
> This is probably most due to my lack of knowledge with databases.
> And what is UserSessions::Location and User::Hash?

I think that the hash column in the user table is to store a hash of the
user's password (MD5 or SHA1). I don't know what to do with location.

> It might also end up being a problem to store these ID's and pass
> them in the program.

The ID colums are generated in the database and are used as indexes
into some of the tables and to link rows in the tables together. We
really shouldn't have to worry about having the IDs used in the
program.

> > >
> > > I think the easiest way to incorporate the non-session
> specific data
> > > (i.e. the Nessus-plugin table and Services table and the
> like...) is
> > > to write a simple [perl?] script to create an sql modules directly
> > > from the plugins' directory and hard code a script to
> deal with the
> > > services.
> >
> > For the plugin information you already have a way to do
> it. Check the
> > nessus-extract tool:
> >
> http://cvs.nessus.org/cgi-bin/cvsweb.cgi/nessus-tools/nessus-e
xtract/nessus-extract.pl?rev=1.4&content-type=text/x-cvsweb-markup
>
> It does not (yet) output in the schema that I proposed a while back for
> plugins (CVE/Bugtraq references should be held in a separate table to
> accomodate better with having multiple references per plugin)
>
>
>Right now, I am mainly trying to focus on where the information for the
>database can be best obtained, for some stuff with the plugins, such
>as RevisionNumber, you might have to integrate some code into NASL and
>such. I don't know for certain though.
>
>
> As for the service information it should be pretty easy to make a
> script to take all services (from /etc/services, the nmap services file
> or whatever) and add them to a table. However, since services are not
> referenced by port you need to first look into the service table (and
> extract the ServiceId before inserting into the DetectedService table.
> Regards
>
> Javi
>
> PS: Starting next week I hope I will have more time available to work on
> this.
>
>

Will this script extract the data from the compiled plugins i.e. *.nes files?
>> > For the plugin information you already have a way to do
>> it. Check the
>> > nessus-extract tool:
>> >
>> http://cvs.nessus.org/cgi-bin/cvsweb.cgi/nessus-tools/nessus-e
> xtract/nessus-extract.pl?rev=1.4&content-type=text/x-cvsweb-markup

Alex.

Alex Zimin wrote:
> Will this script extract the data from the compiled plugins i.e. *.nes files?

No. It won't.

Javi

William Heinbockel wrote:
>> Which ones specifically?
>>
>
> All of the ID's...
> Currently I'm using the timestamp for the SessionID, but have no
> idea what to do with the UserID, ExecutionID, ProtocolID...

ExecutionID, ProtocolID and UserID should be autoincrement IDs used by
the database. You don't have to worry about them, when you insert any
new item in the table it should get a unique (and distinct ID).
Say you want to create a new session:

INSERT INTO Session (StartTime, Preferences, Configuration,
IPAdresses...) VALUES (X, Y, Z, W)

Say you want to retrieve the SessionID:

SELECT SessionID from Session where StarTime='X';

You can do this inmediately after inserting so you can be sure that's
your proper SessionID and pass that info between functions.

Say you want to add a new service detected by a port scanner, the
service runs in TCP portnumber 80:

Service = SELECT ServiceID FROM Services, Protocol WHERE
Services.ProtocolID=Protocol.ProtocolID AND Protocol=6 and PortNumber=80

(Note: TCP is protocol #6 TCP)

INSERT INTO DetectedService (PluginID,HostId,SessionID,ServiceID) VALUES
(X,Y,Z, 'Service')

> This is probably most due to my lack of knowledge with databases.
> And what is UserSessions::Location and User::Hash?

UserSessions::Location is the place the Nessus client is connected from
(IP address). User::Hash is the password IIRC. Will have to check.

>
> It might also end up being a problem to store these ID's and pass
> them in the program.

There's no need to pass them back. Just retrieve them using an SQL
statement whenever it's needed. It's also better to pass this (numeric)
IDs from function to function that to pass a whole lot of other information.

>
> Right now, I am mainly trying to focus on where the information for the
> database can be best obtained, for some stuff with the plugins, such
> as RevisionNumber, you might have to integrate some code into NASL and
> such. I don't know for certain though.
>

Don't worry about information that should be in the database _before_
running a scan. These information should be created based on a SQL
script that gets loaded (by the user) as part of the database creation
he needs to (manually) do. After all he needs to create the schema for
the database before using it. The following tables, IMHO, should be
created that way (instead of having the nessus server populate them):

- Plugin information: Nessus-plugin, Plugin-CVE, Plugin-BID
- Services information: Services, Protocol

User information should be probably included by the nessus-adduser
scripts. Whileas UserSession/Session information should be included when
the session starts/ends.

Regards

Javi

Minderhout, Doug wrote:

>
>>It might also end up being a problem to store these ID's and pass
>>them in the program.
>
>
> The ID colums are generated in the database and are used as indexes
> into some of the tables and to link rows in the tables together. We
> really shouldn't have to worry about having the IDs used in the
> program.
>

No, they should be used by the code which adds new table elements which
reference them. Otherwise you end up with a scan that does not include
any valuable information in the DetectedService table, for example
(since you omit the inclusion of the ServiceID which is the information
you really want). Or which does not crossreference the SessionID when a
new plugin gets executed and is introduced in the ExecutedPlugins table.

Table IDs have are generated by the database (when the code inserts new
items) but need to be retrieved whenever they are going to be used and
probably passed along between functions.

Regards


Javi