Mailing List Archive

Clamd terminating (Malformed database)
Hi Folks

I seem to have a problem ! It's happened a few times now. Clamd seems to be starting up happily enough but as you can see from the logs below, it is doing a few things that I feel it should not be doing.

1. It seems to be detecting database modifications when freshclam has not been run.
2. After a number of hours it bombs out saying that the database is corrupt but when restarted it is quite happy again.

I have looked on the archives but cannot find this in relation to the win32 version.

Any ideas please ?

Many thanks

Phil

Fri Aug 22 03:09:16 2008 -> d:\Merak Mail\temp\200808220309140035.tmp: OK
Fri Aug 22 03:10:43 2008 -> d:\Merak Mail\temp\200808220310140037.tmp: OK
Fri Aug 22 03:18:56 2008 -> d:\Merak Mail\temp\200808220318160039.tmp: OK
Fri Aug 22 05:21:43 2008 -> SelfCheck: Database modification detected. Forcing reload.
Fri Aug 22 05:21:43 2008 -> Reading databases from C:\Program Files\clamAV\data
Fri Aug 22 05:21:43 2008 -> d:\Merak Mail\temp\20080822052143003B.tmp: OK
Fri Aug 22 05:22:43 2008 -> Database correctly reloaded (709009 signatures)
Fri Aug 22 05:38:21 2008 -> d:\Merak Mail\temp\200808220538200011p.tmp: OK
Fri Aug 22 05:48:20 2008 -> d:\Merak Mail\temp\200808220548200013p.tmp: OK
Fri Aug 22 07:19:57 2008 -> SelfCheck: Database modification detected. Forcing reload.
Fri Aug 22 07:19:57 2008 -> Reading databases from C:\Program Files\clamAV\data
Fri Aug 22 07:19:57 2008 -> d:\Merak Mail\temp\20080822071956003D.tmp: OK
Fri Aug 22 07:20:57 2008 -> Database correctly reloaded (709009 signatures)
Fri Aug 22 07:20:57 2008 -> d:\Merak Mail\temp\20080822072025003F.tmp: OK
Fri Aug 22 07:52:44 2008 -> SelfCheck: Database modification detected. Forcing reload.
Fri Aug 22 07:52:44 2008 -> Reading databases from C:\Program Files\clamAV\data
Fri Aug 22 07:52:44 2008 -> d:\Merak Mail\temp\200808220752430041.tmp: OK
Fri Aug 22 07:53:43 2008 -> Database correctly reloaded (709009 signatures)
Fri Aug 22 08:01:08 2008 -> d:\Merak Mail\temp\200808220801040042.tmp: OK
Fri Aug 22 08:43:03 2008 -> SelfCheck: Database modification detected. Forcing reload.
Fri Aug 22 08:43:03 2008 -> Reading databases from C:\Program Files\clamAV\data
Fri Aug 22 08:43:03 2008 -> d:\Merak Mail\temp\200808220842590043.tmp: OK
Fri Aug 22 08:44:10 2008 -> ERROR: reload db failed: Malformed database
Fri Aug 22 08:44:10 2008 -> Terminating because of a fatal error.
Fri Aug 22 08:44:10 2008 -> Shutting down the main socket.
Fri Aug 22 08:44:10 2008 -> Closing the main socket.
Fri Aug 22 08:44:10 2008 -> --- Stopped at Fri Aug 22 08:44:10 2008

_____________________________________________

Website Hosting from only £5.00 per month.
www.medwayhosting.com - +44 (0)1634 856965
_____________________________________________

Digital & Traditional Printing, and much more
www.medwayprint.com - +44 (0)1634 281199
_____________________________________________
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
Phil in <007601c90441$2bf6cdf0$0600a8c0@phil>:

Hi Phil,

> 1. It seems to be detecting database modifications when freshclam has not been run.
> 2. After a number of hours it bombs out saying that the database is corrupt but when restarted it is quite happy again.

Some shots in the dark...

In case it's just data corruption I'd delete data\*.cvd and re-run
freshclam. If the problem persists after the download look for anything
interesting in "freshclam -v --debug" and "clamd --debug".

I'm not sure which distribution you're using but I'd check the .conf
files for paths with spaces - these might need to be quoted (with speech
marks), e.g., "C:\Program Files\ClamAV\data".

The official port expects C:\WINDOWS\TEMP which isn't a default under
Win2k so you may need to create the directory or edit clamd.conf.

Cheers,
Martin






_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
----- Original Message -----
From: "Martin Clayton" <local.mc.clamav-win32@clayreed.com>
To: <clamav-win32@lists.clamav.net>
Sent: Friday, August 22, 2008 12:35 PM
Subject: Re: [clamav-win32] Clamd terminating (Malformed database)


> In case it's just data corruption I'd delete data\*.cvd and re-run
> freshclam.

Ok done - will have to wait and see now !! One thing I noticed - with the
old files, in the clam log it said:
Fri Aug 22 10:49:51 2008 -> Loaded 709009 signatures.
With the new defs, it now says:
Fri Aug 22 13:57:41 2008 -> Loaded 1113131 signatures.

Does that figure sound right ? If it is then it certainly indicates the
originals were corrupt after all.

>
> I'm not sure which distribution you're using but I'd check the .conf
> files for paths with spaces - these might need to be quoted (with speech
> marks), e.g., "C:\Program Files\ClamAV\data".

Yeh - that was part of the problem with me getting it running in the first
place !!

>
> The official port expects C:\WINDOWS\TEMP which isn't a default under
> Win2k so you may need to create the directory or edit clamd.conf.

I noticed that actually, so rather than playing around with config more than
I had to I created what it wanted !

Many thanks for the suggestions - I will be back for more if it still does
it !!

All the best

Phil


_____________________________________________

Website Hosting from only £5.00 per month.
www.medwayhosting.com - +44 (0)1634 856965
_____________________________________________

Digital & Traditional Printing, and much more
www.medwayprint.com - +44 (0)1634 281199
_____________________________________________
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
On 22 Aug 2008 at 14:18, Phil (Medway Hosting) wrote:

>
> ----- Original Message -----
> From: "Martin Clayton" <local.mc.clamav-win32@clayreed.com>
> To: <clamav-win32@lists.clamav.net>
> Sent: Friday, August 22, 2008 12:35 PM
> Subject: Re: [clamav-win32] Clamd terminating (Malformed database)
>
>
> > In case it's just data corruption I'd delete data\*.cvd and re-run
> > freshclam.
>
> Ok done - will have to wait and see now !! One thing I noticed - with the
> old files, in the clam log it said:
> Fri Aug 22 10:49:51 2008 -> Loaded 709009 signatures.
> With the new defs, it now says:
> Fri Aug 22 13:57:41 2008 -> Loaded 1113131 signatures.
>
> Does that figure sound right ? If it is then it certainly indicates the
> originals were corrupt after all.

From freshclam, the total of the main and daily is 401,469 (earlier today). The number
(re)loaded by clamd can be different if you have extra signatures (from Sanesecurity
etc.), but that number seems way out. You may have some old versions there.

I would either delete everything and do a complete reinstall, or carefully go through all
files where the signatures are and remove the old ones. Old versions (<0.93) also used
subdirectories.

If in doubt, post a directory listing here.

paul

_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
Paul Whelan wrote:
> On 22 Aug 2008 at 14:18, Phil (Medway Hosting) wrote:
>
>
>> ----- Original Message -----
>> From: "Martin Clayton" <local.mc.clamav-win32@clayreed.com>
>> To: <clamav-win32@lists.clamav.net>
>> Sent: Friday, August 22, 2008 12:35 PM
>> Subject: Re: [clamav-win32] Clamd terminating (Malformed database)
>>
>>
>>
>>> In case it's just data corruption I'd delete data\*.cvd and re-run
>>> freshclam.
>>>
>> Ok done - will have to wait and see now !! One thing I noticed - with the
>> old files, in the clam log it said:
>> Fri Aug 22 10:49:51 2008 -> Loaded 709009 signatures.
>> With the new defs, it now says:
>> Fri Aug 22 13:57:41 2008 -> Loaded 1113131 signatures.
>>
>> Does that figure sound right ? If it is then it certainly indicates the
>> originals were corrupt after all.
>>
>
> From freshclam, the total of the main and daily is 401,469 (earlier today). The number
> (re)loaded by clamd can be different if you have extra signatures (from Sanesecurity
> etc.), but that number seems way out. You may have some old versions there.
>
> I would either delete everything and do a complete reinstall, or carefully go through all
> files where the signatures are and remove the old ones. Old versions (<0.93) also used
> subdirectories.
>
I'd concur with Paul. I use the base plus SaneSecurity signatures, and
clamd says:
Fri Aug 22 08:16:35 2008 -> Reading databases from c:\Program
Files\clamAV\data
Fri Aug 22 08:16:39 2008 -> Loaded 423994 signatures.

So yes, 1,113,131 sounds way, way too high-- close to triple normal.

Bret

_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
----- Original Message -----
From: "Bret Miller" <bret.miller@wcg.org>
To: <clamav-win32@lists.clamav.net>
Sent: Friday, August 22, 2008 4:53 PM
Subject: Re: [clamav-win32] Clamd terminating (Malformed database)


> So yes, 1,113,131 sounds way, way too high-- close to triple normal.

AHHH - Ok - Kick me !

I renamed the original cvd's and did the freshclam. I didn't realise clamd
picks up ALL the cvd's in data nomatter what they are called. Moved the old
ones to a different folder and now I am back down to 400670.

So - that's that problem solved - now to simply wait to see if it bombs
again !!

Many thanks for your help.

All the best

Phil

p.s. This list is odd - I received Brett's reply to Paul's post 12 mins
BEFORE I received Paul's post itself - interesting !


_____________________________________________

Website Hosting from only £5.00 per month.
www.medwayhosting.com - +44 (0)1634 856965
_____________________________________________

Digital & Traditional Printing, and much more
www.medwayprint.com - +44 (0)1634 281199
_____________________________________________
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
On 22 Aug 2008 at 17:27, Phil (Medway Hosting) wrote:

> I renamed the original cvd's and did the freshclam. I didn't realise clamd
> picks up ALL the cvd's in data nomatter what they are called. Moved the old
> ones to a different folder and now I am back down to 400670.

That's lower than the current total, but Clamd only reports the signatures it loads
according to the .conf file. You probably exclude PUA sigs.

> p.s. This list is odd - I received Brett's reply to Paul's post 12 mins
> BEFORE I received Paul's post itself - interesting !

The mysteries of mailing lists and gateway processing!

paul


_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
----- Original Message -----
From: "Phil (Medway Hosting)" <phil@medwayhosting.com>
To: <clamav-win32@lists.clamav.net>
Sent: Friday, August 22, 2008 5:27 PM
Subject: Re: [clamav-win32] Clamd terminating (Malformed database)


> So - that's that problem solved - now to simply wait
> to see if it bombs again !!

Hi Folks

Ok - it DID bomb again :-(

Sun Aug 24 07:33:41 2008 -> SelfCheck: Database modification detected.
Forcing reload.
Sun Aug 24 07:33:41 2008 -> Reading databases from C:\Program
Files\clamAV\data
Sun Aug 24 07:33:41 2008 -> d:\Merak Mail\temp\200808240733390102.tmp: OK
Sun Aug 24 07:34:05 2008 -> Database correctly reloaded (400852 signatures)
Sun Aug 24 09:01:04 2008 -> SelfCheck: Database modification detected.
Forcing reload.
Sun Aug 24 09:01:04 2008 -> Reading databases from C:\Program
Files\clamAV\data
Sun Aug 24 09:01:04 2008 -> d:\Merak Mail\temp\200808240901020103.tmp: OK
Sun Aug 24 09:01:30 2008 -> ERROR: reload db failed: Malformed database
Sun Aug 24 09:01:30 2008 -> Terminating because of a fatal error.
Sun Aug 24 09:01:30 2008 -> Shutting down the main socket.
Sun Aug 24 09:01:30 2008 -> Closing the main socket.
Sun Aug 24 09:01:30 2008 -> --- Stopped at Sun Aug 24 09:01:30 2008

First question - Why might it be detecting database modifications when
freshclam hasn't been run ?
2nd - Why does it start up quite happily again after it detected "Malformed
database" ?

Also when it bombs, only clamd fails but srvany.exe doesn't. Is there a way
to force srvany.exe to fail if it's child process fails ? because the system
won't recover and restart clamd all the time srvany.exe is running.

Many thanks

Phil


_____________________________________________

Website Hosting from only £5.00 per month.
www.medwayhosting.com - +44 (0)1634 856965
_____________________________________________

Digital & Traditional Printing, and much more
www.medwayprint.com - +44 (0)1634 281199
_____________________________________________
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
On 26 Aug 2008 at 18:09, Phil (Medway Hosting) wrote:

> Sun Aug 24 09:01:04 2008 -> SelfCheck: Database modification detected.
> Forcing reload.
> Sun Aug 24 09:01:04 2008 -> Reading databases from C:\Program
> Files\clamAV\data
> Sun Aug 24 09:01:04 2008 -> d:\Merak Mail\temp\200808240901020103.tmp: OK
> Sun Aug 24 09:01:30 2008 -> ERROR: reload db failed: Malformed database
> Sun Aug 24 09:01:30 2008 -> Terminating because of a fatal error.
> Sun Aug 24 09:01:30 2008 -> Shutting down the main socket.
> Sun Aug 24 09:01:30 2008 -> Closing the main socket.
> Sun Aug 24 09:01:30 2008 -> --- Stopped at Sun Aug 24 09:01:30 2008
>
> First question - Why might it be detecting database modifications when
> freshclam hasn't been run ?

clamd does a check as set in clamd.conf (selfcheck) - default 1800 (30mins). Has
anything else got rights to those data files that might stop clamd reading them? Any
disk problems in system event log? It's strange too, that it reads the db just before
processing a mail file, and then fails. How do you run freshclam - on a schedule?

> 2nd - Why does it start up quite happily again after it detected "Malformed
> database" ?

It's misreporting the error?

> Also when it bombs, only clamd fails but srvany.exe doesn't. Is there a way
> to force srvany.exe to fail if it's child process fails ? because the system
> won't recover and restart clamd all the time srvany.exe is running.

Don't you want the service to restart automatically? If not run clamd manually without
srvany. (There are other service wrappers like firedaemon, srvstart, ntwrapper, but I'm
not sure if any of them can be set to stop if the app fails.)

paul

_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
Phil (Medway Hosting) wrote:
> ----- Original Message -----
> From: "Phil (Medway Hosting)" <phil@medwayhosting.com>
> To: <clamav-win32@lists.clamav.net>
> Sent: Friday, August 22, 2008 5:27 PM
> Subject: Re: [clamav-win32] Clamd terminating (Malformed database)
>
>
>
>> So - that's that problem solved - now to simply wait
>> to see if it bombs again !!
>>
>
> Hi Folks
>
> Ok - it DID bomb again :-(
>
> Sun Aug 24 07:33:41 2008 -> SelfCheck: Database modification detected.
> Forcing reload.
> Sun Aug 24 07:33:41 2008 -> Reading databases from C:\Program
> Files\clamAV\data
> Sun Aug 24 07:33:41 2008 -> d:\Merak Mail\temp\200808240733390102.tmp: OK
> Sun Aug 24 07:34:05 2008 -> Database correctly reloaded (400852 signatures)
> Sun Aug 24 09:01:04 2008 -> SelfCheck: Database modification detected.
> Forcing reload.
> Sun Aug 24 09:01:04 2008 -> Reading databases from C:\Program
> Files\clamAV\data
> Sun Aug 24 09:01:04 2008 -> d:\Merak Mail\temp\200808240901020103.tmp: OK
> Sun Aug 24 09:01:30 2008 -> ERROR: reload db failed: Malformed database
> Sun Aug 24 09:01:30 2008 -> Terminating because of a fatal error.
> Sun Aug 24 09:01:30 2008 -> Shutting down the main socket.
> Sun Aug 24 09:01:30 2008 -> Closing the main socket.
> Sun Aug 24 09:01:30 2008 -> --- Stopped at Sun Aug 24 09:01:30 2008
>
> First question - Why might it be detecting database modifications when
> freshclam hasn't been run ?
> 2nd - Why does it start up quite happily again after it detected "Malformed
> database" ?
>
> Also when it bombs, only clamd fails but srvany.exe doesn't. Is there a way
> to force srvany.exe to fail if it's child process fails ? because the system
> won't recover and restart clamd all the time srvany.exe is running.

OK... So I've not a clue why it's dying. It's almost as if it's having
trouble reading the disk.

You could try my service app (requires .NET 1.1 framework). The service
program monitors clamd to make sure it stays running, and restarts it if
not.
*http://mail.wcg.org/~postmaster/ClamAVService_Setup.exe*

You'll want to configure some settings in the registry if you chose to
use this. The settings are in
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ClamAV for
Windows\Parameters.
ClamPath = the path to the folder clamd.exe resides in.
ClamOptions = the arguments to pass to clamd.exe
MonitorInterval = how often to check clamd in milliseconds, default
60000 (60 seconds).
FreshclamMinutes = how often to run freshclam.exe. Set to 0 if you're
running it from somewhere else.
ProgramUpdateHours = how often to check for a program update and install
it. Set this to 0, please.
ProgramDownloadURL = where to download a program update from. Not needed
if you set the above to 0.

The service writes some information to the Windows application log.

Source available on request.

HTH,
Bret

_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
----- Original Message -----
From: "Bret Miller" <bret.miller@wcg.org>
To: <clamav-win32@lists.clamav.net>
Sent: Tuesday, August 26, 2008 7:29 PM
Subject: Re: [clamav-win32] Clamd terminating (Malformed database)


> You could try my service app (requires .NET 1.1 framework). The service
> program monitors clamd to make sure it stays running, and restarts it if
> not.
> *http://mail.wcg.org/~postmaster/ClamAVService_Setup.exe*

Many thanks for that. I may well try it, if I can't find the reason for the
crashing.

What is the overhead of checking clamd is running more regularly (e.g. 10
seconds instead of the default 30 ?)

All the best

Phil


_____________________________________________

Website Hosting from only £5.00 per month.
www.medwayhosting.com - +44 (0)1634 856965
_____________________________________________

Digital & Traditional Printing, and much more
www.medwayprint.com - +44 (0)1634 281199
_____________________________________________
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
----- Original Message -----
From: "Paul Whelan" <Paul+cv@blakecomp.co.uk>
To: <clamav-win32@lists.clamav.net>
Sent: Tuesday, August 26, 2008 7:26 PM
Subject: Re: [clamav-win32] Clamd terminating (Malformed database)


> clamd does a check as set in clamd.conf (selfcheck) - default 1800
(30mins). Has
> anything else got rights to those data files that might stop clamd reading
them?

Not that I am aware. Very little running on that machine at all. Merak Mail
Server, Clamd, Serv-U Ftp Server, Process Explorer - that's it.

> Any
> disk problems in system event log?

None whatsoever. All I get is:

Event Type: Error
Event Source: clamd
Event Category: None
Event ID: 1
Date: 24/08/2008
Time: 09:01:30
User: N/A
Computer: FTPSERVER
Description:
reload db failed: Malformed database

> It's strange too, that it reads the db just before
> processing a mail file, and then fails. How do you run freshclam - on a
schedule?

At the moment freshclam isn't scheduled at all. Manual run only - I tend to
only set up one new thing at a time - leave it running a few weeks, then try
the next thng.

>
> > 2nd - Why does it start up quite happily again after it detected
"Malformed
> > database" ?
>
> It's misreporting the error?

I have restarted it with "Debug yes" to see if that gives any extra info.

> Don't you want the service to restart automatically? If not run clamd
manually without
> srvany. (There are other service wrappers like firedaemon, srvstart,
ntwrapper, but I'm
> not sure if any of them can be set to stop if the app fails.)

Yes I do want it to restart, but at the moment srvany.exe stays running when
clamd crashes, so the service does not get restarted. I will probably try
Brett's suggestion if I can't actually stop it crashing.

A question - In the description I found on how to set Clamd up as a service
for Merak
(http://linuxforum.icewarp.com/viewtopic.php?t=2183&sid=2460433938d2f2cbe503
571e80fc8ebf) it said the following:

==================================================
C:\Program Files\Windows Resource Kits\Tools\instsrv ClamAV "C:\Program
Files\Windows Resource Kits\Tools\srvany.exe"

Edit the registry entry for your custom service.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ClamAV

Make a Key called "Parameters". From the Edit menu, click Add Key. Type the
following and click OK:

Key Name: Parameters
Class : <leave blank>

Select the Parameters key.
From the Edit menu, click Add Value. Type the following and click OK:

Value Name: Application
Data Type : REG_SZ
String : C:\Program Files\clamAV\clamd.exe
====================================================

Cannot Clamd not simply be run as a service itself without running it
through srvany.exe ?

Many thanks

Phil


_____________________________________________

Website Hosting from only £5.00 per month.
www.medwayhosting.com - +44 (0)1634 856965
_____________________________________________

Digital & Traditional Printing, and much more
www.medwayprint.com - +44 (0)1634 281199
_____________________________________________
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
Re: Clamd terminating (Malformed database) [ In reply to ]
It uses TCP/IP to communicate, so clamd has to be listening on 127.0.0.1
on the default port. The overhead of the check is simply the overhead of
issuing a PING command to clamd, which should be minimal I would think.

Bret

Phil (Medway Hosting) wrote:
> ----- Original Message -----
> From: "Bret Miller" <bret.miller@wcg.org>
> To: <clamav-win32@lists.clamav.net>
> Sent: Tuesday, August 26, 2008 7:29 PM
> Subject: Re: [clamav-win32] Clamd terminating (Malformed database)
>
>
>
>> You could try my service app (requires .NET 1.1 framework). The service
>> program monitors clamd to make sure it stays running, and restarts it if
>> not.
>> *http://mail.wcg.org/~postmaster/ClamAVService_Setup.exe*
>>
>
> Many thanks for that. I may well try it, if I can't find the reason for the
> crashing.
>
> What is the overhead of checking clamd is running more regularly (e.g. 10
> seconds instead of the default 30 ?)
>
> All the best
>
> Phil
>
>
> _____________________________________________
>
> Website Hosting from only £5.00 per month.
> www.medwayhosting.com - +44 (0)1634 856965
> _____________________________________________
>
> Digital & Traditional Printing, and much more
> www.medwayprint.com - +44 (0)1634 281199
> _____________________________________________
> _______________________________________________
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32
>
_______________________________________________
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-win32