Mailing List Archive: [clamav-users] reloading database problem

[clamav-users] reloading database problem

Feb 9, 2022, 12:45 AM

Post #1 of 7 (549 views)

Hello,

I have clamav 0.103.5 installed on debian 11 and I'm getting too often
errors when reloading database.

looking back this problem started appearing on:

Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
Mon May 10 12:48:13 2021 -> WARNING: Database reload failed, keeping the previous instance
Mon May 10 13:22:53 2021 -> ERROR: reload_th: Database load failed: Can't allocate memory
Mon May 10 13:22:55 2021 -> WARNING: Database reload failed, keeping the previous instance
Mon May 10 13:55:26 2021 -> ERROR: reload_th: Database load failed: Can't allocate memory
Mon May 10 13:55:28 2021 -> WARNING: Database reload failed, keeping the previous instance
Mon May 10 14:54:47 2021 -> ERROR: reload_th: Database load failed: Malformed database
Mon May 10 14:54:49 2021 -> WARNING: Database reload failed, keeping the previous instance
Mon May 10 15:52:53 2021 -> SelfCheck: Database modification detected. Forcing reload.
Mon May 10 15:53:56 2021 -> ERROR: reload_th: Database load failed: Malformed database
Mon May 10 15:53:58 2021 -> WARNING: Database reload failed, keeping the previous instance
Mon May 10 17:47:55 2021 -> ERROR: reload_th: Database load failed: Can't allocate memory
Mon May 10 17:47:57 2021 -> WARNING: Database reload failed, keeping the previous instance
Mon May 10 20:47:48 2021 -> Database correctly reloaded (12708784 signatures)

Yesterday I checked all databases using:

clamscan -d "$file" /var/lib/clamav-unofficial-sigs/configs/scan-test.txt

... no error was produced.

this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB of RAM:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2169 clamav 20 0 1705796 1.5g 6380 S 0.0 39.5 0:33.83 clamd

I use multiple third-party signatures
- last added securiteinfo on 2020/05/04, no huge difference in sigcount

- clamav was upgraded from 0.102.4 to 0.103.2 on 2021-04-24
and this was last change before this happened:

-rw-r----- 1 root clamav 1395 May 4 2020 /etc/clamav-unofficial-sigs.conf
-rw-r--r-- 1 root root 1873 Feb 8 2020 /etc/clamav/clamd.conf
-r--r--r-- 1 clamav adm 715 Apr 24 2021 /etc/clamav/freshclam.conf

I wonder if this problem may be caused by i386 architecture with 3GB limit
per process:

Does clamd reload signature database in the same process?
(many servers use fork themselves and load config to a new process, would
avoid this error)

is the "Malformed database" just incorrect error code for this case?

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
We are but packets in the Internet of life (userfriendly.org)

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] reloading database problem [ In reply to ]

clamav-users at lists

Feb 9, 2022, 1:44 AM

Post #2 of 7 (549 views)

Permalink

Hi there,

On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:

> I have clamav 0.103.5 installed on debian 11 and I'm getting too often
> errors when reloading database.
>
> looking back this problem started appearing on:
>
> Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
> Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
> ...

What a lot of signatures! I'm at around 8.8 million at the moment,
with about 45 additional third-party databases and yara rule sets.

> this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...

With 8.8M sigs on ARM7 64 bit with 4G RAM I'm using about 1.2GB of
resident memory and concurrent reloads give no trouble. There were
some 'malformed' bleatings in the log back at the end of June - early
July, but I think that was a real database problem which was promptly
fixed. Nothing at all since then.

> I wonder if this problem may be caused by i386 architecture with 3GB limit ...
> Does clamd reload signature database in the same process?

It's a very long time since I ran ClamAV on i386 so I've no experience
to offer. If your suspicion is correct it might be a problem specific
to the machine:

https://en.wikipedia.org/wiki/3_GB_barrier

There's a configuration option to avoid the doubled memory usage on a
database reload, look in the configuration file for clamd for the
'ConcurrentDatabaseReload' directive. Be aware of the issues, you
might not want to pause scanning during reloads.

> is the "Malformed database" just incorrect error code for this case?

It's not impossible. One of the most valuable lessons I learned early
in my career was not to put too much faith in the error messages given
by most computer software. Sometimes I will recompile an executable
with a bunch extra error messages when I wonder if I understand what's
going on (the ClamAV error handling is generally pretty well organized
which makes that easy). But if you stress things enough you're always
going to find corner cases.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] reloading database problem [ In reply to ]

uhlar at fantomas

Feb 10, 2022, 12:58 AM

Post #3 of 7 (547 views)

Permalink

>On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:
>
>>I have clamav 0.103.5 installed on debian 11 and I'm getting too often
>>errors when reloading database.
>>
>>looking back this problem started appearing on:
>>
>>Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
>>Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
>>...

On 09.02.22 09:44, G.W. Haywood via clamav-users wrote:
>What a lot of signatures! I'm at around 8.8 million at the moment,
>with about 45 additional third-party databases and yara rule sets.

I think most of it comes from securiteinfo.com feed, which I have
subscribed into. I have this machine for personal use.

it seems their signatures are the most commonly catched:

% zgrep -Fih FOUND `ls -1tr clamav.log*` | awk '$8 == "(deleted):" {print $9;next} {print $8}' | cut -f1 -d. | sort | uniq -c|sort -nr
84 SecuriteInfo
62 Porcupine
32 Sanesecurity
2 PhishTank
1 Bofhland

(there may be duplicates so the real difference may be smaller)

>>this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...
>
>With 8.8M sigs on ARM7 64 bit with 4G RAM I'm using about 1.2GB of
>resident memory and concurrent reloads give no trouble. There were
>some 'malformed' bleatings in the log back at the end of June - early
>July, but I think that was a real database problem which was promptly
>fixed. Nothing at all since then.
>
>>I wonder if this problem may be caused by i386 architecture with 3GB limit ...
>>Does clamd reload signature database in the same process?
>
>It's a very long time since I ran ClamAV on i386 so I've no experience
>to offer. If your suspicion is correct it might be a problem specific
>to the machine:
>
>https://en.wikipedia.org/wiki/3_GB_barrier

yes, this is what I'm guessing.
I'm just curious if someone can confirm this or I have to try.
so far I was lazy to convert this machine (or at least part of it) to
64-bit. 64-bit kernel should help to move the barrier to 4G.

>There's a configuration option to avoid the doubled memory usage on a
>database reload, look in the configuration file for clamd for the
>'ConcurrentDatabaseReload' directive. Be aware of the issues, you
>might not want to pause scanning during reloads.

I know of this feature, just wanted to avoid it.

>>is the "Malformed database" just incorrect error code for this case?
>
>It's not impossible. One of the most valuable lessons I learned early
>in my career was not to put too much faith in the error messages given
>by most computer software. Sometimes I will recompile an executable
>with a bunch extra error messages when I wonder if I understand what's
>going on (the ClamAV error handling is generally pretty well organized
>which makes that easy). But if you stress things enough you're always
>going to find corner cases.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
10 GOTO 10 : REM (C) Bill Gates 1998, All Rights Reserved!

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] reloading database problem [ In reply to ]

clamav-users at lists

Feb 10, 2022, 1:38 AM

Post #4 of 7 (547 views)

Permalink

Hi there,

On Thu, 10 Feb 2022, Matus UHLAR - fantomas wrote:

> ...
> I think most of it comes from securiteinfo.com feed, which I have subscribed
> into. I have this machine for personal use.
>
> it seems their signatures are the most commonly catched:
>
> % zgrep -Fih FOUND `ls -1tr clamav.log*` | awk ...
> 84 SecuriteInfo
> 62 Porcupine
> 32 Sanesecurity

That's a bit odd. You seem to be getting roughly twice the hits from
Porcupine that you get from Sansecurity, and over here it's the other
way around although the difference is smaller. We see about 50%-60%
more from Sanesecurity than from Porcupine, 85 and 55 respectively to
date in February. In fact my Yara rules catch many more than that, I
wonder if they catch more of what Porcupine would have caught and your
SecuriteInfo sigs catch more of what Sanesecurity would have caught.

I've looked into telling ClamAV to report all the matches it can find
instead of just the first, but actually doing that hasn't yet reached
the top of this 'in' tray. I'll stop. A fellow could go nuts.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] reloading database problem [ In reply to ]

uhlar at fantomas

Feb 13, 2022, 2:14 AM

Post #5 of 7 (540 views)

Permalink

>>On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:
>>>I have clamav 0.103.5 installed on debian 11 and I'm getting too often
>>>errors when reloading database.
>>>
>>>looking back this problem started appearing on:
>>>
>>>Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
>>>Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
[...]
>>>this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...

>>>I wonder if this problem may be caused by i386 architecture with 3GB limit ...
>>>Does clamd reload signature database in the same process?

>On 09.02.22 09:44, G.W. Haywood via clamav-users wrote:
>>It's a very long time since I ran ClamAV on i386 so I've no experience
>>to offer. If your suspicion is correct it might be a problem specific
>>to the machine:
>>
>>https://en.wikipedia.org/wiki/3_GB_barrier

On 10.02.22 09:58, Matus UHLAR - fantomas wrote:
>yes, this is what I'm guessing.
>I'm just curious if someone can confirm this or I have to try.
>so far I was lazy to convert this machine (or at least part of it) to
>64-bit. 64-bit kernel should help to move the barrier to 4G.

I have rebooted into 64-bit kernel, without changing any installed software.
looks like database updates work flawlessly since:

Fri Feb 11 19:52:38 2022 -> SelfCheck: Database modification detected. Forcing reload.
Fri Feb 11 19:53:03 2022 -> ERROR: reload_th: Database load failed: Can't allocate memory
Fri Feb 11 19:53:04 2022 -> WARNING: Database reload failed, keeping the previous instance
Fri Feb 11 20:42:57 2022 -> +++ Started at Fri Feb 11 20:42:57 2022
Fri Feb 11 20:42:57 2022 -> Not loading PUA signatures.
Fri Feb 11 20:43:28 2022 -> Loaded 12726414 signatures.
Fri Feb 11 20:49:16 2022 -> Database correctly reloaded (12726430 signatures)
Fri Feb 11 20:49:16 2022 -> Activating the newly loaded database...
Fri Feb 11 21:54:23 2022 -> Database correctly reloaded (12726435 signatures)
Fri Feb 11 21:54:23 2022 -> Activating the newly loaded database...
Fri Feb 11 22:49:08 2022 -> SelfCheck: Database modification detected. Forcing reload.
Fri Feb 11 22:49:45 2022 -> Database correctly reloaded (12726434 signatures)
Fri Feb 11 22:49:45 2022 -> Activating the newly loaded database...

So the 3GB barrier applies to clamav (no wonder) when reloading signatures.
- unlike other SW, no new clamd instance after reload.

>>There's a configuration option to avoid the doubled memory usage on a
>>database reload, look in the configuration file for clamd for the
>>'ConcurrentDatabaseReload' directive. Be aware of the issues, you
>>might not want to pause scanning during reloads.
>
>I know of this feature, just wanted to avoid it.

even my swap usage is lower, which is a good thing.

I'm going to activate zswap again. Before this change, my machine was
running quite slowly, apparently because of excessive swapping due to
repeated attempts to reload signature.

I have learnt something...

>>What a lot of signatures! I'm at around 8.8 million at the moment,
>>with about 45 additional third-party databases and yara rule sets.

>On Thu, 10 Feb 2022, Matus UHLAR - fantomas wrote:
>>I think most of it comes from securiteinfo.com feed, which I have
>>subscribed into. I have this machine for personal use.
>>
>>it seems their signatures are the most commonly catched:
>>
>>% zgrep -Fih FOUND `ls -1tr clamav.log*` | awk ...
>> 84 SecuriteInfo
>> 62 Porcupine
>> 32 Sanesecurity
[...]
>>(there may be duplicates so the real difference may be smaller)

On 10.02.22 09:38, G.W. Haywood via clamav-users wrote:
>That's a bit odd. You seem to be getting roughly twice the hits from
>Porcupine that you get from Sansecurity, and over here it's the other
>way around although the difference is smaller. We see about 50%-60%
>more from Sanesecurity than from Porcupine, 85 and 55 respectively to
>date in February. In fact my Yara rules catch many more than that, I
>wonder if they catch more of what Porcupine would have caught and your
>SecuriteInfo sigs catch more of what Sanesecurity would have caught.

that's what I meant by duplicates.

>I've looked into telling ClamAV to report all the matches it can find
>instead of just the first, but actually doing that hasn't yet reached
>the top of this 'in' tray. I'll stop. A fellow could go nuts.

this could eliminate many duplicates, which could help us quite a bit.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
42.7 percent of all statistics are made up on the spot.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] reloading database problem [ In reply to ]

uhlar at fantomas

Feb 13, 2022, 4:16 AM

Post #6 of 7 (539 views)

Permalink

On 13.02.22 11:14, Matus UHLAR - fantomas wrote:
>>>On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:
>>>>I have clamav 0.103.5 installed on debian 11 and I'm getting too often
>>>>errors when reloading database.
>>>>
>>>>looking back this problem started appearing on:
>>>>
>>>>Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
>>>>Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database

>>>>this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...

>>>>I wonder if this problem may be caused by i386 architecture with 3GB limit ...
>>>>Does clamd reload signature database in the same process?

>I have rebooted into 64-bit kernel, without changing any installed software.
>looks like database updates work flawlessly since:
[...]

>So the 3GB barrier applies to clamav (no wonder) when reloading signatures.
>- unlike other SW, no new clamd instance after reload.

I just encountered the DB reload, watched it closely:

Sun Feb 13 12:46:13 2022 -> Reading databases from /var/lib/clamav
Sun Feb 13 12:46:50 2022 -> Database correctly reloaded (12732534 signatures)
Sun Feb 13 12:46:50 2022 -> Activating the newly loaded database...

meanwhile clamd usage doubled and crossed 3.350G
after 2-3 minutes it dropped back to 1.7G.

I'd invite clamd logging message about database successfully activated.

I'm enabling zswap again, hopefully this time it won't kill system
performance.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
How does cat play with mouse? cat /dev/mouse

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] reloading database problem [ In reply to ]

uhlar at fantomas

Feb 23, 2022, 4:11 AM

Post #7 of 7 (520 views)

Permalink

>>>>On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:
>>>>>I have clamav 0.103.5 installed on debian 11 and I'm getting too often
>>>>>errors when reloading database.
>>>>>
>>>>>looking back this problem started appearing on:
>>>>>
>>>>>Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
>>>>>Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
>
>>>>>this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...
>
>>>>>I wonder if this problem may be caused by i386 architecture with 3GB limit ...
>>>>>Does clamd reload signature database in the same process?

>On 13.02.22 11:14, Matus UHLAR - fantomas wrote:
>>I have rebooted into 64-bit kernel, without changing any installed software.
>>looks like database updates work flawlessly since:
>[...]
>
>>So the 3GB barrier applies to clamav (no wonder) when reloading signatures.
>>- unlike other SW, no new clamd instance after reload.

On 13.02.22 13:16, Matus UHLAR - fantomas wrote:
>I just encountered the DB reload, watched it closely:
>
>Sun Feb 13 12:46:13 2022 -> Reading databases from /var/lib/clamav
>Sun Feb 13 12:46:50 2022 -> Database correctly reloaded (12732534 signatures)
>Sun Feb 13 12:46:50 2022 -> Activating the newly loaded database...
>
>meanwhile clamd usage doubled and crossed 3.350G
>after 2-3 minutes it dropped back to 1.7G.
>
>I'd invite clamd logging message about database successfully activated.
>
>I'm enabling zswap again, hopefully this time it won't kill system
>performance.

notes after some time:

after moving to 64-bit OS with 4G available for 32-bit clamav swap issue
became a bit better, swap usage was higher, but the system was not killed by
continuously swapping when trying to reload the database (and failing).

I have removed the biggest database "securiteinfoold.hdb" that took about
300MB on disk (IIRC, the size was close to main and daily)
- number of signatures lowered from 12733384 to 9148084
- RAM usage lowered from 1705796 virt/ 1.5G res to 1364984 virt / 1.2g res
(TOP output)

I currently have 0.5G of swap used.

I will keep it running like this for some time and then fetch
securiteinfoold.hdb and then see how many of mails it catches.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
My mind is like a steel trap - rusty and illegal in 37 states.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml