Mailing List Archive

Google safebrowsing types and usage questions
Hi,
I've just registered for and received a Google safebrowing API key and
configured clamav-safebrowsing
(https://github.com/Cisco-Talos/clamav-safebrowsing) on a fedora32
server system.

btw, I found out the hard way that having a percent sign in the
password causes the clamav-safebrowsing script to fail.

It appears to have loaded another 3M signatures. Where can I find more
info about those signatures? I'm especially interested in the types of
attacks it is designed to stop. I've located this URL that appears to
describe four categories, but is there any more info available?

https://developers.google.com/safe-browsing/v4/reference/rest/v4/ThreatType

Are there any more specifics available about each category? Do the
patterns have names in the same way the sanesecurity patterns do?

What is the purpose of the mysql database if the signatures are in a
GDB file in /var/lib/clamav? I'm assuming the database is updated then
"build" is used to dump it to a file instead of having to download it
in full every time?

I'd like to replicate the database across all servers to save on
bandwidth and just have the master be updated. Does this make sense? I
can then rsync the GDB file from the master server, or is it possible
to just dump the database without also trying to update it?

I also still have the old safebrowsing.cld database from the end of
2019 (version: 49191, sigs: 2213119, f-level: 63, builder: google).
Should I delete that?

How much memory needs to be allocated for clamav to store/process 14M
signatures?

Thanks,
Alex

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi there,

On Fri, 16 Oct 2020, Alex via clamav-users wrote:

> ... I found out the hard way that having a percent sign in the
> password causes the clamav-safebrowsing script to fail.

Perhaps you just need to escape it.

> It appears to have loaded another 3M signatures. Where can I find more
> info about those signatures?

Ask the provider of the signatures?

Try using 'sigtool' on the signature database file?

> I'm especially interested in the types of attacks it is designed to
> stop.

Seems a bit cart-before-horse.

> I've located this URL that appears to describe four categories, but
> is there any more info available?

There may be people here who are more familiar than I am with Google's
safebrowsing and can answer your questions better, but I'm not sure
that you're asking in the right place.

> What is the purpose of the mysql database if the signatures are in a
> GDB file in /var/lib/clamav?

What mysql database is this?

> I'm assuming ...

There's a witticism about that around here somewhere.

> I'd like to replicate the database across all servers to save on
> bandwidth and just have the master be updated. Does this make sense?

Yes. There's a document about local mirrors on the ClamAV Website
which you should read:

https://www.clamav.net/documents/private-local-mirrors

> ... is it possible to just dump the database without also trying to
> update it?

I don't understand what you're trying to achieve. Do you mean dump as
in 'mysqldump'? The ClamAV database files are just flat files, ASCII
text, usually one line per signature. You can read them with a pager
or a text editor, pipe them through 'grep', and that kind of thing (I
do that quite a lot, mostly in response to questions on this list).
The file formats are documented on the ClamAV Website:

https://www.clamav.net/documents/creating-signatures-for-clamav

> I also still have the old safebrowsing.cld database from the end of
> 2019 (version: 49191, sigs: 2213119, f-level: 63, builder: google).
> Should I delete that?

If it were mine, and it was at risk of being overwritten, I'd move
somewhere it rather than deleting it.

> How much memory needs to be allocated for clamav to store/process 14M
> signatures?

Try running 'top'. Depends on the signatures of course, and there's
the cart-and-horse thing. With 11.3M signatures from a great variety
of sources, my own clamd server uses about 1.3GBytes for the clamd
daemon alone (and that's briefly doubled on database reload, but there
are ways around that). I don't use clamav-milter and the server does
very little else. It has 4G of RAM. My *rough* guess for 14M sigs is
about 1.5 GBytes apart from the reloading issue, but it's just a guess
because (a) you haven't said what ClamAV will be doing for you, and
(b) I know very little about your signatures profile.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi Alex,

I'm glad to hear that the clamav-safebrowsing tool is working for you. Please do report bugs to the project's github issue tracker though do bear in mind that it may take a while before anyone has time to work on it. If you're able to submit bugfixes yourself, pull requests are always appreciated.

The clamav-safebrowsing tool was developed by another team and gifted to my team for OSS maintenance. While I've tested it a little bit, I'm not intimately familiar with it. I'll try to answer your questions as best I can inline...

> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> Alex via clamav-users

> btw, I found out the hard way that having a percent sign in the password
> causes the clamav-safebrowsing script to fail.

That's a good observation. If it's not an escaping issue as Ged suggested, then can you please create a github issue for this item?

> It appears to have loaded another 3M signatures. Where can I find more info
> about those signatures? I'm especially interested in the types of attacks it is
> designed to stop. I've located this URL that appears to describe four
> categories, but is there any more info available?
>
> https://developers.google.com/safe-browsing/v4/reference/rest/v4/ThreatType
>
> Are there any more specifics available about each category? Do the patterns
> have names in the same way the sanesecurity patterns do?'

As far as I know, google safebrowsing rules have no names and you simply have to trust that they are sites which are not safe to browse.

> What is the purpose of the mysql database if the signatures are in a GDB file in
> /var/lib/clamav? I'm assuming the database is updated then "build" is used to
> dump it to a file instead of having to download it in full every time?
>
> I'd like to replicate the database across all servers to save on bandwidth and
> just have the master be updated. Does this make sense? I can then rsync the
> GDB file from the master server, or is it possible to just dump the database
> without also trying to update it?

The mysql database exists for precisely that reason. The safebrowsing rules change a lot and as you noted it's a huge ruleset. Mysql is more efficient to update than our .gdb file format. Your idea to rsync the .gdb file after each update makes good sense to me.

> I also still have the old safebrowsing.cld database from the end of
> 2019 (version: 49191, sigs: 2213119, f-level: 63, builder: google).
> Should I delete that?

Yes, if you're generating your own safebrowsing ruleset, the one safebrowsing.cld file is obsolete.

> How much memory needs to be allocated for clamav to store/process 14M
> signatures?

I don't know off-hand. It certainly varies by signature type. If you're asking about safebrowsing rules, then what you can do is modify your clamd.conf to point DatabaseDirectory to a location that only contains safebrowsing rules. When it loads, check how much ram it's using. The summary info for `clamscan -d <DATABASE> blah` will tell you the # of "known viruses" (loaded signatures). After that, a little math will help you estimate how much RAM a larger rule set would require.


Regards,
Micah

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi Micah,

On Fri, 16 Oct 2020, Micah Snyder (micasnyd) via clamav-users wrote:

> The clamav-safebrowsing tool was developed by another team and
> gifted to my team for OSS maintenance.

As if you didn't have enough to do already!

> ... The mysql database exists for precisely that reason.
> ... the one safebrowsing.cld file is obsolete.

There's something I'm missing. None of this seems to be mentioned at

https://www.clamav.net/documents/safebrowsing

If you'd like a documentation patch I can do some reading and give it
a go, what's the best place to start?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi all,The script for safe browsing is not downloading all the definitions all the time since it will be time/resource-consuming. Definitions are stored in mysql and only the delta is downloaded from google. After the download is successfully and the mysql data is up-to-date you can generate the file for clamav.Basically you have two scripts:clamsbsync.py to update the mysql db from googleclamsbwrite.py to create the .gdb file for clamavps: don't put too much faith in safebrowsing database. I rarely have any hit even if there are almost 3 milion of signatures.In contrast i have way more many hits with Sanesecurity.Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: "G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> Date: 10/17/20 12:23 (GMT+02:00) To: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net> Cc: "G.W. Haywood" <clamav@jubileegroup.co.uk> Subject: Re: [clamav-users] Google safebrowsing types and usage questions Hi Micah,On Fri, 16 Oct 2020, Micah Snyder (micasnyd) via clamav-users wrote:> The clamav-safebrowsing tool was developed by another team and> gifted to my team for OSS maintenance.As if you didn't have enough to do already!> ... The mysql database exists for precisely that reason.> ... the one safebrowsing.cld file is obsolete.There's something I'm missing.  None of this seems to be mentioned athttps://www.clamav.net/documents/safebrowsingIf you'd like a documentation patch I can do some reading and give ita go, what's the best place to start?-- 73,Ged._______________________________________________clamav-users mailing listclamav-users@lists.clamav.nethttps://lists.clamav.net/mailman/listinfo/clamav-usersHelp us build a comprehensive ClamAV guide:https://github.com/vrtadmin/clamav-faqhttp://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi there,

On Sat, 17 Oct 2020, Iulian Stan via clamav-users wrote:
> "G.W. Haywood via clamav-users" wrote:
>
>> what's the best place to start?
>
> Definitions are stored in mysql and only the delta is downloaded
> from google. After the download is successfully and the mysql data
> is up-to-date you can generate the file for clamav. Basically you
> have two scripts:clamsbsync.py to update the mysql db from
> googleclamsbwrite.py to create the .gdb file ...

Thanks, that's a very clear explanation. So instead of using freshclam
you just need to install mysql, Python, and some scripts. Hmmm...

> don't put too much faith in safebrowsing database. I rarely have any
> hit even if there are almost 3 milion of signatures. In contrast i
> have way more many hits with Sanesecurity.

Perhaps this DB would be more important for people who use on-access
scanning and are less fastidious in their browsing habits.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi Ged,Yes, the definition that you can download via freshclam directly from clamav site is outdated ( from 2019).You need to use https://github.com/Cisco-Talos/clamav-safebrowsingMy way to implement this was quite simple and i've also took the advantage of freshclam.On one central system i've installed the python/mysql/(other python libraries needed) and copy the generated clamav db to a place which is accessible via http/https.Then in every clamav server i've simple added in the freshclam config: DatabaseCustomURL https://blabla.domain.tld/safebrowsing.gdbI've expected to have more hits because now-days most of these URLs are used for phishing.Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: "G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> Date: 10/17/20 13:39 (GMT+02:00) To: Iulian Stan via clamav-users <clamav-users@lists.clamav.net> Cc: "G.W. Haywood" <clamav@jubileegroup.co.uk> Subject: Re: [clamav-users] Google safebrowsing types and usage questions Hi there,On Sat, 17 Oct 2020, Iulian Stan via clamav-users wrote:> "G.W. Haywood via clamav-users" wrote:> >> what's the best place to start?> > Definitions are stored in mysql and only the delta is downloaded> from google. After the download is successfully and the mysql data> is up-to-date you can generate the file for clamav.  Basically you> have two scripts:clamsbsync.py to update the mysql db from> googleclamsbwrite.py to create the .gdb file ...Thanks, that's a very clear explanation.  So instead of using freshclamyou just need to install mysql, Python, and some scripts.  Hmmm...> don't put too much faith in safebrowsing database. I rarely have any> hit even if there are almost 3 milion of signatures. In contrast i> have way more many hits with Sanesecurity.Perhaps this DB would be more important for people who use on-accessscanning and are less fastidious in their browsing habits.-- 73,Ged._______________________________________________clamav-users mailing listclamav-users@lists.clamav.nethttps://lists.clamav.net/mailman/listinfo/clamav-usersHelp us build a comprehensive ClamAV guide:https://github.com/vrtadmin/clamav-faqhttp://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
That documentation lives here: https://github.com/Cisco-Talos/clamav-faq/blob/master/faq/faq-safebrowsing.md

A pull request will allow me to review and approve.

Sent from my ? iPhone

> On Oct 17, 2020, at 07:56, Iulian Stan via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?
> Hi Ged,
>
> Yes, the definition that you can download via freshclam directly from clamav site is outdated ( from 2019).
>
> You need to use https://github.com/Cisco-Talos/clamav-safebrowsing
>
> My way to implement this was quite simple and i've also took the advantage of freshclam.
>
> On one central system i've installed the python/mysql/(other python libraries needed) and copy the generated clamav db to a place which is accessible via http/https.
> Then in every clamav server i've simple added in the freshclam config: DatabaseCustomURL https://blabla.domain.tld/safebrowsing.gdb
>
> I've expected to have more hits because now-days most of these URLs are used for phishing.
>
>
> Sent from my Samsung Galaxy smartphone.
>
>
> -------- Original message --------
> From: "G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net>
> Date: 10/17/20 13:39 (GMT+02:00)
> To: Iulian Stan via clamav-users <clamav-users@lists.clamav.net>
> Cc: "G.W. Haywood" <clamav@jubileegroup.co.uk>
> Subject: Re: [clamav-users] Google safebrowsing types and usage questions
>
> Hi there,
>
> On Sat, 17 Oct 2020, Iulian Stan via clamav-users wrote:
> > "G.W. Haywood via clamav-users" wrote:
> >
> >> what's the best place to start?
> >
> > Definitions are stored in mysql and only the delta is downloaded
> > from google. After the download is successfully and the mysql data
> > is up-to-date you can generate the file for clamav. Basically you
> > have two scripts:clamsbsync.py to update the mysql db from
> > googleclamsbwrite.py to create the .gdb file ...
>
> Thanks, that's a very clear explanation. So instead of using freshclam
> you just need to install mysql, Python, and some scripts. Hmmm...
>
> > don't put too much faith in safebrowsing database. I rarely have any
> > hit even if there are almost 3 milion of signatures. In contrast i
> > have way more many hits with Sanesecurity.
>
> Perhaps this DB would be more important for people who use on-access
> scanning and are less fastidious in their browsing habits.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi Joel,

On Sat, 17 Oct 2020, Joel Esler (jesler) via clamav-users wrote:

> That documentation lives here: https://github.com/Cisco-Talos/clamav-faq/blob/master/faq/faq-safebrowsing.md
>
> A pull request will allow me to review and approve.

I'm sorry Joel, I did try to use Github's Web interface to do the edit
but by the time I'd spent an hour searching for ways around a greyed-
out commit button I'd had enough. There's too little of life left to
me to spend the rest of it fighting with productivity tools.

Here's a suggested replacement for the file. If you don't like it,
please let me know in general terms how you'd like it improved and
I'll be happy to have a go (as long as you don't make me use Github).

8<----------------------------------------------------------------------
# Safebrowsing #

CURRENT STATUS at October 2020.

The safebrowsing feature has now been spun off into a related project.
It requires substantially more effort to implement safebrowsing than
simply enabling the relevant freshclam.conf configuration option.

Briefly, tools are needed to

1. Download the data from Google to a local mysql database using
Google's API [*];

2. produce a local copy of the safebrowsing database file in a form
suitable for use by the ClamAV tools;

3. distribute this database file to the systems which need it; and

4. optionally notify any clamd daemons of the change.

[*] For efficiency, the API permits downloading differences, in much
the same way that ClamAV itself uses .cdiff files.

Documentation can be found at

https://github.com/Cisco-Talos/clamav-safebrowsing


HISTORY

ClamAV 0.95 introduced support for the Google Safe Browsing database.

For use with ClamAV a copy of the database was packed inside the file
"safebrowsing.cvd" which was distributed in the same way as the other
ClamAV database files via the ClamAV mirror network. Downloading the
database was disabled by default, and the feature was to be enabled
only with extreme caution. In order to enable this feature it was
necessary to add the option `SafeBrowsing Yes` to freshclam.conf.
This would tell freshclam to download the safebrowsing.cvd database,
and when ClamAV found the database in the database directory it would
enable the safe browsing feature. To turn it off it was necessary to
remove the configuration option from freshclam.conf AND to remove the
safebrowsing files from the database directory. If clamd was running
it was necessary to restart it.

Updates to the safebrowsing.cvd database were discontinued in 2019 and
it was declared obsolete.
8<----------------------------------------------------------------------

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hello,

I was trying to understand why there are no hits for google
safe-browsing. I was looking around the API, what is downloaded in the
mysql DB and to be honest i got stuck.

I have two issues which i cannot understand and i can split the
questions in two:

(1) is related how clamav is actually treating .gdb files.

According to https://www.clamav.net/documents/phishsigs for .gdb you
need to have:

S:P:HostPrefix[:FuncLevelSpec]
S:F:Sha256hash[:FuncLevelSpec]

Let's make a test, i will take as example http://www.google.com/

sha256hash is:
dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf
4-byte prefix of the sha256 hash: dd014af5

creating db file:
#cat bla.gdb
S1:F:dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf
S1:P:dd014af5

Creating file to be tested:

#cat /tmp/clam.txt
http://www.google.com/
www.google.com
http://www.google.com/asdasdasd

Running scanner:

clamscan --debug -d bla.gdb /tmp/clam.txt
LibClamAV debug: searching for unrar, user-searchpath: /usr/lib64
LibClamAV debug: unrar support loaded from
/usr/lib64/libclamunrar_iface.so.9.0.4 libclamunrar_iface_so_9_0
LibClamAV debug: Initialized 0.102.4 engine
LibClamAV debug: Initializing phishcheck module
LibClamAV debug: Phishcheck: Compiling regex: ^
*(http|https|ftp:(//)?)?[0-9]{1,3}(\.[0-9]{1,3}){3}[/?:]? *$
LibClamAV debug: Phishcheck module initialized
LibClamAV debug: Bytecode initialized in interpreter mode
LibClamAV debug: Loading regex_list
LibClamAV debug: bla.gdb loaded
LibClamAV debug: Initializing engine->root[0]
LibClamAV debug: Initializing AC pattern matcher of root[0]
LibClamAV debug: cli_initroots: Initializing BM tables of root[0]
LibClamAV debug: Initializing engine->root[1]
LibClamAV debug: Initializing AC pattern matcher of root[1]
LibClamAV debug: cli_initroots: Initializing BM tables of root[1]
LibClamAV debug: Initializing engine->root[2]
LibClamAV debug: Initializing AC pattern matcher of root[2]
LibClamAV debug: Initializing engine->root[3]
LibClamAV debug: Initializing AC pattern matcher of root[3]
LibClamAV debug: Initializing engine->root[4]
LibClamAV debug: Initializing AC pattern matcher of root[4]
LibClamAV debug: Initializing engine->root[5]
LibClamAV debug: Initializing AC pattern matcher of root[5]
LibClamAV debug: Initializing engine->root[6]
LibClamAV debug: Initializing AC pattern matcher of root[6]
LibClamAV debug: Initializing engine->root[7]
LibClamAV debug: Initializing AC pattern matcher of root[7]
LibClamAV debug: Initializing engine->root[8]
LibClamAV debug: Initializing AC pattern matcher of root[8]
LibClamAV debug: Initializing engine->root[9]
LibClamAV debug: Initializing AC pattern matcher of root[9]
LibClamAV debug: Initializing engine->root[10]
LibClamAV debug: Initializing AC pattern matcher of root[10]
LibClamAV debug: Initializing engine->root[11]
LibClamAV debug: Initializing AC pattern matcher of root[11]
LibClamAV debug: Initializing engine->root[12]
LibClamAV debug: Initializing AC pattern matcher of root[12]
LibClamAV debug: Initializing engine->root[13]
LibClamAV debug: Initializing AC pattern matcher of root[13]
LibClamAV debug: Initializing engine->root[14]
LibClamAV debug: Initializing AC pattern matcher of root[14]
LibClamAV debug: Loaded 155 filetype definitions
LibClamAV debug: Using filter for trie 0
LibClamAV debug: Matcher[0]: GENERIC: AC sigs: 82 (reloff: 1, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 32
LibClamAV debug: Using filter for trie 1
LibClamAV debug: Matcher[1]: PE: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0
LibClamAV debug: Matcher[2]: OLE2: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[3]: HTML: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Using filter for trie 4
LibClamAV debug: Matcher[4]: MAIL: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[5]: GRAPHICS: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[6]: ELF: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Using filter for trie 7
LibClamAV debug: Matcher[7]: ASCII: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[8]: NOT USED: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[9]: MACH-O: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[10]: PDF: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[11]: FLASH: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[12]: JAVA: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[13]: INTERNAL: AC sigs: 0 (reloff: 0, absoff:
0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[14]: OTHER: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Building regex list
LibClamAV debug: Using filter for trie 0
LibClamAV debug: hashtab: Freeing hashset, elements: 1, capacity:
1048576
LibClamAV debug: Dynamic engine configuration settings:
LibClamAV debug: --------------------------------------
LibClamAV debug: Module PE: On
LibClamAV debug: * Submodule PARITE: On
LibClamAV debug: * Submodule KRIZ: On
LibClamAV debug: * Submodule MAGISTR: On
LibClamAV debug: * Submodule POLIPOS: On
LibClamAV debug: * Submodule MD5SECT: On
LibClamAV debug: * Submodule UPX: On
LibClamAV debug: * Submodule FSG: On
LibClamAV debug: * Submodule SWIZZOR: ** Off **
LibClamAV debug: * Submodule PETITE: On
LibClamAV debug: * Submodule PESPIN: On
LibClamAV debug: * Submodule YC: On
LibClamAV debug: * Submodule WWPACK: On
LibClamAV debug: * Submodule NSPACK: On
LibClamAV debug: * Submodule MEW: On
LibClamAV debug: * Submodule UPACK: On
LibClamAV debug: * Submodule ASPACK: On
LibClamAV debug: * Submodule CATALOG: On
LibClamAV debug: * Submodule CERTS: On
LibClamAV debug: * Submodule MATCHICON: On
LibClamAV debug: * Submodule IMPTBL: On
LibClamAV debug: Module ELF: On
LibClamAV debug: Module MACHO: On
LibClamAV debug: Module ARCHIVE: On
LibClamAV debug: * Submodule RAR: On
LibClamAV debug: * Submodule ZIP: On
LibClamAV debug: * Submodule GZIP: On
LibClamAV debug: * Submodule BZIP: On
LibClamAV debug: * Submodule ARJ: On
LibClamAV debug: * Submodule SZDD: On
LibClamAV debug: * Submodule CAB: On
LibClamAV debug: * Submodule CHM: On
LibClamAV debug: * Submodule OLE2: On
LibClamAV debug: * Submodule TAR: On
LibClamAV debug: * Submodule CPIO: On
LibClamAV debug: * Submodule BINHEX: On
LibClamAV debug: * Submodule SIS: On
LibClamAV debug: * Submodule NSIS: On
LibClamAV debug: * Submodule AUTOIT: On
LibClamAV debug: * Submodule ISHIELD: On
LibClamAV debug: * Submodule 7zip: On
LibClamAV debug: * Submodule ISO9660: On
LibClamAV debug: * Submodule DMG: On
LibClamAV debug: * Submodule XAR: On
LibClamAV debug: * Submodule HFSPLUS: On
LibClamAV debug: * Submodule XZ: On
LibClamAV debug: * Submodule PASSWD: On
LibClamAV debug: * Submodule MBR: On
LibClamAV debug: * Submodule GPT: On
LibClamAV debug: * Submodule APM: On
LibClamAV debug: * Submodule EGG: On
LibClamAV debug: Module DOCUMENT: On
LibClamAV debug: * Submodule HTML: On
LibClamAV debug: * Submodule RTF: On
LibClamAV debug: * Submodule PDF: On
LibClamAV debug: * Submodule SCRIPT: On
LibClamAV debug: * Submodule HTMLSKIPRAW: On
LibClamAV debug: * Submodule JSNORM: On
LibClamAV debug: * Submodule SWF: On
LibClamAV debug: * Submodule OOXML: On
LibClamAV debug: * Submodule MSPML: On
LibClamAV debug: * Submodule HWP: On
LibClamAV debug: Module MAIL: On
LibClamAV debug: * Submodule MBOX: On
LibClamAV debug: * Submodule TNEF: On
LibClamAV debug: Module OTHER: On
LibClamAV debug: * Submodule UUENCODED: On
LibClamAV debug: * Submodule SCRENC: On
LibClamAV debug: * Submodule RIFF: On
LibClamAV debug: * Submodule JPEG: On
LibClamAV debug: * Submodule CRYPTFF: On
LibClamAV debug: * Submodule DLP: On
LibClamAV debug: * Submodule MYDOOMLOG: On
LibClamAV debug: * Submodule PREFILTERING: On
LibClamAV debug: * Submodule PDFNAMEOBJ: On
LibClamAV debug: * Submodule PRTNINTXN: On
LibClamAV debug: * Submodule LZW: On
LibClamAV debug: Module PHISHING On
LibClamAV debug: * Submodule ENGINE: On
LibClamAV debug: * Submodule ENTCONV: On
LibClamAV debug: Module BYTECODE On
LibClamAV debug: * Submodule INTERPRETER: On
LibClamAV debug: * Submodule JIT X86: On
LibClamAV debug: * Submodule JIT PPC: On
LibClamAV debug: * Submodule JIT ARM: ** Off **
LibClamAV debug: Module STATS Off
LibClamAV debug: Module PCRE On
LibClamAV debug: * Submodule SUPPORT: On
LibClamAV debug: * Submodule OPTIONS: On
LibClamAV debug: * Submodule GLOBAL: On
LibClamAV debug: pool memory used: 7.155 MB
LibClamAV debug: No bytecodes loaded, not running builtin test
LibClamAV debug: Checking realpath of /tmp/clam.txt
LibClamAV debug: in cli_magic_scandesc (reclevel: 0/16)
LibClamAV debug: Recognized ASCII text
LibClamAV debug: cache_check: 0bf05034b9d1bb3690d44da0f9e358d1 is
negative
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: in cli_scanscript()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 0bf05034b9d1bb3690d44da0f9e358d1 (level 0)
/tmp/clam.txt: OK
LibClamAV debug: Cleaning up phishcheck
LibClamAV debug: Freeing phishcheck struct
LibClamAV debug: Phishcheck cleaned up

----------- SCAN SUMMARY -----------
Known viruses: 2
Engine version: 0.102.4
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.016 sec (0 m 0 s)

As you can see it does not detect the URL as expect.

Any thouths ?

(2.) You can check if an URL is present in google safe browsing using
fallowing methods:

a) via a browser:
https://transparencyreport.google.com/safe-browsing/search and you
simply put the URL there
b) via look-up API (
https://developers.google.com/safe-browsing/v4/lookup-api )

I used curl and it's working ok:

example:(remove ((()))) in order to work from url in order to work and
also put your own API_KEY)

curl --header "Content-Type: application/json" \
--request POST \
--data ' {
"client": {
"clientId": "balbla",
"clientVersion": "1.5.2"
},
"threatInfo": {
"threatTypes": ["MALWARE", "SOCIAL_ENGINEERING"],
"platformTypes": ["WINDOWS"],
"threatEntryTypes": ["URL"],
"threatEntries": [
{"url": "http://(((())))pagesblokd3((((.))))ga/"}
]
}
}
' \
https://safebrowsing.googleapis.com/v4/threatMatches:find?key=YOUR_API_KEY


c) via update API (fullHashes.find)
https://developers.google.com/safe-browsing/v4/update-api
This method use hashes to determine if the URL is safe or not.
Unfortunatelly i didn't managed to create such request for the above url
to get the same result ( in this case url is not safe)

Further more i was looking in the mysql db(table sbclient_v4_hashes) for
the hash and the prefix.
I didn't found the sha256(or the prefix) for the above link. For obvious
reason those cannot be found also in the created clamav signature file.

Do you have any clue why the sha256 for the above link is not in the
mysql db ? Or, did you manage to create a API query using hashed url(yes
i know is base64-encoded) and get the same result as (a) and (b) ?

Basically in this moment i don't understand if the problem is on
clamav(see point 1), or google(maybe he is not updating all the hashes
?) .... or me ? :)))

Best regards,
Iulian Stan
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi there,

Just some thoughts, as you asked. Sorry is isn't more helpful.

On Mon, 19 Oct 2020, iulian stan via clamav-users wrote:

> #cat bla.gdb
> S1:F:dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf
> S1:P:dd014af5
> Creating file to be tested:
> #cat /tmp/clam.txt
> http://www.google.com/
> www.google.com
> http://www.google.com/asdasdasd

I repeated your tests with 0.103-rc2 and got the same results. I
looked for obvious things like line terminators being included by
accident, but I didn't find anything.

> Running scanner:
> clamscan --debug -d bla.gdb /tmp/clam.txt
> LibClamAV debug: Module <....> On

I wondered if there's a module that should be being loaded and isn't.

> LibClamAV debug: Recognized ASCII text

I wondered does it need to recognize the file as HTML, and also if
there's some length limit below which the scanner won't bother doing
the scan (I've seen mention of something like that when I've been
reading the code looking for something else) but I tried wrapping your
text in some html tags, and added some padding, and it made no
difference. This is incidentally one of those cases where the values
printed in the output for "Data scanned" and "Data read" could be more
useful...

8<----------------------------------------------------------------------
...
LibClamAV debug: Recognized ASCII text
LibClamAV debug: Matched signature for file type HTML data at 0
...
----------- SCAN SUMMARY -----------
Known viruses: 2
Engine version: 0.103.0-rc2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.20 MB
Data read: 0.10 MB (ratio 2.00:1)
8<----------------------------------------------------------------------

Lastly

> ----------- SCAN SUMMARY -----------
> Known viruses: 2

This doesn't seem right to me. There's really only one signature.

Basically I haven't seen anything here which might make me think the
problem is you, but I don't use the safebrowsing stuff so I don't have
the experience (and I don't have the time right now) to investigate it
further. It seems to me that even if there isn't something wrong with
clamd (which I guess means that it's faulty documentation) it really
shouldn't be this difficult - that alone would make it worth a report
to the ClamAV Bugzilla.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Dear Ged/All,

After a beer things started to look more clear :)

You were right about something: indeed clamav is looking for something
before starting to look after URL but it's actually looking for what
should be the start of email headers. In short words is looking for:
"From someone".
Basically the test can be:
echo -e "From test\n\n http://www.google.com/" | clamscan -d bla.gdb -
or
echo -e "From test\n\n<a href=http://www.google.com/>test</a>" |
clamscan -d bla.gdb -

with the fallowing result:
----------- SCAN SUMMARY -----------
Known viruses: 2
Engine version: 0.102.4
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.051 sec (0 m 0 s)

I totally agree with you that "Know viruses" should be 1 but this is
another story for another time.


Now comes the funny part which explains why i didn't found the sha256
hash in my mysql and also why the above test will fail if you don't
create the hash correctly.

If you read https://developers.google.com/safe-browsing/v4/urls-hashing
(very carefully, not like I've did in the beginning) you will see that
you can create multiple hashes for the same url but you first need to
strip http[s]://

The same is also seen in the clamav debugging.
If we take for example url
"http://www.google.com/jhgfedwsqasdfgh/234tewdas.txt" the debug will be.

LibClamAV debug: getHrefs: html_normalise_mem returned
LibClamAV debug: Phishcheck:Checking url
http://www.google.com/jhgfedwsqasdfgh/234tewdas.txt</p>->
LibClamAV debug: Looking up hash
DDEF6ACD0DF553A77CBC6B3537BDAA766E0CD819733D0B712AFD9A41B5888AB5 for
google.com/(11)jhgfedwsqasdfgh/234tewdas.txt</p>(31)
LibClamAV debug: Looking up hash
B8047D0B3763184FF29E17D4F649BA05E469538C40018FBB901437822F0066C6 for
www.google.com/(15)jhgfedwsqasdfgh/234tewdas.txt</p>(31)
LibClamAV debug: Looking up hash
6D92531661EBF105F3C03BE8EA6C7E585F2A1603B5FF4D501BC0846755355018 for
google.com/(11)jhgfedwsqasdfgh/234tewdas.txt</p>(16)
LibClamAV debug: Looking up hash
DA983C0FAA7401A96BBBF6068F29762557B63F0811A0418BC046D95795999AFB for
www.google.com/(15)jhgfedwsqasdfgh/234tewdas.txt</p>(16)
LibClamAV debug: Looking up hash
88981E6263BE34A6C0B53ADA73D168B68828DD643723D34A812E9F8A6ABB5EE9 for
google.com/(11)jhgfedwsqasdfgh/234tewdas.txt</p>(0)
LibClamAV debug: Looking up hash
BC9A8F2B6FFFD58571E188BB110545F8FB3AF51CDF1A63696D505A9870A85BE5 for
www.google.com/(15)jhgfedwsqasdfgh/234tewdas.txt</p>(0)
LibClamAV debug: This hash matched:
BC9A8F2B6FFFD58571E188BB110545F8FB3AF51CDF1A63696D505A9870A85BE5
LibClamAV debug: Hash matched for:
http://www.google.com/jhgfedwsqasdfgh/234tewdas.txt</p>
LibClamAV debug: Phishcheck: Phishing scan result: Blacklisted
LibClamAV debug: blobDestroy



Long story short, safebrowsing is working ok but there are no hits which
is quite surprising i can say seeing the magnitude of the database
entries and the scam/phishing flowing trough emails now-days.

---
Best regards,
Iulian Stan


On 2020-10-19 20:01, G.W. Haywood via clamav-users wrote:
> Hi there,
>
> Just some thoughts, as you asked. Sorry is isn't more helpful.
>
> On Mon, 19 Oct 2020, iulian stan via clamav-users wrote:
>
>> #cat bla.gdb
>> S1:F:dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf
>> S1:P:dd014af5 Creating file to be tested: #cat /tmp/clam.txt
>> http://www.google.com/
>> www.google.com
>> http://www.google.com/asdasdasd
>
> I repeated your tests with 0.103-rc2 and got the same results. I
> looked for obvious things like line terminators being included by
> accident, but I didn't find anything.
>
>> Running scanner: clamscan --debug -d bla.gdb /tmp/clam.txt
>> LibClamAV debug: Module <....> On
>
> I wondered if there's a module that should be being loaded and isn't.
>
>> LibClamAV debug: Recognized ASCII text
>
> I wondered does it need to recognize the file as HTML, and also if
> there's some length limit below which the scanner won't bother doing
> the scan (I've seen mention of something like that when I've been
> reading the code looking for something else) but I tried wrapping your
> text in some html tags, and added some padding, and it made no
> difference. This is incidentally one of those cases where the values
> printed in the output for "Data scanned" and "Data read" could be more
> useful...
>
> 8<----------------------------------------------------------------------
> ...
> LibClamAV debug: Recognized ASCII text
> LibClamAV debug: Matched signature for file type HTML data at 0
> ...
> ----------- SCAN SUMMARY -----------
> Known viruses: 2
> Engine version: 0.103.0-rc2
> Scanned directories: 0
> Scanned files: 1
> Infected files: 0
> Data scanned: 0.20 MB
> Data read: 0.10 MB (ratio 2.00:1)
> 8<----------------------------------------------------------------------
>
> Lastly
>
>> ----------- SCAN SUMMARY -----------
>> Known viruses: 2
>
> This doesn't seem right to me. There's really only one signature.
>
> Basically I haven't seen anything here which might make me think the
> problem is you, but I don't use the safebrowsing stuff so I don't have
> the experience (and I don't have the time right now) to investigate it
> further. It seems to me that even if there isn't something wrong with
> clamd (which I guess means that it's faulty documentation) it really
> shouldn't be this difficult - that alone would make it worth a report
> to the ClamAV Bugzilla.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Hi there,

On Tue, 20 Oct 2020, iulian stan via clamav-users wrote:

> After a beer things started to look more clear :)

Why didn't I think of that? :)

> You were right about something: indeed clamav is looking for something before
> starting to look after URL but it's actually looking for what should be the
> start of email headers. In short words is looking for: "From someone".

Oh, bother, I didn't think of that either - and I even do it in my
milter, but it's a while since I wrote it. So is this safebrowsing
thing only going to work for links in emails? That seems odd, I'd
somehow thought it might be useful for people using clamd to scan
their HTTP traffic in in real time. A sort of scan-on-access. Not
that I'm advocating such a thing, but I've seen it mentioned.

> If you read ... carefully ...

That often works better. :)

> Long story short, safebrowsing is working ok ...

Fine - but I still think the documentation needs work.

> ... there are no hits which is quite surprising surprising i can say
> seeing the magnitude of the database entries and the scam/phishing
> flowing trough emails now-days.

It certainly seems true that the volume is increasing. Recently I've
been seeing thousands just from Hotmail accounts. They're trivial to
stop but I'd expect Microsoft to do a lot better at their servers. As
for Google's safebrowsing, as I said I don't use it - but given that
Hotmail's number one position in the spam charts here was previously
occupied by Gmail, I can't say that I plan to spend much time on it.

Glad you got it sorted out. :)

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
?On 10/19/20, 3:33 PM, "clamav-users on behalf of G.W. Haywood via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

> Long story short, safebrowsing is working ok ...

Fine - but I still think the documentation needs work.

We'd always love help with the documentation!
Please submit PR's to the clamav-faq project: https://github.com/Cisco-Talos/clamav-faq/blob/master/manual/UserManual/Signatures/PhishSigs.md#gdb-format

Regards,
Micah


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
It does seem we forgot to update the safebrowsing doc on clamav.net. This is the source, if you're up for working on it: https://github.com/Cisco-Talos/clamav-faq/blob/master/faq/faq-safebrowsing.md
Sorry about the extended delays between emails; I'm presently preoccupied with debugging the PNG parser. Give me a shout any time though.

Regards,
Micah


?On 10/17/20, 2:22 AM, "G.W. Haywood" <clamav@jubileegroup.co.uk> wrote:

Hi Micah,

On Fri, 16 Oct 2020, Micah Snyder (micasnyd) via clamav-users wrote:

> The clamav-safebrowsing tool was developed by another team and
> gifted to my team for OSS maintenance.

As if you didn't have enough to do already!

> ... The mysql database exists for precisely that reason.
> ... the one safebrowsing.cld file is obsolete.

There's something I'm missing. None of this seems to be mentioned at

https://www.clamav.net/documents/safebrowsing

If you'd like a documentation patch I can do some reading and give it
a go, what's the best place to start?

--

73,
Ged.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Oops! Just saw this reply. Your suggestion looks like a good start.

I don't want to step on toes - Joel, are you running with this or would you like me to put in a PR with something derived from Ged's suggestion?

-Micah

?On 10/17/20, 8:43 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

Hi Joel,

On Sat, 17 Oct 2020, Joel Esler (jesler) via clamav-users wrote:

> That documentation lives here: https://github.com/Cisco-Talos/clamav-faq/blob/master/faq/faq-safebrowsing.md
>
> A pull request will allow me to review and approve.

I'm sorry Joel, I did try to use Github's Web interface to do the edit
but by the time I'd spent an hour searching for ways around a greyed-
out commit button I'd had enough. There's too little of life left to
me to spend the rest of it fighting with productivity tools.

Here's a suggested replacement for the file. If you don't like it,
please let me know in general terms how you'd like it improved and
I'll be happy to have a go (as long as you don't make me use Github).

8<----------------------------------------------------------------------
# Safebrowsing #

CURRENT STATUS at October 2020.

The safebrowsing feature has now been spun off into a related project.
It requires substantially more effort to implement safebrowsing than
simply enabling the relevant freshclam.conf configuration option.

Briefly, tools are needed to

1. Download the data from Google to a local mysql database using
Google's API [*];

2. produce a local copy of the safebrowsing database file in a form
suitable for use by the ClamAV tools;

3. distribute this database file to the systems which need it; and

4. optionally notify any clamd daemons of the change.

[*] For efficiency, the API permits downloading differences, in much
the same way that ClamAV itself uses .cdiff files.

Documentation can be found at

https://github.com/Cisco-Talos/clamav-safebrowsing


HISTORY

ClamAV 0.95 introduced support for the Google Safe Browsing database.

For use with ClamAV a copy of the database was packed inside the file
"safebrowsing.cvd" which was distributed in the same way as the other
ClamAV database files via the ClamAV mirror network. Downloading the
database was disabled by default, and the feature was to be enabled
only with extreme caution. In order to enable this feature it was
necessary to add the option `SafeBrowsing Yes` to freshclam.conf.
This would tell freshclam to download the safebrowsing.cvd database,
and when ClamAV found the database in the database directory it would
enable the safe browsing feature. To turn it off it was necessary to
remove the configuration option from freshclam.conf AND to remove the
safebrowsing files from the database directory. If clamd was running
it was necessary to restart it.

Updates to the safebrowsing.cvd database were discontinued in 2019 and
it was declared obsolete.
8<----------------------------------------------------------------------

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Google safebrowsing types and usage questions [ In reply to ]
Added.

> On Oct 17, 2020, at 11:41 AM, G.W. Haywood <clamav@jubileegroup.co.uk> wrote:
>
> Hi Joel,
>
> On Sat, 17 Oct 2020, Joel Esler (jesler) via clamav-users wrote:
>
>> That documentation lives here: https://github.com/Cisco-Talos/clamav-faq/blob/master/faq/faq-safebrowsing.md
>>
>> A pull request will allow me to review and approve.
>
> I'm sorry Joel, I did try to use Github's Web interface to do the edit
> but by the time I'd spent an hour searching for ways around a greyed-
> out commit button I'd had enough. There's too little of life left to
> me to spend the rest of it fighting with productivity tools.
>
> Here's a suggested replacement for the file. If you don't like it,
> please let me know in general terms how you'd like it improved and
> I'll be happy to have a go (as long as you don't make me use Github).
>
> 8<----------------------------------------------------------------------
> # Safebrowsing #
>
> CURRENT STATUS at October 2020.
>
> The safebrowsing feature has now been spun off into a related project.
> It requires substantially more effort to implement safebrowsing than
> simply enabling the relevant freshclam.conf configuration option.
>
> Briefly, tools are needed to
>
> 1. Download the data from Google to a local mysql database using
> Google's API [*];
>
> 2. produce a local copy of the safebrowsing database file in a form
> suitable for use by the ClamAV tools;
>
> 3. distribute this database file to the systems which need it; and
>
> 4. optionally notify any clamd daemons of the change.
>
> [*] For efficiency, the API permits downloading differences, in much
> the same way that ClamAV itself uses .cdiff files.
>
> Documentation can be found at
>
> https://github.com/Cisco-Talos/clamav-safebrowsing
>
>
> HISTORY
>
> ClamAV 0.95 introduced support for the Google Safe Browsing database.
>
> For use with ClamAV a copy of the database was packed inside the file
> "safebrowsing.cvd" which was distributed in the same way as the other
> ClamAV database files via the ClamAV mirror network. Downloading the
> database was disabled by default, and the feature was to be enabled
> only with extreme caution. In order to enable this feature it was
> necessary to add the option `SafeBrowsing Yes` to freshclam.conf.
> This would tell freshclam to download the safebrowsing.cvd database,
> and when ClamAV found the database in the database directory it would
> enable the safe browsing feature. To turn it off it was necessary to
> remove the configuration option from freshclam.conf AND to remove the
> safebrowsing files from the database directory. If clamd was running
> it was necessary to restart it.
>
> Updates to the safebrowsing.cvd database were discontinued in 2019 and
> it was declared obsolete.
> 8<----------------------------------------------------------------------
>
> --
>
> 73,
> Ged.