Hello,
I was trying to understand why there are no hits for google
safe-browsing. I was looking around the API, what is downloaded in the
mysql DB and to be honest i got stuck.
I have two issues which i cannot understand and i can split the
questions in two:
(1) is related how clamav is actually treating .gdb files.
According to
https://www.clamav.net/documents/phishsigs for .gdb you
need to have:
S:P:HostPrefix[:FuncLevelSpec]
S:F:Sha256hash[:FuncLevelSpec]
Let's make a test, i will take as example
http://www.google.com/ sha256hash is:
dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf
4-byte prefix of the sha256 hash: dd014af5
creating db file:
#cat bla.gdb
S1:F:dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf
S1:P:dd014af5
Creating file to be tested:
#cat /tmp/clam.txt
http://www.google.com/ www.google.com
http://www.google.com/asdasdasd Running scanner:
clamscan --debug -d bla.gdb /tmp/clam.txt
LibClamAV debug: searching for unrar, user-searchpath: /usr/lib64
LibClamAV debug: unrar support loaded from
/usr/lib64/libclamunrar_iface.so.9.0.4 libclamunrar_iface_so_9_0
LibClamAV debug: Initialized 0.102.4 engine
LibClamAV debug: Initializing phishcheck module
LibClamAV debug: Phishcheck: Compiling regex: ^
*(http|https|ftp:(//)?)?[0-9]{1,3}(\.[0-9]{1,3}){3}[/?:]? *$
LibClamAV debug: Phishcheck module initialized
LibClamAV debug: Bytecode initialized in interpreter mode
LibClamAV debug: Loading regex_list
LibClamAV debug: bla.gdb loaded
LibClamAV debug: Initializing engine->root[0]
LibClamAV debug: Initializing AC pattern matcher of root[0]
LibClamAV debug: cli_initroots: Initializing BM tables of root[0]
LibClamAV debug: Initializing engine->root[1]
LibClamAV debug: Initializing AC pattern matcher of root[1]
LibClamAV debug: cli_initroots: Initializing BM tables of root[1]
LibClamAV debug: Initializing engine->root[2]
LibClamAV debug: Initializing AC pattern matcher of root[2]
LibClamAV debug: Initializing engine->root[3]
LibClamAV debug: Initializing AC pattern matcher of root[3]
LibClamAV debug: Initializing engine->root[4]
LibClamAV debug: Initializing AC pattern matcher of root[4]
LibClamAV debug: Initializing engine->root[5]
LibClamAV debug: Initializing AC pattern matcher of root[5]
LibClamAV debug: Initializing engine->root[6]
LibClamAV debug: Initializing AC pattern matcher of root[6]
LibClamAV debug: Initializing engine->root[7]
LibClamAV debug: Initializing AC pattern matcher of root[7]
LibClamAV debug: Initializing engine->root[8]
LibClamAV debug: Initializing AC pattern matcher of root[8]
LibClamAV debug: Initializing engine->root[9]
LibClamAV debug: Initializing AC pattern matcher of root[9]
LibClamAV debug: Initializing engine->root[10]
LibClamAV debug: Initializing AC pattern matcher of root[10]
LibClamAV debug: Initializing engine->root[11]
LibClamAV debug: Initializing AC pattern matcher of root[11]
LibClamAV debug: Initializing engine->root[12]
LibClamAV debug: Initializing AC pattern matcher of root[12]
LibClamAV debug: Initializing engine->root[13]
LibClamAV debug: Initializing AC pattern matcher of root[13]
LibClamAV debug: Initializing engine->root[14]
LibClamAV debug: Initializing AC pattern matcher of root[14]
LibClamAV debug: Loaded 155 filetype definitions
LibClamAV debug: Using filter for trie 0
LibClamAV debug: Matcher[0]: GENERIC: AC sigs: 82 (reloff: 1, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 32
LibClamAV debug: Using filter for trie 1
LibClamAV debug: Matcher[1]: PE: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0
LibClamAV debug: Matcher[2]: OLE2: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[3]: HTML: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Using filter for trie 4
LibClamAV debug: Matcher[4]: MAIL: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[5]: GRAPHICS: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[6]: ELF: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Using filter for trie 7
LibClamAV debug: Matcher[7]: ASCII: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[8]: NOT USED: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[9]: MACH-O: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[10]: PDF: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[11]: FLASH: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[12]: JAVA: AC sigs: 0 (reloff: 0, absoff: 0) BM
sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen
0 (ac_only mode)
LibClamAV debug: Matcher[13]: INTERNAL: AC sigs: 0 (reloff: 0, absoff:
0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[14]: OTHER: AC sigs: 0 (reloff: 0, absoff: 0)
BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0)
maxpatlen 0 (ac_only mode)
LibClamAV debug: Building regex list
LibClamAV debug: Using filter for trie 0
LibClamAV debug: hashtab: Freeing hashset, elements: 1, capacity:
1048576
LibClamAV debug: Dynamic engine configuration settings:
LibClamAV debug: --------------------------------------
LibClamAV debug: Module PE: On
LibClamAV debug: * Submodule PARITE: On
LibClamAV debug: * Submodule KRIZ: On
LibClamAV debug: * Submodule MAGISTR: On
LibClamAV debug: * Submodule POLIPOS: On
LibClamAV debug: * Submodule MD5SECT: On
LibClamAV debug: * Submodule UPX: On
LibClamAV debug: * Submodule FSG: On
LibClamAV debug: * Submodule SWIZZOR: ** Off **
LibClamAV debug: * Submodule PETITE: On
LibClamAV debug: * Submodule PESPIN: On
LibClamAV debug: * Submodule YC: On
LibClamAV debug: * Submodule WWPACK: On
LibClamAV debug: * Submodule NSPACK: On
LibClamAV debug: * Submodule MEW: On
LibClamAV debug: * Submodule UPACK: On
LibClamAV debug: * Submodule ASPACK: On
LibClamAV debug: * Submodule CATALOG: On
LibClamAV debug: * Submodule CERTS: On
LibClamAV debug: * Submodule MATCHICON: On
LibClamAV debug: * Submodule IMPTBL: On
LibClamAV debug: Module ELF: On
LibClamAV debug: Module MACHO: On
LibClamAV debug: Module ARCHIVE: On
LibClamAV debug: * Submodule RAR: On
LibClamAV debug: * Submodule ZIP: On
LibClamAV debug: * Submodule GZIP: On
LibClamAV debug: * Submodule BZIP: On
LibClamAV debug: * Submodule ARJ: On
LibClamAV debug: * Submodule SZDD: On
LibClamAV debug: * Submodule CAB: On
LibClamAV debug: * Submodule CHM: On
LibClamAV debug: * Submodule OLE2: On
LibClamAV debug: * Submodule TAR: On
LibClamAV debug: * Submodule CPIO: On
LibClamAV debug: * Submodule BINHEX: On
LibClamAV debug: * Submodule SIS: On
LibClamAV debug: * Submodule NSIS: On
LibClamAV debug: * Submodule AUTOIT: On
LibClamAV debug: * Submodule ISHIELD: On
LibClamAV debug: * Submodule 7zip: On
LibClamAV debug: * Submodule ISO9660: On
LibClamAV debug: * Submodule DMG: On
LibClamAV debug: * Submodule XAR: On
LibClamAV debug: * Submodule HFSPLUS: On
LibClamAV debug: * Submodule XZ: On
LibClamAV debug: * Submodule PASSWD: On
LibClamAV debug: * Submodule MBR: On
LibClamAV debug: * Submodule GPT: On
LibClamAV debug: * Submodule APM: On
LibClamAV debug: * Submodule EGG: On
LibClamAV debug: Module DOCUMENT: On
LibClamAV debug: * Submodule HTML: On
LibClamAV debug: * Submodule RTF: On
LibClamAV debug: * Submodule PDF: On
LibClamAV debug: * Submodule SCRIPT: On
LibClamAV debug: * Submodule HTMLSKIPRAW: On
LibClamAV debug: * Submodule JSNORM: On
LibClamAV debug: * Submodule SWF: On
LibClamAV debug: * Submodule OOXML: On
LibClamAV debug: * Submodule MSPML: On
LibClamAV debug: * Submodule HWP: On
LibClamAV debug: Module MAIL: On
LibClamAV debug: * Submodule MBOX: On
LibClamAV debug: * Submodule TNEF: On
LibClamAV debug: Module OTHER: On
LibClamAV debug: * Submodule UUENCODED: On
LibClamAV debug: * Submodule SCRENC: On
LibClamAV debug: * Submodule RIFF: On
LibClamAV debug: * Submodule JPEG: On
LibClamAV debug: * Submodule CRYPTFF: On
LibClamAV debug: * Submodule DLP: On
LibClamAV debug: * Submodule MYDOOMLOG: On
LibClamAV debug: * Submodule PREFILTERING: On
LibClamAV debug: * Submodule PDFNAMEOBJ: On
LibClamAV debug: * Submodule PRTNINTXN: On
LibClamAV debug: * Submodule LZW: On
LibClamAV debug: Module PHISHING On
LibClamAV debug: * Submodule ENGINE: On
LibClamAV debug: * Submodule ENTCONV: On
LibClamAV debug: Module BYTECODE On
LibClamAV debug: * Submodule INTERPRETER: On
LibClamAV debug: * Submodule JIT X86: On
LibClamAV debug: * Submodule JIT PPC: On
LibClamAV debug: * Submodule JIT ARM: ** Off **
LibClamAV debug: Module STATS Off
LibClamAV debug: Module PCRE On
LibClamAV debug: * Submodule SUPPORT: On
LibClamAV debug: * Submodule OPTIONS: On
LibClamAV debug: * Submodule GLOBAL: On
LibClamAV debug: pool memory used: 7.155 MB
LibClamAV debug: No bytecodes loaded, not running builtin test
LibClamAV debug: Checking realpath of /tmp/clam.txt
LibClamAV debug: in cli_magic_scandesc (reclevel: 0/16)
LibClamAV debug: Recognized ASCII text
LibClamAV debug: cache_check: 0bf05034b9d1bb3690d44da0f9e358d1 is
negative
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: in cli_scanscript()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 0bf05034b9d1bb3690d44da0f9e358d1 (level 0)
/tmp/clam.txt: OK
LibClamAV debug: Cleaning up phishcheck
LibClamAV debug: Freeing phishcheck struct
LibClamAV debug: Phishcheck cleaned up
----------- SCAN SUMMARY -----------
Known viruses: 2
Engine version: 0.102.4
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.016 sec (0 m 0 s)
As you can see it does not detect the URL as expect.
Any thouths ?
(2.) You can check if an URL is present in google safe browsing using
fallowing methods:
a) via a browser:
https://transparencyreport.google.com/safe-browsing/search and you
simply put the URL there
b) via look-up API (
https://developers.google.com/safe-browsing/v4/lookup-api )
I used curl and it's working ok:
example:(remove ((()))) in order to work from url in order to work and
also put your own API_KEY)
curl --header "Content-Type: application/json" \
--request POST \
--data ' {
"client": {
"clientId": "balbla",
"clientVersion": "1.5.2"
},
"threatInfo": {
"threatTypes": ["MALWARE", "SOCIAL_ENGINEERING"],
"platformTypes": ["WINDOWS"],
"threatEntryTypes": ["URL"],
"threatEntries": [
{"url": "http://(((())))pagesblokd3((((.))))ga/"}
]
}
}
' \
https://safebrowsing.googleapis.com/v4/threatMatches:find?key=YOUR_API_KEY c) via update API (fullHashes.find)
https://developers.google.com/safe-browsing/v4/update-api This method use hashes to determine if the URL is safe or not.
Unfortunatelly i didn't managed to create such request for the above url
to get the same result ( in this case url is not safe)
Further more i was looking in the mysql db(table sbclient_v4_hashes) for
the hash and the prefix.
I didn't found the sha256(or the prefix) for the above link. For obvious
reason those cannot be found also in the created clamav signature file.
Do you have any clue why the sha256 for the above link is not in the
mysql db ? Or, did you manage to create a API query using hashed url(yes
i know is base64-encoded) and get the same result as (a) and (b) ?
Basically in this moment i don't understand if the problem is on
clamav(see point 1), or google(maybe he is not updating all the hashes
?) .... or me ? :)))
Best regards,
Iulian Stan