Mailing List Archive

Spamssassin seems to append .com TLD to uri link domains found
Hi list

we currently see the following "issue" where SA does append .com TLD to
uri domains found in body.

Add a uri like "www.ch" to a txt body part and SA will add "ch.com" to
domains to be checked

Nov 6 15:23:52.527 [16955] dbg: uri: canonicalizing parsed uri:
http://www.ch
Nov 6 15:23:52.527 [16955] dbg: uri: cleaned uri: http://www.ch.com
Nov 6 15:23:52.527 [16955] dbg: uri: added host: www.ch.com domain: ch.com
[...]
Nov 6 15:23:52.602 [16955] dbg: async: launching
A/ch.com.multi.uribl.com for DNSBL:ch.com:multi.uribl.com

Any other hostname than "www" does not trigger that behavior. So ftp.ch
get correctly queried as ftp.ch and not ch.com

We use SA 3.4.4

SpamAssassin Server version 3.4.4
running on Perl 5.16.3
with SSL support (IO::Socket::SSL 1.94)
with zlib support (Compress::Zlib 2.061)

Is that a bug or intended?

Cheers

tobi
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Fri, 6 Nov 2020 15:40:31 +0100
"Tobi wrote:

> Hi list
>
> we currently see the following "issue" where SA does append .com TLD
> to uri domains found in body.
>
> Add a uri like "www.ch" to a txt body part and SA will add "ch.com" to
> domains to be checked
>
> Nov 6 15:23:52.527 [16955] dbg: uri: canonicalizing parsed uri:
> http://www.ch
> Nov 6 15:23:52.527 [16955] dbg: uri: cleaned uri: http://www.ch.com
> Nov 6 15:23:52.527 [16955] dbg: uri: added host: www.ch.com domain:
> ch.com [...]
> Nov 6 15:23:52.602 [16955] dbg: async: launching
> A/ch.com.multi.uribl.com for DNSBL:ch.com:multi.uribl.com
>
> Any other hostname than "www" does not trigger that behavior. So
> ftp.ch get correctly queried as ftp.ch and not ch.com

I think it's from:

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6596

It looks to be about Firefox adding .com in some cases where the
apparent domain is not resolving. www.ch is a weird special case as a
domain name, so I'm not surprized it isn't handled separately.

However, I can't get an up-to-date Firefox to add .com, so the feature
may already be obsolete.
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
> www.ch is a weird special case as a
> domain name

Did you check whois for www.ch? It's a registered domain and it
resolves, so the owner has a whitelisting in SA or better his www.ch
links which may host a very sophisticated spearphish page will never be
queried by SA against blacklists.

Sorry but that imho is a bug that should (better must) be fixed :-)

Cheers

tobi

On 11/6/20 5:10 PM, RW wrote:
> On Fri, 6 Nov 2020 15:40:31 +0100
> "Tobi wrote:
>
>> Hi list
>>
>> we currently see the following "issue" where SA does append .com TLD
>> to uri domains found in body.
>>
>> Add a uri like "www.ch" to a txt body part and SA will add "ch.com" to
>> domains to be checked
>>
>> Nov 6 15:23:52.527 [16955] dbg: uri: canonicalizing parsed uri:
>> http://www.ch
>> Nov 6 15:23:52.527 [16955] dbg: uri: cleaned uri: http://www.ch.com
>> Nov 6 15:23:52.527 [16955] dbg: uri: added host: www.ch.com domain:
>> ch.com [...]
>> Nov 6 15:23:52.602 [16955] dbg: async: launching
>> A/ch.com.multi.uribl.com for DNSBL:ch.com:multi.uribl.com
>>
>> Any other hostname than "www" does not trigger that behavior. So
>> ftp.ch get correctly queried as ftp.ch and not ch.com
>
> I think it's from:
>
> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6596
>
> It looks to be about Firefox adding .com in some cases where the
> apparent domain is not resolving. www.ch is a weird special case as a
> domain name, so I'm not surprized it isn't handled separately.
>
> However, I can't get an up-to-date Firefox to add .com, so the feature
> may already be obsolete.
>
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Fri, 6 Nov 2020 17:24:58 +0100
Tobi wrote:

> > www.ch is a weird special case as a
> > domain name
>
> Did you check whois for www.ch? It's a registered domain and it
> resolves, so the owner has a whitelisting in SA or better his www.ch
> links which may host a very sophisticated spearphish page will never
> be queried by SA against blacklists.


The .com version is added as an additional URL, www.ch is still
queried.
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
ah understand, should have better checked what SA really adds to domain
list. So both versions are checked. Just bad luck if the expanded
version of the uri domain (ex ch.com) has a blacklisting at uribl or
spamhaus ;-)
But that's another story

Have a good weekend

tobi

On 11/6/20 5:42 PM, RW wrote:
> On Fri, 6 Nov 2020 17:24:58 +0100
> Tobi wrote:
>
>>> www.ch is a weird special case as a
>>> domain name
>>
>> Did you check whois for www.ch? It's a registered domain and it
>> resolves, so the owner has a whitelisting in SA or better his www.ch
>> links which may host a very sophisticated spearphish page will never
>> be queried by SA against blacklists.
>
>
> The .com version is added as an additional URL, www.ch is still
> queried.
>
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
Tobi skrev den 2020-11-06 17:51:
> ah understand, should have better checked what SA really adds to domain
> list. So both versions are checked. Just bad luck if the expanded
> version of the uri domain (ex ch.com) has a blacklisting at uribl or
> spamhaus ;-)
> But that's another story
>
> Have a good weekend

i followed this thread, it was mentioned it was firefox that try to help
usefull domain name ?

but i lost how this went over to a bug in spamassassin ?
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Sat, 07 Nov 2020 09:32:39 +0100
Benny Pedersen wrote:

> Tobi skrev den 2020-11-06 17:51:
> > ah understand, should have better checked what SA really adds to
> > domain list. So both versions are checked. Just bad luck if the
> > expanded version of the uri domain (ex ch.com) has a blacklisting
> > at uribl or spamhaus ;-)
> > But that's another story
> >
> > Have a good weekend
>
> i followed this thread, it was mentioned it was firefox that try to
> help usefull domain name ?
>
> but i lost how this went over to a bug in spamassassin ?

If a spammer has the domain example.com they could put http://example
in the text. An mua could see that as a link and pass it to firefox
which will open http://example.com.
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Fri, 6 Nov 2020 16:10:18 +0000
RW wrote:


> However, I can't get an up-to-date Firefox to add .com, so the feature
> may already be obsolete.

It take that back, it does.
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Sat, 7 Nov 2020, Benny Pedersen wrote:

> Tobi skrev den 2020-11-06 17:51:
>> ah understand, should have better checked what SA really adds to domain
>> list. So both versions are checked. Just bad luck if the expanded
>> version of the uri domain (ex ch.com) has a blacklisting at uribl or
>> spamhaus ;-)
>> But that's another story
>>
>> Have a good weekend
>
> i followed this thread, it was mentioned it was firefox that try to help
> usefull domain name ?
>
> but i lost how this went over to a bug in spamassassin ?


The bug was to implement the same (mis)behavior in SA URI parsing.


--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Trusting in anti-gun laws to keep you from being shot is like
refusing to wear your seatbelt because you trust traffic laws to
keep you from being in a car accident. -- Erin Palette
-----------------------------------------------------------------------
2 days until The 82nd anniversary of Kristallnacht - disarmament enables genocide
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Sat, 7 Nov 2020, RW wrote:

> On Fri, 6 Nov 2020 16:10:18 +0000
> RW wrote:
>
>
>> However, I can't get an up-to-date Firefox to add .com, so the feature
>> may already be obsolete.
>
> It take that back, it does.

What does it do for the example at hand, http://www.ch ?

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Sat, 7 Nov 2020 10:05:21 -0800 (PST)
John Hardin wrote:

> On Sat, 7 Nov 2020, RW wrote:
>
> > On Fri, 6 Nov 2020 16:10:18 +0000
> > RW wrote:
> >
> >
> >> However, I can't get an up-to-date Firefox to add .com, so the
> >> feature may already be obsolete.
> >
> > I take that back, it does.
>
> What does it do for the example at hand, http://www.ch ?

Firefox only adds .com if the domain doesn't resolve.

www.ch resolves and then redirects to https://meteo.ch/

If SA is to allow for what Firefox does then I think the behaviour is
reasonable. A DNS lookup would be overkill, and there's no particular
reason to exclude labels that happen to be TLDs.
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Sat, 7 Nov 2020, RW wrote:

> On Sat, 7 Nov 2020 10:05:21 -0800 (PST)
> John Hardin wrote:
>
>> On Sat, 7 Nov 2020, RW wrote:
>>
>>> On Fri, 6 Nov 2020 16:10:18 +0000
>>> RW wrote:
>>>
>>>
>>>> However, I can't get an up-to-date Firefox to add .com, so the
>>>> feature may already be obsolete.
>>>
>>> I take that back, it does.
>>
>> What does it do for the example at hand, http://www.ch ?
>
> Firefox only adds .com if the domain doesn't resolve.
>
> www.ch resolves and then redirects to https://meteo.ch/
>
> If SA is to allow for what Firefox does then I think the behaviour is
> reasonable. A DNS lookup would be overkill,

Agreed.

> and there's no particular reason to exclude labels that happen to be
> TLDs.

Do you mean *valid* TLDs? Because I think that suppressing that behavior
for valid TLDs would be an appropriate modification to avoid potential
URIBL FPs (which, granted, is probably fairly unlikely) and to avoid the
overhead of extra lookups.


--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
A government is a lot like a gun: It's always loaded,
and it's stupid and dangerous to point it at anything
you don't intend to hurt. -- GOF at TSM
-----------------------------------------------------------------------
2 days until The 82nd anniversary of Kristallnacht - disarmament enables genocide
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
John,

> Because I think that suppressing that behavior for valid TLDs would be
an appropriate modification to avoid potential URIBL FPs

fully agree. SA should not append .com if the domain has a valid tld and
a domain label. We know of at least one FP related to "www.ch" when (the
expanded version) "ch.com" was checked on uribl lists.

Cheers

tobi

On 11/7/20 8:04 PM, John Hardin wrote:
> On Sat, 7 Nov 2020, RW wrote:
>
>> On Sat, 7 Nov 2020 10:05:21 -0800 (PST)
>> John Hardin wrote:
>>
>>> On Sat, 7 Nov 2020, RW wrote:
>>>
>>>> On Fri, 6 Nov 2020 16:10:18 +0000
>>>> RW wrote:
>>>>
>>>>
>>>>> However, I can't get an up-to-date Firefox to add .com, so the
>>>>> feature may already be obsolete.
>>>>
>>>> I take that back, it does.
>>>
>>> What does it do for the example at hand, http://www.ch ?
>>
>> Firefox only adds .com if the domain doesn't resolve.
>>
>> www.ch resolves and then redirects to https://meteo.ch/
>>
>> If SA is to allow for what Firefox does then I think the behaviour is
>> reasonable. A DNS lookup would be overkill,
>
> Agreed.
>
>> and there's no particular reason to exclude labels that happen to be
>> TLDs.
>
> Do you mean *valid* TLDs? Because I think that suppressing that behavior
> for valid TLDs would be an appropriate modification to avoid potential
> URIBL FPs (which, granted, is probably fairly unlikely) and to avoid the
> overhead of extra lookups.
>
>
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On Mon, 9 Nov 2020 08:26:59 +0100
Tobi wrote:

> John,
>
> > Because I think that suppressing that behavior for valid TLDs would
> > be
> an appropriate modification to avoid potential URIBL FPs
>
> fully agree. SA should not append .com if the domain has a valid tld
> and a domain label.

Only domains of the form "<tld>" and "www.<tld>" are affected.

In my experience corporate TLDs like "amazon" aren't getting used as
single label domains on websites, and if they were the .com version
would probably be the same site anyway.

The second form is very rare, and I doubt links like that are
widely used in email as they look suspicious.

>We know of at least one FP related to "www.ch"
> when (the expanded version) "ch.com" was checked on uribl lists.

When you say FP do you mean that ch.com was listed in a domain
blocklist?

On the whole domain names of the form <tld>.com are going to be
expensive, so they aren't likely to be owned directly by mainstream
spammers.

If the .com domain points to a hacked server, the firefox trick could
be used to extend the useful life of the hack until both domains are
listed. I think there's a case to be made either way.
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
On 11/9/20 3:18 PM, RW wrote:
> When you say FP do you mean that ch.com was listed in a domain
> blocklist?

yes, the "ch.com" was listed in one of the domain blocklists. And as the
customer used his own domain "www.ch" as footer it got caught by the
lookup for "ch.com" which seems to be a chinese news page or something
like that.

Its hard to explain to a customer that a URI domain he did not use in
the message lead to a hit on blocklists lookup ;-)

Cheers

tobi
Re: Spamssassin seems to append .com TLD to uri link domains found [ In reply to ]
Tobi, can you open a bug about this and add a spample of the issue? ch is
in the TLD list 20_aux_tlds.cf.
--
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171


On Mon, Nov 9, 2020 at 9:56 AM Tobi <jahlives@gmx.ch> wrote:

>
>
> On 11/9/20 3:18 PM, RW wrote:
> > When you say FP do you mean that ch.com was listed in a domain
> > blocklist?
>
> yes, the "ch.com" was listed in one of the domain blocklists. And as the
> customer used his own domain "www.ch" as footer it got caught by the
> lookup for "ch.com" which seems to be a chinese news page or something
> like that.
>
> Its hard to explain to a customer that a URI domain he did not use in
> the message lead to a hit on blocklists lookup ;-)
>
> Cheers
>
> tobi
>