Mailing List Archive

On Sat, Apr 8, 2023 at 2:14?PM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:

> Hi,
>
> I have an apache-2.4.56 install on fedora37 and trying to block some bots
> from accessing the site, unless they're trying to access our RSS feeds. How
> can I do this?
>
> I'm blocking the bots with SetEnvIF lines in the .htacess file in the
> document root like:
>
> SetEnvIf user-agent "(?i:libwww)" stayout=1
> deny from env=stayout
> <RequireAll>
> Require all granted
> Require not env stayout
> </RequireAll>
>
> However, creating an entry that explicitly allows access to the XML files
> before or after doesn't seem to take effect:
>
> RewriteRule linuxsecurity_features\.xml$ - [L]
>
> It is still blocked by the user-agent setting above. I understood the file
> was processed from the top down, and when a match is made, it stops
> processing. Is that not the case? Shouldn't the RewriteRule above, if
> placed before the env rule, be enough to stop processing the htaccess file
> and allow access?
>

The [L] flag only stops later RewriteRule directives from being processed.

Every module still gets its configuration merged from every matching config
context, then decides what to do with its configuration when passed control
at various times.

setenvif is processed very early, so if you can stay with it for
manipulating this variable it will be much more intuitive


> I've also tried adding these RewriteRule entries to the server config
> htaccess with an Include, but it appears the .htaccess in the document root
> is always processed afterwards, even after finding match in the server
> config htaccess.
>

I'd suggest the following:

1. Ditch the "deny", requireall, and require all granted leaving just "Require
not env stayout"
2. Ditch the RewriteRule and do a second SetEnvIf for the exception
(SetEnvIf Request_URI linuxsecurity_features\.xml$ !stayout"

--

Eric Covener
covener@gmail.com

trahiet88@gmail.com
Tuk tuk kampot drivers

On Sun, Apr 9, 2023, 1:24 AM Eric Covener <covener@gmail.com> wrote:

>
>
> On Sat, Apr 8, 2023 at 2:14?PM Dave Wreski
> <dwreski@guardiandigital.com.invalid> wrote:
>
>> Hi,
>>
>> I have an apache-2.4.56 install on fedora37 and trying to block some bots
>> from accessing the site, unless they're trying to access our RSS feeds. How
>> can I do this?
>>
>> I'm blocking the bots with SetEnvIF lines in the .htacess file in the
>> document root like:
>>
>> SetEnvIf user-agent "(?i:libwww)" stayout=1
>> deny from env=stayout
>> <RequireAll>
>> Require all granted
>> Require not env stayout
>> </RequireAll>
>>
>> However, creating an entry that explicitly allows access to the XML files
>> before or after doesn't seem to take effect:
>>
>> RewriteRule linuxsecurity_features\.xml$ - [L]
>>
>> It is still blocked by the user-agent setting above. I understood the
>> file was processed from the top down, and when a match is made, it stops
>> processing. Is that not the case? Shouldn't the RewriteRule above, if
>> placed before the env rule, be enough to stop processing the htaccess file
>> and allow access?
>>
>
> The [L] flag only stops later RewriteRule directives from being processed.
>
> Every module still gets its configuration merged from every matching
> config context, then decides what to do with its configuration when passed
> control at various times.
>
> setenvif is processed very early, so if you can stay with it for
> manipulating this variable it will be much more intuitive
>
>
>> I've also tried adding these RewriteRule entries to the server config
>> htaccess with an Include, but it appears the .htaccess in the document root
>> is always processed afterwards, even after finding match in the server
>> config htaccess.
>>
>
> I'd suggest the following:
>
> 1. Ditch the "deny", requireall, and require all granted leaving just "Require
> not env stayout"
> 2. Ditch the RewriteRule and do a second SetEnvIf for the exception
> (SetEnvIf Request_URI linuxsecurity_features\.xml$ !stayout"
>
> --
>
> Eric Covener
> covener@gmail.com
>

Hi,

> I have an apache-2.4.56 install on fedora37 and trying to block
> some bots from accessing the site, unless they're trying to access
> our RSS feeds. How can I do this?
>
> I'm blocking the bots with SetEnvIF lines in the .htacess file in
> the document root like:
>
>     SetEnvIf user-agent "(?i:libwww)" stayout=1
>     deny from env=stayout
>     <RequireAll>
>        Require all granted
>        Require not env stayout
>     </RequireAll>
>
> However, creating an entry that explicitly allows access to the
> XML files before or after doesn't seem to take effect:
>
>     RewriteRule linuxsecurity_features\.xml$ - [L]
>
> It is still blocked by the user-agent setting above. I understood
> the file was processed from the top down, and when a match is
> made, it stops processing. Is that not the case? Shouldn't the
> RewriteRule above, if placed before the env rule, be enough to
> stop processing the htaccess file and allow access?
>
>
> The [L] flag only stops later RewriteRule directives from being processed.
>
> Every module still gets its configuration merged from every matching
> config context, then decides what to do with its configuration when
> passed control at various times.
>
> setenvif is processed very early, so if you can stay with it for
> manipulating this variable it will be much more intuitive
>
> I've also tried adding these RewriteRule entries to the server
> config htaccess with an Include, but it appears the .htaccess in
> the document root is always processed afterwards, even after
> finding match in the server config htaccess.
>
>
> I'd suggest the following:
>
> 1. Ditch the "deny", requireall, and require all granted leaving just
> "Require not env stayout"
> 2. Ditch the RewriteRule and do a second SetEnvIf for the exception
> (SetEnvIf  Request_URI linuxsecurity_features\.xml$ !stayout"

This doesn't fix it, assuming I'm implementing it as you've described.
Removing the RequireAll section produces a site-wide 500 error in error_log:

.htaccess: negative Require directive has no effect in <RequireAny>
directive

SetEnvIf user-agent "(?i:libwww-perl)" stayout=1
SetEnvIf Request_URI ^linuxsecurity_features\.*$ !stayout
RewriteRule linuxsecurity_features\.xml$ - [L]

198.74.49.155 - - [10/Apr/2023:10:32:33 -0400] "GET
/linuxsecurity_features.xml HTTP/1.1" 403 199 "-" "LWP::Simple/6.00
libwww-perl/6.05" X:"SAMEORIGIN" 0/9629 979/8868/199 H:HTTP/1.1

This is all designed to prevent bots from being able to easily mirror
our website. Even though I understand individuals could just change
their user agent, sites like yandex/Acunetix and other services won't.

dave

>
> .htaccess: negative Require directive has no effect in <RequireAny>
> directive
>

Ah, I guess you'll have to restore the RequireAll and its contents.

Hi,

On 4/10/23 10:51 AM, Eric Covener wrote:
>
> .htaccess: negative Require directive has no effect in
> <RequireAny> directive
>
>
> Ah, I guess you'll have to restore the RequireAll and its contents.
>
Yes, I should have mentioned that I've already done that - without it,
it was producing a 500 error for virtually all accesses.

Other ideas welcome :-)

dave

Hello.

I don't know what is troubling you...
For now, don't use the combination of Order,Allow,Deny and Require* in configuration file that will be created in 202x. :)

AllowDeny of ENV should be replaced by the following.

Allow from env=X_FOOBAR -> Require env X_FOOBAR

It can write more complex conditions.

Regards.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Hi,

> I don't know what is troubling you...
> For now, don't use the combination of Order,Allow,Deny and Require* in configuration file that will be created in 202x. :)
>
> AllowDeny of ENV should be replaced by the following.
>
> Allow from env=X_FOOBAR -> Require env X_FOOBAR
>
> It can write more complex conditions.

I've made quite a few changes over the last few days to try to get this
to work, but it's still not right.

Not every user agent is defined as either a bot or not a bot, so just
"require env !stayout" results in 403 errors on every page with
"AH01630: client denied by server configuration:"

Here's again what I have that doesn't appear to work:

SetEnvIf user-agent "(?i:TurnitinBot)" stayout=1
SetEnvIf Request_URI "^linuxsecurity_features\.*$" !stayout
<RequireAll>
    Require env !stayout
</RequireAll>

RewriteRule linuxsecurity_features\.*$ - [L]

Ideas greatly appreciated.

Thanks,
Dave

Dave Wreski wrote on 2023/04/11 10:54:
> SetEnvIf user-agent "(?i:TurnitinBot)" stayout=1
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" !stayout

I have done it in the past, too.
It was like allowing another level of conditions to be attached to Allow and Deny, depending on the condition that defines the variable and the condition that undefines the variable, right? :)
This is no longer needed, so change the name to something that makes sense.

SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2

And let it meet all the requirements.

<RequireAll>
Require all granted
Require not env SOMENAME1
Require env SOMENAME2
</RequireAll>

...According to mod_authz_core.html, it looks like we don't even need SetEnvIf.

<RequireAny>
<RequireAll>
Require all granted
Require not expr "%{HTTP_USER_AGENT} =~ /SomeBot/"
</RequireAll>
Require expr "%{REQUEST_URI} =~ /^linuxfoo/"
</RequireAny>

The outermost <RequireAny> is verbose.
I'm on the teaching side, but I've never used expr, so I don't know if the syntax is correct. :)

I did my research on it a long time ago and I don't know if it is still correct...
The result of this access control is broken by the Order, Allow, and Deny directives :)
Sure, I thought that all of those directives should be allowed by
Order Deny,Allow
or
Allow from all
, but I forgot :)
For now, Order, Allow, and Deny should be removed if they exist somewhere :)

Regards.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

The legacy authz directives from 2.2 will indeed cause issues when mixed
with Require (2.4). Do not mix them.


On Mon, Apr 10, 2023 at 11:48?PM Tatsuki Makino <tatsuki_makino@hotmail.com>
wrote:

> Dave Wreski wrote on 2023/04/11 10:54:
> > SetEnvIf user-agent "(?i:TurnitinBot)" stayout=1
> > SetEnvIf Request_URI "^linuxsecurity_features\.*$" !stayout
>
> I have done it in the past, too.
> It was like allowing another level of conditions to be attached to Allow
> and Deny, depending on the condition that defines the variable and the
> condition that undefines the variable, right? :)
> This is no longer needed, so change the name to something that makes sense.
>
> SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2
>
> And let it meet all the requirements.
>
> <RequireAll>
> Require all granted
> Require not env SOMENAME1
> Require env SOMENAME2
> </RequireAll>
>
> ...According to mod_authz_core.html, it looks like we don't even need
> SetEnvIf.
>
> <RequireAny>
> <RequireAll>
> Require all granted
> Require not expr "%{HTTP_USER_AGENT} =~ /SomeBot/"
> </RequireAll>
> Require expr "%{REQUEST_URI} =~ /^linuxfoo/"
> </RequireAny>
>
> The outermost <RequireAny> is verbose.
> I'm on the teaching side, but I've never used expr, so I don't know if the
> syntax is correct. :)
>
> I did my research on it a long time ago and I don't know if it is still
> correct...
> The result of this access control is broken by the Order, Allow, and Deny
> directives :)
> Sure, I thought that all of those directives should be allowed by
> Order Deny,Allow
> or
> Allow from all
> , but I forgot :)
> For now, Order, Allow, and Deny should be removed if they exist somewhere
> :)
>
> Regards.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Hello.

Frank Gingras wrote on 2023/04/11 13:36:
> The legacy authz directives from 2.2 will indeed cause issues when mixed
> with Require (2.4). Do not mix them.

Yes.
Basically, they MUST NOT BE MIXED, but we should know how they behave when the mod_access_compat is loaded.
I think it was like this :)

+--- mod_authz_core --+ +--- mod_access_compat --+
| | | |
(entry) | granted | | Allow |
-------------------------+----------------------------------+-----------------> 200
| | | | | |
| denied | | | Deny | |
+--------- | ---------+ +----------- | ----------+
| |
v v
403 403

Is that correct? :)
If Order directive is not written, it defaults to Order Deny,Allow.
When Order is Deny,Allow, all are allowed by default, so mod_access_compat is passed by...
...I guess that's what it means.

Regards.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Hi,

On 4/10/23 11:48 PM, Tatsuki Makino wrote:
> Dave Wreski wrote on 2023/04/11 10:54:
>> SetEnvIf user-agent "(?i:TurnitinBot)" stayout=1
>> SetEnvIf Request_URI "^linuxsecurity_features\.*$" !stayout
> I have done it in the past, too.
> It was like allowing another level of conditions to be attached to Allow and Deny, depending on the condition that defines the variable and the condition that undefines the variable, right? :)
> This is no longer needed, so change the name to something that makes sense.

Yes, the "deny" was removed long ago. That is not the problem now. I
also previously neglected to post the "<IfModule !mod_authz_core.c>"
qualifier around the "deny from env=stayout" which would only load it
for apache-2.2 systems.

> SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2
>
> And let it meet all the requirements.
>
> <RequireAll>
> Require all granted
> Require not env SOMENAME1
> Require env SOMENAME2
> </RequireAll>

This had the effect of blocking elements on every page (and the page
itself, I think) on the site with 403s and "AH01630: client denied by
server configuration:" entries in the error log.

Is it possible there's something else going on here?

SetEnvIf Request_URI "^linuxsecurity_features\.*$" rssfeeds
SetEnvIf user-agent "(?i:TurnitinBot)" stayout
<RequireAll>
   Require all granted
   Require not env stayout
   Require env rssfeeds
</RequireAll>
RewriteRule linuxsecurity_features\.*$ - [L]

Thanks,
Dave

On Tue, Apr 11, 2023 at 9:29?AM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> On 4/10/23 11:48 PM, Tatsuki Makino wrote:
>
> Dave Wreski wrote on 2023/04/11 10:54:
>
> SetEnvIf user-agent "(?i:TurnitinBot)" stayout=1
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" !stayout
>
> I have done it in the past, too.
> It was like allowing another level of conditions to be attached to Allow and Deny, depending on the condition that defines the variable and the condition that undefines the variable, right? :)
> This is no longer needed, so change the name to something that makes sense.
>
> Yes, the "deny" was removed long ago. That is not the problem now. I also previously neglected to post the "<IfModule !mod_authz_core.c>" qualifier around the "deny from env=stayout" which would only load it for apache-2.2 systems.
>
> SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2
>
> And let it meet all the requirements.
>
> <RequireAll>
> Require all granted
> Require not env SOMENAME1
> Require env SOMENAME2
> </RequireAll>
>
> This had the effect of blocking elements on every page (and the page itself, I think) on the site with 403s and "AH01630: client denied by server configuration:" entries in the error log.
>
> Is it possible there's something else going on here?
>
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" rssfeeds

Can't anchor to ^. Unlike rewrite in htaccess, this will always
compare against the actual requested URL. Not the weird remainder of
the URL.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Hi,


>> SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
>> SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2
>>
>> And let it meet all the requirements.
>>
>> <RequireAll>
>> Require all granted
>> Require not env SOMENAME1
>> Require env SOMENAME2
>> </RequireAll>
>>
>> This had the effect of blocking elements on every page (and the page itself, I think) on the site with 403s and "AH01630: client denied by server configuration:" entries in the error log.
>>
>> Is it possible there's something else going on here?
>>
>> SetEnvIf Request_URI "^linuxsecurity_features\.*$" rssfeeds
> Can't anchor to ^. Unlike rewrite in htaccess, this will always
> compare against the actual requested URL. Not the weird remainder of
> the URL.

There's something more going on than just an errant caret.

Once the "Require env SOMENAME2" is included, as above, it immediately
starts to 403 every page on the site. It's like each Require above is
being considered independently, or that it somehow supersedes the
previous Require.

With the last Require commented out, it works as expected (blocking all
bots listed in the SetEnvIf), with the exception that it also restricts
libwww access to the RSS feeds.

dave

On Tue, Apr 11, 2023 at 10:49?AM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:

> Hi,
>
> SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2
>
> And let it meet all the requirements.
>
> <RequireAll>
> Require all granted
> Require not env SOMENAME1
> Require env SOMENAME2
> </RequireAll>
>
> This had the effect of blocking elements on every page (and the page itself, I think) on the site with 403s and "AH01630: client denied by server configuration:" entries in the error log.
>
> Is it possible there's something else going on here?
>
> SetEnvIf Request_URI "^linuxsecurity_features\.*$" rssfeeds
>
> Can't anchor to ^. Unlike rewrite in htaccess, this will always
> compare against the actual requested URL. Not the weird remainder of
> the URL.
>
> There's something more going on than just an errant caret.
>
> Once the "Require env SOMENAME2" is included, as above, it immediately
> starts to 403 every page on the site. It's like each Require above is being
> considered independently, or that it somehow supersedes the previous
> Require.
>
> With the last Require commented out, it works as expected (blocking all
> bots listed in the SetEnvIf), with the exception that it also restricts
> libwww access to the RSS feeds.
>

I don't follow the full intended logic, but if SOMENAME2 is required for
access but can never be set due to the regex, I don't think there is
necessarily something more to it.


--
Eric Covener
covener@gmail.com

Hi,

>>> SetEnvIf user-agent "(?i:TurnitinBot)" SOMENAME1
>>> SetEnvIf Request_URI "^linuxsecurity_features\.*$" SOMENAME2
>>>
>>> And let it meet all the requirements.
>>>
>>> <RequireAll>
>>> Require all granted
>>> Require not env SOMENAME1
>>> Require env SOMENAME2
>>> </RequireAll>
>>>
>>> This had the effect of blocking elements on every page (and the page itself, I think) on the site with 403s and "AH01630: client denied by server configuration:" entries in the error log.
>>>
>>> Is it possible there's something else going on here?
>>>
>>> SetEnvIf Request_URI "^linuxsecurity_features\.*$" rssfeeds
>> Can't anchor to ^. Unlike rewrite in htaccess, this will always
>> compare against the actual requested URL. Not the weird remainder of
>> the URL.
>
> There's something more going on than just an errant caret.
>
> Once the "Require env SOMENAME2" is included, as above, it
> immediately starts to 403 every page on the site. It's like each
> Require above is being considered independently, or that it
> somehow supersedes the previous Require.
>
> With the last Require commented out, it works as expected
> (blocking all bots listed in the SetEnvIf), with the exception
> that it also restricts libwww access to the RSS feeds.
>
>
> I don't follow the full intended logic, but if SOMENAME2 is required
> for access but can never be set due to the regex, I don't think there
> is necessarily something more to it.

In case I wasn't clear, simply removing the caret was not enough to make
this work. The "Require env SOMENAME2" was enough to begin blocking
every page on the site with a 403, not just the RSS feeds or the bots,
but every legitimate request.

I'm going to do more research on RequireAll, but isn't that essentially
saying AND? Require all AND Require not env SOMENAME1 AND Require env
SOMENAME2? How can that ever match?

I suppose that also means it wouldn't be RequireAny because it would
always match the Require all.

dave

Dave Wreski wrote on 2023/04/12 01:39:
> In case I wasn't clear, simply removing the caret was not enough to make this work. The "Require env SOMENAME2" was enough to begin blocking every page on the site with a 403, not just the RSS feeds or the bots, but every legitimate request.
>
> I'm going to do more research on RequireAll, but isn't that essentially saying AND? Require all AND Require not env SOMENAME1 AND Require env SOMENAME2? How can that ever match?
>
> I suppose that also means it wouldn't be RequireAny because it would always match the Require all.

The purpose of this is to include related keywords, and I wrote it without understanding the logic required for actual operation.
There might be a funny reversal going on somewhere.

Is this correct? :)

<RequireAny>
<RequireAll>
Require all granted
Require not env IS_BOT
</RequireAll>
Require env IS_RSS
</RequireAny>

Regards.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Hi,

On 4/11/23 8:59 PM, Tatsuki Makino wrote:
> Dave Wreski wrote on 2023/04/12 01:39:
>> In case I wasn't clear, simply removing the caret was not enough to make this work. The "Require env SOMENAME2" was enough to begin blocking every page on the site with a 403, not just the RSS feeds or the bots, but every legitimate request.
>>
>> I'm going to do more research on RequireAll, but isn't that essentially saying AND? Require all AND Require not env SOMENAME1 AND Require env SOMENAME2? How can that ever match?
>>
>> I suppose that also means it wouldn't be RequireAny because it would always match the Require all.
> The purpose of this is to include related keywords, and I wrote it without understanding the logic required for actual operation.
> There might be a funny reversal going on somewhere.
>
> Is this correct? :)
>
> <RequireAny>
> <RequireAll>
> Require all granted
> Require not env IS_BOT
> </RequireAll>
> Require env IS_RSS
> </RequireAny>

Yes, that's correct :-)

I also simultaneously came to this outcome, but had a single errant "
character that affected the entire htaccess file, resulting in accesses
being rejected, yikes. Once I realized that, it started to work.

One other question - is there an order of processing the .htaccess in
the document root and the virtual host config? Are they both processed
together, or does one take precedence over the other?

Thanks so much for all of your help.

dave

>
> One other question - is there an order of processing the .htaccess in the
> document root and the virtual host config? Are they both processed
> together, or does one take precedence over the other?
>

If VirtualHost is the only enclosing scope, it is processed earlier so it
has lower precedence in VirtualHost. But many other configuration sections
are counter-intuitively merged in later than things in (just) .htaccess:
https://httpd.apache.org/docs/2.4/sections.html#merging