Mailing List Archive

RewriteMap and patterns
Hi,

I'm trying to use RewriteMap on a few thousand older articles to map
them to their modern equivalent. The pattern matches, but then the
redirect doesn't occur. The examples and apache docs say I should be
using [PT] to pass-through, but it results in a 404:

[Thu Mar 07 09:56:47.696040 2024] [rewrite:trace5] [pid 95091:tid 95121]
mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - -
[linuxsecurity.com/sid#5590c7db70c8]
[rid#7f02f4016480/initial] map lookup OK: map=lsv2ids[txt] key=161567 ->
val=/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago

[Thu Mar 07 09:56:47.696125 2024] [rewrite:trace2] [pid 95091:tid 95121]
mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - -
[linuxsecurity.com/sid#5590c7db70c8]
[rid#7f02f4016480/initial] rewrite '/content/view/161567' ->
'/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago'

I have the following in the main VirtualHost section of my apache config
for this domain:

  RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
  RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [PT]

My map file simply contains this line:

161567
/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago

This is a valid URL appearing as a 404:

68.195.111.33 - - [07/Mar/2024:10:13:59 -0500] "GET /content/view/161567
HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
X:"SAMEORIGIN" 1/1161742 1802/11477/2983 H:HTTP/1.1
U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago

If I replace [PT] with [L,R=301] it successfully loads the destination
link, but I'm concerned I may be creating an additional redirect. What's
the proper way to do this in my case?
Re: RewriteMap and patterns [ In reply to ]
What you have looks reasonable, PT should be URL-to-URL.
You might try logging %f in your logformat.

What kind of handler is meant to handle these requests and how is it configured?

On Thu, Mar 7, 2024 at 10:24?AM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> I'm trying to use RewriteMap on a few thousand older articles to map them to their modern equivalent. The pattern matches, but then the redirect doesn't occur. The examples and apache docs say I should be using [PT] to pass-through, but it results in a 404:
>
> [Thu Mar 07 09:56:47.696040 2024] [rewrite:trace5] [pid 95091:tid 95121] mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - - [linuxsecurity.com/sid#5590c7db70c8]
> [rid#7f02f4016480/initial] map lookup OK: map=lsv2ids[txt] key=161567 -> val=/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
>
> [Thu Mar 07 09:56:47.696125 2024] [rewrite:trace2] [pid 95091:tid 95121] mod_rewrite.c(493): [client 68.195.111.33:0] 68.195.111.33 - - [linuxsecurity.com/sid#5590c7db70c8]
> [rid#7f02f4016480/initial] rewrite '/content/view/161567' -> '/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago'
>
> I have the following in the main VirtualHost section of my apache config for this domain:
>
> RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
> RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [PT]
>
> My map file simply contains this line:
>
> 161567 /news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
>
> This is a valid URL appearing as a 404:
>
> 68.195.111.33 - - [07/Mar/2024:10:13:59 -0500] "GET /content/view/161567 HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" X:"SAMEORIGIN" 1/1161742 1802/11477/2983 H:HTTP/1.1 U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
>
> If I replace [PT] with [L,R=301] it successfully loads the destination link, but I'm concerned I may be creating an additional redirect. What's the proper way to do this in my case?
>
>


--
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: RewriteMap and patterns [ In reply to ]
Hi,

> What you have looks reasonable, PT should be URL-to-URL.
> You might try logging %f in your logformat.

I've added %f and it just shows some of what looks like components from
the 404 page, not the redirected page.

> What kind of handler is meant to handle these requests and how is it configured?

Do you mean php-fpm?

SetHandler "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"

This is the first log entry:

f:proxy:fcgi://localhost/var/www/linuxsecurity.com/html/index.php
68.195.111.33 - - [08/Mar/2024:15:29:19 -0500] "GET /content/view/161567
HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
X:"SAMEORIGIN" 0/932130 1690/11576/2983 H:HTTP/1.1
U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago
ls443 s:404 V:linuxsecurity.com v:linuxsecurity.com

Thanks,
Dave
Re: RewriteMap and patterns [ In reply to ]
I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
rewrites put the substitution) to setup the proxy sub-modules when
using SetHandler to configure proxying.

On Fri, Mar 8, 2024 at 3:46?PM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> What you have looks reasonable, PT should be URL-to-URL.
> You might try logging %f in your logformat.
>
> I've added %f and it just shows some of what looks like components from the 404 page, not the redirected page.
>
> What kind of handler is meant to handle these requests and how is it configured?
>
> Do you mean php-fpm?
>
> SetHandler "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
>
> This is the first log entry:
>
> f:proxy:fcgi://localhost/var/www/linuxsecurity.com/html/index.php 68.195.111.33 - - [08/Mar/2024:15:29:19 -0500] "GET /content/view/161567 HTTP/1.1" 404 2983 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" X:"SAMEORIGIN" 0/932130 1690/11576/2983 H:HTTP/1.1 U:/news/hackscracks/historic-hacker-attack-on-ebay-happened-3-months-ago ls443 s:404 V:linuxsecurity.com v:linuxsecurity.com
>
> Thanks,
> Dave
>
>


--
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: RewriteMap and patterns [ In reply to ]
Hi,

> I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
> rewrites put the substitution) to setup the proxy sub-modules when
> using SetHandler to configure proxying.

I'm not sure I understand what that means - do you have a recommendation
for how I should configure this instead?

dave
Re: RewriteMap and patterns [ In reply to ]
Try without [PT].

On Sat, Mar 9, 2024 at 11:17?AM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:
>
> Hi,
>
> I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
> rewrites put the substitution) to setup the proxy sub-modules when
> using SetHandler to configure proxying.
>
> I'm not sure I understand what that means - do you have a recommendation for how I should configure this instead?
>
> dave
>
>
>


--
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: RewriteMap and patterns [ In reply to ]
Might have to prefix with %{DOCUMENT_ROOT}

On Sat, Mar 9, 2024 at 11:48?AM Eric Covener <covener@gmail.com> wrote:
>
> Try without [PT].
>
> On Sat, Mar 9, 2024 at 11:17?AM Dave Wreski
> <dwreski@guardiandigital.com.invalid> wrote:
> >
> > Hi,
> >
> > I think the issue is that mod_proxy uses r->filename (i.e. what non-PT
> > rewrites put the substitution) to setup the proxy sub-modules when
> > using SetHandler to configure proxying.
> >
> > I'm not sure I understand what that means - do you have a recommendation for how I should configure this instead?
> >
> > dave
> >
> >
> >
>
>
> --
> Eric Covener
> covener@gmail.com



--
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: RewriteMap and patterns [ In reply to ]
Hi Eric,

> Might have to prefix with %{DOCUMENT_ROOT}

I've tried variations of that, including the following, but no luck.
They all fail with 400 or "invalid request"

RewriteRule "/content/view/(.*)" "${lsv2ids:$1}"
RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}" [PT]
RewriteRule "/content/view/(.*)"
"https://linuxsecurity.com${lsv2ids:$1}" [PT]
RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}"

dave
Re: RewriteMap and patterns [ In reply to ]
On 09/03/2024 22:59, Dave Wreski wrote:
> Hi Eric,
>
>> Might have to prefix with %{DOCUMENT_ROOT}
>
> I've tried variations of that, including the following, but no luck.
> They all fail with 400 or "invalid request"
>
> RewriteRule "/content/view/(.*)" "${lsv2ids:$1}"
> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}" [PT]
> RewriteRule "/content/view/(.*)"
> "https://linuxsecurity.com${lsv2ids:$1}" [PT]
> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}"
>
> dave
>
>

If I understood correctly you want the resulting path to be loaded by
your php-fpm backend.

Do you have a fallbackresource, such as:
"FallbackResource /index.php" or similar?

If not, then you may want to add it.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: RewriteMap and patterns [ In reply to ]
On 3/11/24 4:42 PM, apmail-dferradal@apache.org wrote:
> On 09/03/2024 22:59, Dave Wreski wrote:
>> Hi Eric,
>>
>>> Might have to prefix with %{DOCUMENT_ROOT}
>>
>> I've tried variations of that, including the following, but no luck.
>> They all fail with 400 or "invalid request"
>>
>> RewriteRule "/content/view/(.*)" "${lsv2ids:$1}"
>> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}" [PT]
>> RewriteRule "/content/view/(.*)"
>> "https://linuxsecurity.com${lsv2ids:$1}" [PT]
>> RewriteRule "/content/view/(.*)" "%{DOCUMENT_ROOT}${lsv2ids:$1}"
>>
> If I understood correctly you want the resulting path to be loaded by
> your php-fpm backend.
>
> Do you have a fallbackresource, such as:
> "FallbackResource /index.php" or similar?
>
> If not, then you may want to add it.

Yes, that's correct, I'm using php-fpm. I don't currently have a
fallbackresource. I'll add it, if necessary, but I'd also like to
identify the actual source reason why it's happening in the first place.
How do I troubleshoot this further?

Thanks,
Dave
Re: RewriteMap and patterns [ In reply to ]
On 12/03/2024 20:45, Dave Wreski wrote:

>>>
>> If I understood correctly you want the resulting path to be loaded by
>> your php-fpm backend.
>>
>> Do you have a fallbackresource, such as:
>> "FallbackResource /index.php" or similar?
>>
>> If not, then you may want to add it.
>
> Yes, that's correct, I'm using php-fpm. I don't currently have a
> fallbackresource. I'll add it, if necessary, but I'd also like to
> identify the actual source reason why it's happening in the first place.
> How do I troubleshoot this further?
>
> Thanks,
> Dave
>
>
>
>

What I would do is RewriteRule with just [L] tag for internal
redirection or [R,L] if you don't mind the redirection to be external.

Since I haven't seen the whole configuration my guess is you don't have
the proper handler defined for the resulting url and that's why you get
a 404, and in those cases a is when fallbackresource is ideal, to hand
those requests to where they must end.

Generally when I use passthrough [PT] is because the end url will be
dealt with the appropiate handler, but you don't seem to have a rule for
the destination to be handled in php, so how come you expect anything
else other than a 404?

That is briefly what I would do:

First make the proper redirections:

RewriteEngine on
RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]

Define the fallback for requests not ending in PHP but for which I hand
the php handler to deal them.
FallbackResource /index.php

Define the handler for php file requests
<FilesMatch \.php>
SetHandler "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
</Filesmatch>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: RewriteMap and patterns [ In reply to ]
Hi,

> What I would do is RewriteRule with just [L] tag for internal
> redirection or [R,L] if you don't mind the redirection to be external.
>
> Since I haven't seen the whole configuration my guess is you don't
> have the proper handler defined for the resulting url and that's why
> you get a 404, and in those cases a is when fallbackresource is ideal,
> to hand those requests to where they must end.
>
> Generally when I use passthrough [PT] is because the end url will be
> dealt with the appropiate handler, but you don't seem to have a rule
> for the destination to be handled in php, so how come you expect
> anything else other than a 404?
>
> That is briefly what I would do:
>
> First make the proper redirections:
>
>   RewriteEngine on
>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]
>
> Define the fallback for requests not ending in PHP but for which I
> hand the php handler to deal them.
>   FallbackResource /index.php
>
> Define the handler for php file requests
> <FilesMatch \.php>
>   SetHandler
> "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
> </Filesmatch>

I've got all this working. However, we currently have a PHP script
that's used to catch requests that are not in our static map. It is
spawned from Joomla. Normally, when a URL like the following is
requested and no match is found in our static map:

https://linuxsecurity.com/content/view/127853

it is then passed onto the PHP script from within Joomla that sets the
Location to something like:

https://linuxsecurity.com/redirection/index.php?type=view&ids=127853&uri=/content/view/127853/

The redirection.php script's function is to perform an SQL query that
tries to identify the current name from that article ID and redirects
the user to something like the following:

https://linuxsecurity.com/news/network-security/the-changing-security-threat-of-email-attacks

I'd like to be able to use the static map for the IDs I do know, and
send any others to the redirect script directly from within my apache
config directly. Is this possible?

Something like this:

  RewriteEngine on
  RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
  RewriteRule "/content/view/([0-9]{6})/?" "${lsv2ids:$1}" [L]

  RewriteRule "/content/view/([0-9]{6})/?"
"https://linuxsecurity.com/redirection/index.php?type=view&ids=$1&uri=$1"

I don't see how this would work, since both rules contain the same
pattern. Ideas on how to go about this would be appreciated. Perhaps a
RewriteMap prg rule?

  RewriteMap mapper "prg:/etc/httpd/conf.d/mapper.php"
  RewriteRule "/content/view/([0-9]{6})/?" "${mapper:$1}"

Thanks,
Dave
Re: RewriteMap and patterns [ In reply to ]
Hi,

>> First make the proper redirections:
>>
>>   RewriteEngine on
>>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]
>>
>> Define the fallback for requests not ending in PHP but for which I
>> hand the php handler to deal them.
>>   FallbackResource /index.php
>>
>> Define the handler for php file requests
>> <FilesMatch \.php>
>>   SetHandler
>> "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
>> </Filesmatch>

Can I ask about this again? I've got this all set up with a static map,
but request for /content/view/NNNNN where NNNNN is some key that is not
mapped to a value results in a 400 error.

Without this rewritemap/rewriterule, the request falls through to Joomla
where a redirection.php script is called that maps the key to the
corresponding value in a database.

With the rewritemap/rewriterule, it appears it doesn't fall through to
the redirection.php script when there is no match.

Maybe I'm missing the role of the FallbackResource here? However, it
doesn't appear it takes an argument - only an environment variable, so
not sure how that would apply.

Ideas greatly appreciated.
Re: RewriteMap and patterns [ In reply to ]
Hi,

>>> First make the proper redirections:
>>>
>>>   RewriteEngine on
>>>   RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
>>>   RewriteRule "/content/view/(.*)" "${lsv2ids:$1}" [R,L]
>>>
>>> Define the fallback for requests not ending in PHP but for which I
>>> hand the php handler to deal them.
>>>   FallbackResource /index.php
>>>
>>> Define the handler for php file requests
>>> <FilesMatch \.php>
>>>   SetHandler
>>> "proxy:unix:/run/php-fpm/linuxsecurity.sock|fcgi://localhost"
>>> </Filesmatch>
>
> Can I ask about this again? I've got this all set up with a static
> map, but request for /content/view/NNNNN where NNNNN is some key that
> is not mapped to a value results in a 400 error.
>
> Without this rewritemap/rewriterule, the request falls through to
> Joomla where a redirection.php script is called that maps the key to
> the corresponding value in a database.
>
> With the rewritemap/rewriterule, it appears it doesn't fall through to
> the redirection.php script when there is no match.
>
> Maybe I'm missing the role of the FallbackResource here? However, it
> doesn't appear it takes an argument - only an environment variable, so
> not sure how that would apply.
>
> Ideas greatly appreciated.
>
If I could add to this, another idea I had was to run the
redirection.php script directly from apache rather than in Joomla as a
rewriterule. Something like this:

RewriteRule ^/content/view/([0-9]+)/?
/redirection/index.php?type=view&ids=$1&uri=/content/view/$1 [L,R=301]

However, it's the same rewrite pattern as the above rewriterule, so not
sure how that would work.

I should also mention the reason why I need both the static map and the
redirection.php script is that I can't predict every conceivable pattern
that could exist in a map. I was also thinking of putting the top 5000
key/value in the map to be really quick, then let the redirection.php
script process the remaining legacy 70,000 or so...




--

Profile Photo



Dave Wreski

Chief Executive Officer



Guardian Digital Logo <https://www.guardiandigital.com>

*We Make Email Safe For Business*



Guardian Digital Facebook <https://www.facebook.com/gdlinux> Guardian
Digital Twitter <https://twitter.com/gdlinux> Guardian Digital Linkedin
<https://www.linkedin.com/company/guardiandigital>

Phone Icon <tel:+16408009446>



(640)-800-9446 <tel:+16408009446>



Email Security

Web Icon <https://www.guardiandigital.com>



www.guardiandigital.com <https://www.guardiandigital.com>



Sender Fraud Protection

Email Icon <mailto:dwreski@guardiandigital.com>



dwreski@guardiandigital.com



Training & Awareness

Location Icon <https://maps.app.goo.gl/aLZyUyCJznWoW1Jh8>



103 Godwin Ave, Suite 314, Midland Park, NJ 07432
<https://maps.app.goo.gl/aLZyUyCJznWoW1Jh8>
Re: RewriteMap and patterns [ In reply to ]
On Sun, Mar 24, 2024 at 2:31?AM Dave Wreski
<dwreski@guardiandigital.com.invalid> wrote:
>
> I'd like to be able to use the static map for the IDs I do know, and send any others to the redirect script directly from within my apache config directly. Is this possible?
>
> Something like this:
>
> RewriteEngine on
> RewriteMap lsv2ids "txt:/etc/httpd/conf.d/linuxsecurity-lsv2ids.map"
> RewriteRule "/content/view/([0-9]{6})/?" "${lsv2ids:$1}" [L]
>
> RewriteRule "/content/view/([0-9]{6})/?" "https://linuxsecurity.com/redirection/index.php?type=view&ids=$1&uri=$1"

Possibly using the "DefaultValue" (i.e. not found) as described in [1]:

RewriteRule "/content/view/([0-9]{6})/?"
"${lsv2ids:$1|https://linuxsecurity.com/redirection/index.php?type=view&ids=$1&uri=$1}"
[L]

[1] https://httpd.apache.org/docs/2.4/rewrite/rewritemap.html


Regards;
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org