Mailing List Archive: Reverse Proxy robots.txt

Reverse Proxy robots.txt

daviddgtntpersonal at gmail

Sep 7, 2021, 5:12 PM

Post #1 of 7 (565 views)

I have had a Raspberry Pi, and am now also running a couple VMs on my new
ProxMox server, and now the leader of the organization I am a part of
called KygekTeam wants me to make my reverse proxy also have a robots.txt.
I have already made the robots.txt and tried for about an hour now and I
still haven't made any progress on making the HTTP requests not get
proxied. The version of httpd as said by apache2 -v is:
Server version: Apache/2.4.38 (Raspbian)
Server built: 2020-08-25T20:08:29
I'm using my Raspberry Pi because I have other apps on it.

Re: Reverse Proxy robots.txt [ In reply to ]

thumbs at apache

Sep 8, 2021, 3:40 PM

Post #2 of 7 (565 views)

You can use ProxyPass /robots.txt !

(the ! is important)

On Tue, 7 Sept 2021 at 20:13, David Jentes <daviddgtntpersonal@gmail.com>
wrote:

> I have had a Raspberry Pi, and am now also running a couple VMs on my new
> ProxMox server, and now the leader of the organization I am a part of
> called KygekTeam wants me to make my reverse proxy also have a robots.txt.
> I have already made the robots.txt and tried for about an hour now and I
> still haven't made any progress on making the HTTP requests not get
> proxied. The version of httpd as said by apache2 -v is:
> Server version: Apache/2.4.38 (Raspbian)
> Server built: 2020-08-25T20:08:29
> I'm using my Raspberry Pi because I have other apps on it.
>

Re: Reverse Proxy robots.txt [ In reply to ]

daviddgtntpersonal at gmail

Sep 8, 2021, 3:43 PM

Post #3 of 7 (565 views)

I've tried it, but it's not working

On Wed, Sep 8, 2021 at 5:40 PM Frank Gingras <thumbs@apache.org> wrote:

> You can use ProxyPass /robots.txt !
>
> (the ! is important)
>
> On Tue, 7 Sept 2021 at 20:13, David Jentes <daviddgtntpersonal@gmail.com>
> wrote:
>
>> I have had a Raspberry Pi, and am now also running a couple VMs on my new
>> ProxMox server, and now the leader of the organization I am a part of
>> called KygekTeam wants me to make my reverse proxy also have a robots.txt.
>> I have already made the robots.txt and tried for about an hour now and I
>> still haven't made any progress on making the HTTP requests not get
>> proxied. The version of httpd as said by apache2 -v is:
>> Server version: Apache/2.4.38 (Raspbian)
>> Server built: 2020-08-25T20:08:29
>> I'm using my Raspberry Pi because I have other apps on it.
>>
>

Re: Reverse Proxy robots.txt [ In reply to ]

56independent at protonmail

Sep 9, 2021, 11:43 AM

Post #4 of 7 (565 views)

> I've tried it, but it's not working

Is robots.txt in the root directory of your website? having example.com/robots/robots.txt won't work. It *has* to be in example.com/robots.txt

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Reverse Proxy robots.txt [ In reply to ]

daviddgtntpersonal at gmail

Sep 9, 2021, 4:50 PM

Post #5 of 7 (565 views)

Yes, I am using it in /robots.txt. It's also what I'm testing it with. It's
not super urgent, but we'd like to finish it ASAP.

On Thu, Sep 9, 2021 at 1:44 PM 56independent
<56independent@protonmail.com.invalid> wrote:

> > I've tried it, but it's not working
>
> Is robots.txt in the root directory of your website? having
> example.com/robots/robots.txt won't work. It *has* to be in
> example.com/robots.txt
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Re: Reverse Proxy robots.txt [ In reply to ]

wefatherley at gmail

Sep 9, 2021, 4:59 PM

Post #6 of 7 (565 views)

Did you try using the alias directive (
https://httpd.apache.org/docs/current/mod/mod_alias.html#alias)?

Yes, I am using it in /robots.txt. It's also what I'm testing it with. It's
> not super urgent, but we'd like to finish it ASAP.
>
>>
>>

Re: Reverse Proxy robots.txt [ In reply to ]

daviddgtntpersonal at gmail

Sep 11, 2021, 6:23 AM

Post #7 of 7 (565 views)

Yes I did.

On Thu, Sep 9, 2021 at 7:00 PM Will Fatherley <wefatherley@gmail.com> wrote:

> Did you try using the alias directive (
> https://httpd.apache.org/docs/current/mod/mod_alias.html#alias)?
>
> Yes, I am using it in /robots.txt. It's also what I'm testing it with.
>> It's not super urgent, but we'd like to finish it ASAP.
>>
>>>
>>>