Mailing List Archive

On Fri, Jun 18, 2021 at 12:24 PM Richard Chivers <r.chivers@zengenti.com> wrote:
>
> Hi,
>
> First post, hope this is the right place.
>
> I am doing some work at the moment moving our config from version 6 to 6.6, also moving from ubuntu bionic to focal.
>
> In the systemd configuration when we start varnish we pass the args (also many more not detailed):
>
> -T localhost:6082 \
> -S /etc/varnish/secret \
>
> We have generated an appropriate secret file etc.
>
> In bionic when we run a varnishadm, we don't need to pass the -T or -S args, it just reads the secret file ( I am assuming) and connects.
>
> In focal this is not the case, I need to pass the args. e.g. varnishadm -T localhost:6082 -S /etc/varnish/secret
>
> Because of this calling /usr/sbin/varnishreload fails because it calls varnishadm -n '' -- vcl.list and gets the response "No -T in shared memory"
>
> So my question is where does this default from, is there an ENV variable to set, or am I just missing something?

Did your system's hostname change between the moment when varnish was
started and when you attempted a reload?

Can you share the rest of your service file? (maybe redact sensitive
parts if any)


> Another strange issue is that varnishlog is not returning anything, it simply hangs and doen't show anything or an error for that matter.
>
> I Installed by adding the repo: deb https://packagecloud.io/varnishcache/varnish66/ubuntu/ focal main
>
> Any ideas or help appreciated.
>
> I have gone back through change logs, but can't spot anything.
>
> Thanks
>
> Richard
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

On Mon, Jun 21, 2021 at 6:45 AM Richard Chivers <r.chivers@zengenti.com> wrote:
>
> Hi, thanks for coming back. No, the hostname didn't change. Here is the rest of the file:
>
> [Unit]
<snip>
> -S /etc/varnish/secret \

Can you try removing the -S option from varnishd?

Since you only listen to the CLI on localhost, there's likely no
remote access, so leaving the secret out will make varnishd generate a
random one. Basically, if you want to use varnishadm you need local
root privileges, same as your current setup.

<snip>
>> > In bionic when we run a varnishadm, we don't need to pass the -T or -S args, it just reads the secret file ( I am assuming) and connects.
>> >
>> > In focal this is not the case, I need to pass the args. e.g. varnishadm -T localhost:6082 -S /etc/varnish/secret

The main entry point is the -n option, and then options (or lack
thereof) like -T and -S can be found from the working directory.

>> > Because of this calling /usr/sbin/varnishreload fails because it calls varnishadm -n '' -- vcl.list and gets the response "No -T in shared memory"
>> >
>> > So my question is where does this default from, is there an ENV variable to set, or am I just missing something?
>>
>> Did your system's hostname change between the moment when varnish was
>> started and when you attempted a reload?
>>
>> Can you share the rest of your service file? (maybe redact sensitive
>> parts if any)

I didn't have time to give more details, but the default value for -n
is the system's hostname, that's why I asked initially about the
hostname changing.

>> > Another strange issue is that varnishlog is not returning anything, it simply hangs and doen't show anything or an error for that matter.

Are you running varnishlog with enough privileges? (most likely
belonging at least to the varnish group.)

If you omit the -d option, varnishlog will print transactions as they
complete, so if by any chance you are inspecting a test system with no
traffic that's not surprising.

A surefire way to see whether varnishlog connects to a running varnish
instance is to try:

varnishlog -d -g raw

>> > I Installed by adding the repo: deb https://packagecloud.io/varnishcache/varnish66/ubuntu/ focal main
>> >
>> > Any ideas or help appreciated.
>> >
>> > I have gone back through change logs, but can't spot anything.
>> >
>> > Thanks
>> >
>> > Richard

Please keep the mailing list CC'd.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

Hey, thanks for coming back to me, I have done some more work, but haven't
got any further at this stage.

On Mon, Jun 21, 2021 at 9:01 AM Dridi Boukelmoune <dridi@varni.sh> wrote:

> On Mon, Jun 21, 2021 at 6:45 AM Richard Chivers <r.chivers@zengenti.com>
> wrote:
> >
> > Hi, thanks for coming back. No, the hostname didn't change. Here is the
> rest of the file:
> >
> > [Unit]
> <snip>
> > -S /etc/varnish/secret \
>
> Can you try removing the -S option from varnishd?
>
> Since you only listen to the CLI on localhost, there's likely no
> remote access, so leaving the secret out will make varnishd generate a
> random one. Basically, if you want to use varnishadm you need local
> root privileges, same as your current setup.
>

I tried this and it makes no difference, I think the fundamental issue is
that calling varnishadm without args seems (regardless the args I pass to
varnishd) to end in the message "No -T in shared memory" if run from root.

If I run from another user, I do get the message "could not get hold of
varnishd, is it running?"

I guess I could update the reload script to pass the -T and -S args, but
this seems wrong, just concerned there is a general issue on focal, Is
anyone else running 6.6 on focal?

Looking at the source code in 6 and 6.6 I can't see anywhere that the -T
would default from and yet on 6 under bionic varnishadm as a root user just
works without any -T or -S flags.

https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishadm/varnishadm.c



> <snip>
> >> > In bionic when we run a varnishadm, we don't need to pass the -T or
> -S args, it just reads the secret file ( I am assuming) and connects.
> >> >
> >> > In focal this is not the case, I need to pass the args. e.g.
> varnishadm -T localhost:6082 -S /etc/varnish/secret
>
> The main entry point is the -n option, and then options (or lack
> thereof) like -T and -S can be found from the working directory.
>
> >> > Because of this calling /usr/sbin/varnishreload fails because it
> calls varnishadm -n '' -- vcl.list and gets the response "No -T in shared
> memory"
> >> >
> >> > So my question is where does this default from, is there an ENV
> variable to set, or am I just missing something?
> >>
> >> Did your system's hostname change between the moment when varnish was
> >> started and when you attempted a reload?
> >>
> >> Can you share the rest of your service file? (maybe redact sensitive
> >> parts if any)
>
> I didn't have time to give more details, but the default value for -n
> is the system's hostname, that's why I asked initially about the
> hostname changing.
>
> >> > Another strange issue is that varnishlog is not returning anything,
> it simply hangs and doen't show anything or an error for that matter.
>
> Are you running varnishlog with enough privileges? (most likely
> belonging at least to the varnish group.)
>
> If you omit the -d option, varnishlog will print transactions as they
> complete, so if by any chance you are inspecting a test system with no
> traffic that's not surprising.
>
> A surefire way to see whether varnishlog connects to a running varnish
> instance is to try:
>
> varnishlog -d -g raw
>

I am running as root. If I execute this it connects but I get no output, I
know it is connected because when I restart the varnish process I get the
message, "Log abandoned (vsm)" which you always see when a new
varbnishd process starts. I am definitely hitting the varnish server, as I
am executing curl requests to localhost:80, but there is no output from
varnishlog.

I am about to spin up some more boxes, so will check to see wheter this is
just specific to this box or not, I did initially install 6.2 on this
server and varnishlog was working as expected with that.

>
> >> > I Installed by adding the repo: deb
> https://packagecloud.io/varnishcache/varnish66/ubuntu/ focal main
> >> >
> >> > Any ideas or help appreciated.
> >> >
> >> > I have gone back through change logs, but can't spot anything.
> >> >
> >> > Thanks
> >> >
> >> > Richard
>
> Please keep the mailing list CC'd.
>
> Dridi
>

>> Can you try removing the -S option from varnishd?
>>
>> Since you only listen to the CLI on localhost, there's likely no
>> remote access, so leaving the secret out will make varnishd generate a
>> random one. Basically, if you want to use varnishadm you need local
>> root privileges, same as your current setup.
>
>
> I tried this and it makes no difference, I think the fundamental issue is that calling varnishadm without args seems (regardless the args I pass to varnishd) to end in the message "No -T in shared memory" if run from root.

This is very strange, I would probably need to somehow peek at the
working directory to figure out what's happening. The only way to see
no -T in shared memory is to explicitly ask for it with `-T none`,
which clearly you didn't do.

> If I run from another user, I do get the message "could not get hold of varnishd, is it running?"

This on the other hand sounds like basic unix permissions coming
short, as one would expect.

> I guess I could update the reload script to pass the -T and -S args, but this seems wrong, just concerned there is a general issue on focal, Is anyone else running 6.6 on focal?

The varnishreload script is meant to focus on the system service
integration use case: you have a local Varnish instance that can also
be operated locally. So we shouldn't need to add options to specify -T
or -S, we should find them on the running instance.

You could use -T none if you have an alternative mode of operations.
For example varnishd -d puts you in debug mode and stdin/stdout is
used for the CLI. The alternative would be the -M option that lets
varnishd "reverse" connect to its operator, but then you would leave
the varnishreload use case.

Not sure about focal users, but I will try to spin up a VM and see if
I can reproduce it.

> Looking at the source code in 6 and 6.6 I can't see anywhere that the -T would default from and yet on 6 under bionic varnishadm as a root user just works without any -T or -S flags.
>
> https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishadm/varnishadm.c

Correct, the default behavior is to inspect the running Varnish
instance to find them.

>> A surefire way to see whether varnishlog connects to a running varnish
>> instance is to try:
>>
>> varnishlog -d -g raw
>
>
> I am running as root. If I execute this it connects but I get no output, I know it is connected because when I restart the varnish process I get the message, "Log abandoned (vsm)" which you always see when a new varbnishd process starts. I am definitely hitting the varnish server, as I am executing curl requests to localhost:80, but there is no output from varnishlog.

That seems to indicate that Varnish may have been started on a
different hostname and what you are "connecting" to is the remnant of
a dead instance.

What is the output of `ls /var/lib/varnish` ?

> I am about to spin up some more boxes, so will check to see wheter this is just specific to this box or not, I did initially install 6.2 on this server and varnishlog was working as expected with that.

I would recommend sticking to the 6.0 LTS series, unless you
absolutely need a feature released after 6.0 that hasn't been
back-ported to the stable branch.

https://packagecloud.io/varnishcache/varnish60lts/install

If you don't use an LTS series, I recommend always sticking to the
latest release. The 6.2 series is EOL and no longer maintained, which
may include security vulnerabilities such as VSV7.

https://varnish-cache.org/security/VSV00007.html

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc