Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Varnish>
  3. Bugs
#1802: Segfault after VCL change
varnish-bugs at varnish-cache
Oct 17, 2015, 8:27 AM
Post #1 of 8 (1734 views)
Permalink
#1802: Segfault after VCL change
----------------------+-------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Keywords:
----------------------+-------------------
Varnish 4.1.0 got into a non-working state this afternoon.
After a VCL change the child segfaulted, but the master process was not
able to restart successfully. Site down.

{{{
root@blade:~# ps -efww | grep varnish
varnish 914 1 0 Oct15 ? 00:00:02 /usr/sbin/varnishd -a :80
-a :8000,PROXY -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S
/etc/varnish/secret -s malloc,256m
root 2952 32506 0 17:24 pts/2 00:00:00 grep varnish
}}}

Debian stretch running 4.2.0 kernel. No other crashes/panics have been
logged.

{{{
Oct 17 17:04:09 blade varnishd[914]: CLI telnet ::1 50786 ::1 6082 Rd auth
xx
Oct 17 17:04:09 blade varnishd[914]: CLI telnet ::1 50786 ::1 6082 Wr 200
-----------------------------#012Varnish Cache CLI 1.0#012----------
-------------------#012Linux,4.2.0-1-amd64,x86_64,-junix,-smalloc,-smalloc,-hcritbit#012varnish-4.1.0
revision 3041728#012#012Type 'help' for
command list.#012Type 'quit' to close CLI session.
Oct 17 17:04:09 blade varnishd[914]: CLI telnet ::1 50786 ::1 6082 Rd ping
Oct 17 17:04:09 blade varnishd[914]: CLI telnet ::1 50786 ::1 6082 Wr 200
PONG 1445094249 1.0
Oct 17 17:04:09 blade varnishd[914]: CLI telnet ::1 50786 ::1 6082 Rd
vcl.load 9824a4ab-fb7f-4f31-9655-0df1fe76459d /etc/varnish/default.vcl
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50786 ::1 6082 Wr 200
VCL compiled.
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50788 ::1 6082 Rd auth
xx
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50788 ::1 6082 Wr 200
-----------------------------#012Varnish Cache CLI 1.0#012----------
-------------------#012Linux,4.2.0-1-amd64,x86_64,-junix,-smalloc,-smalloc,-hcritbit#012varnish-4.1.0
revision 3041728#012#012Type 'help' for
command list.#012Type 'quit' to close CLI session.
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50788 ::1 6082 Rd ping
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50788 ::1 6082 Wr 200
PONG 1445094250 1.0
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50788 ::1 6082 Rd
vcl.use 9824a4ab-fb7f-4f31-9655-0df1fe76459d
Oct 17 17:04:10 blade varnishd[914]: CLI telnet ::1 50788 ::1 6082 Wr 200
VCL '9824a4ab-fb7f-4f31-9655-0df1fe76459d' now active
Oct 17 17:17:31 blade kernel: [170387.618538] varnishd[917]: segfault at
4799ce6b ip 00007f07e2df5cb9 sp 00007ffca79c2130 error 4 in
libgcc_s.so.1[7f07e2de7000+16000]
Oct 17 17:17:32 blade varnishd[914]: Child (917) died signal=11
Oct 17 17:17:32 blade varnishd[914]: child (1814) Started
Oct 17 17:17:33 blade varnishd[914]: Child (1814) ended
Oct 17 17:17:33 blade varnishd[914]: Child (1814) said Child starts
Oct 17 17:17:33 blade varnishd[914]: Child (1814) said Child dies
Oct 17 17:17:34 blade varnishd[914]: CLI telnet ::1 50862 ::1 6082 Rd auth
xx
Oct 17 17:17:34 blade varnishd[914]: CLI telnet ::1 50862 ::1 6082 Wr 200
-----------------------------#012Varnish Cache CLI
1.0#012-----------------------------#012Linux,4.2.0-1-amd64,x86_64,-junix,-smalloc,-smalloc,-hcritbit#012varnish-4.1.0
revision 3041728#012#012Type 'help' for command list.#012Type 'quit' to
close CLI session.#012Type 'start' to launch worker process.
Oct 17 17:17:34 blade varnishd[914]: CLI telnet ::1 50862 ::1 6082 Rd ping
Oct 17 17:17:34 blade varnishd[914]: CLI telnet ::1 50862 ::1 6082 Wr 200
PONG 1445095054 1.0
Oct 17 17:17:34 blade varnishd[914]: CLI telnet ::1 50862 ::1 6082 Rd ban
req.url ~ /
Oct 17 17:17:34 blade varnishd[914]: CLI telnet ::1 50862 ::1 6082 Wr 101
Unknown request in manager process (child not running).#012Type 'help' for
more info.
Oct 17 17:17:45 blade varnishd[914]: CLI telnet ::1 50864 ::1 6082 Rd auth
xx
Oct 17 17:17:45 blade varnishd[914]: CLI telnet ::1 50864 ::1 6082 Wr 200
-----------------------------#012Varnish Cache CLI
1.0#012-----------------------------#012Linux,4.2.0-1-amd64,x86_64,-junix,-smalloc,-smalloc,-hcritbit#012varnish-4.1.0
revision 3041728#012#012Type 'help' for command list.#012Type 'quit' to
close CLI session.#012Type 'start' to launch worker process.
Oct 17 17:17:45 blade varnishd[914]: CLI telnet ::1 50864 ::1 6082 Rd ping
Oct 17 17:17:45 blade varnishd[914]: CLI telnet ::1 50864 ::1 6082 Wr 200
PONG 1445095065 1.0
}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Oct 17, 2015, 8:30 AM
Post #2 of 8 (1726 views)
Permalink
#1802: Segfault after VCL change
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by lkarsten):

{{{
root@blade:~# gdb --batch --eval-command "bt full" -p 914
/usr/sbin/varnishd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f07f0e48510 in __poll_nocancel () at ../sysdeps/unix/syscall-
template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
#0 0x00007f07f0e48510 in __poll_nocancel () at ../sysdeps/unix/syscall-
template.S:81
No locals.
#1 0x00007f07f2537456 in poll (__timeout=<optimized out>,
__nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64
-linux-gnu/bits/poll2.h:46
No locals.
#2 vev_schedule_one (evb=0x7f07f0015040) at vev.c:480
t = <optimized out>
e = 0x7f07f00dd1d0
e2 = <optimized out>
e3 = <optimized out>
i = <optimized out>
j = <optimized out>
tmo = <optimized out>
pfd = <optimized out>
__func__ = "vev_schedule_one"
#3 0x00007f07f2537898 in vev_schedule (evb=0x7f07f0015040) at vev.c:367
i = <optimized out>
__func__ = "vev_schedule"
#4 0x000000000045326b in MGT_Run () at mgt/mgt_child.c:733
sac = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction =
0x1}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 268435456,
sa_restorer = 0x0}
e = <optimized out>
i = <optimized out>
__func__ = "MGT_Run"
#5 0x000000000040f8a0 in main (argc=<optimized out>, argv=<optimized
out>) at mgt/mgt_main.c:773
o = <optimized out>
C_flag = 0
F_flag = 0
b_arg = 0x0
f_arg = 0x7ffca79c3f41 "/etc/varnish/default.vcl"
i_arg = <optimized out>
h_arg = 0x472e97 "critbit"
M_arg = 0x0
n_arg = <optimized out>
P_arg = 0x0
S_arg = 0x7ffca79c3f5d "/etc/varnish/secret"
s_arg = 0x4766f7 "malloc,100m"
W_arg = 0x0
s_arg_given = 1
T_arg = 0x7ffca79c3f2f "localhost:6082"
p = <optimized out>
vcl = <optimized out>
cli = {{magic = 1077466480, sb = 0x7f07f000e070, result = CLIS_OK,
cmd = 0x0, auth = 0, challenge = '\000' <repeats 33 times>, ident = 0x0,
vlu = 0x0, cls = 0x0, limit = 0x7ffca79c326c}}
dirname = 0x7f07f0010040 "/var/lib/varnish/blade/"
av = <optimized out>
clilim = 32768
jailed = <optimized out>
__func__ = "main"
root@blade:~#
}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Dec 7, 2015, 6:16 AM
Post #3 of 8 (1679 views)
Permalink
#1802: Segfault after VCL change
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by Tom Anheyer):

similiar error:


{{{
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41273
127.0.0.1 6082 Rd auth
3f15e9186270a640b5f0958c35b1910e85cc7ab04e038021d983a67e6b0dfba0
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41273
127.0.0.1 6082 Wr 200 -----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,3.16.7-29-desktop,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-4.1.0 revision 3041728

Type 'help' for command list.
Type 'quit' to close CLI session.
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41273
127.0.0.1 6082 Rd ping
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41273
127.0.0.1 6082 Wr 200 PONG 1449219177 1.0
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41273
127.0.0.1 6082 Rd vcl.load Test-2015-12-04-08:52:57 /etc/varnish/vcl.conf
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41273
127.0.0.1 6082 Wr 200 VCL compiled.
…
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41281
127.0.0.1 6082 Rd vcl.use Test-2015-12-04-08:52:57
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41281
127.0.0.1 6082 Wr 200 VCL 'Test-2015-12-04-08:52:57' now active
…
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41285
127.0.0.1 6082 Rd vcl.list
Dec 04 08:52:57 varnish varnishd[13456]: CLI telnet 127.0.0.1 41285
127.0.0.1 6082 Wr 200 available auto/warm 0 boot
active auto/warm 0
Test-2015-12-04-08:52:57
…
Dec 04 09:02:59 varnish varnishd[13456]: Child (13466) not responding to
CLI, killing it.
Dec 04 09:02:59 varnish varnishd[13456]: Child (13466) not responding to
CLI, killing it.
Dec 04 09:02:59 varnish varnishd[13456]: Child (13466) died signal=11
Dec 04 09:02:59 varnish varnishd[13456]: Child (13466) Panic message:
Assert error in
child_sigsegv_handler(), mgt/mgt_child.c line 297:
Condition(Segmentation fault by
instruction at 0x5590) not true.
thread = (cache-main)
version = varnish-4.1.0 revision
3041728
ident =
Linux,3.16.7-29-desktop,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
Dec 04 09:02:59 varnish varnishd[13456]: Child (13466) died signal=11
Dec 04 09:02:59 varnish varnishd[13456]: Child (13466) Panic message:
Dec 04 09:02:59 varnish varnishd[13456]: Assert error in
child_sigsegv_handler(), mgt/mgt_child.c line 297:
Dec 04 09:02:59 varnish varnishd[13456]: Condition(Segmentation fault by
instruction at 0x5590) not true.
Dec 04 09:02:59 varnish varnishd[13456]: thread = (cache-main)
Dec 04 09:02:59 varnish varnishd[13456]: version = varnish-4.1.0 revision
3041728
Dec 04 09:02:59 varnish varnishd[13456]: ident =
Linux,3.16.7-29-desktop,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
Dec 04 09:02:59 varnish varnishd[13456]: Child cleanup complete
Dec 04 09:02:59 varnish varnishd[13456]: Child cleanup complete
Dec 04 09:02:59 varnish varnishd[13456]: child (13778) Started
Dec 04 09:02:59 varnish varnishd[13456]: child (13778) Started
Dec 04 09:03:00 varnish varnishd[13456]: Pushing vcls failed:
Dec 04 09:03:00 varnish varnishd[13456]: Could not load compiled VCL.
Dec 04 09:03:00 varnish varnishd[13456]: dlopen(vcl_boot/vgc.so) =
vcl_boot/vgc.so: cannot open shared object file: No such file or directory
Dec 04 09:03:00 varnish varnishd[13456]: Stopping Child
Dec 04 09:03:00 varnish varnishd[13456]: Pushing vcls failed:
Could not load compiled VCL.
dlopen(vcl_boot/vgc.so) = vcl_boot/vgc.so: cannot open shared object file:
No such file or directory
Dec 04 09:03:00 varnish varnishd[13456]: Stopping Child
Dec 04 09:03:01 varnish varnishd[13456]: Child (13778) ended
Dec 04 09:03:01 varnish varnishd[13456]: Child (13778) ended
Dec 04 09:03:01 varnish varnishd[13456]: Child (13778) said Child starts
Dec 04 09:03:01 varnish varnishd[13456]: Child (13778) said Child dies
Dec 04 09:03:01 varnish varnishd[13456]: Child (13778) said Child starts
Dec 04 09:03:01 varnish varnishd[13456]: Child (13778) said Child dies
Dec 04 09:03:01 varnish varnishd[13456]: Child cleanup complete
Dec 04 09:03:01 varnish varnishd[13456]: Child cleanup complete
…

}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Dec 8, 2015, 6:00 AM
Post #4 of 8 (1679 views)
Permalink
#1802: Segfault after VCL change
----------------------+--------------------
Reporter: lkarsten | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by Tom Anheyer):

Please ignore last comment. We have used a command line like:
{{{
varnishd -C -f "/etc/varnish/vcl.conf"
}}}
to precheck a new vcl before loading. This deletes the shared object of
the current running 'boot' config.

best regards
tom

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:3>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Jan 11, 2016, 4:05 AM
Post #5 of 8 (1640 views)
Permalink
#1802: Segfault after VCL change
----------------------+--------------------
Reporter: lkarsten | Owner: phk
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------
Changes (by phk):

* owner: => phk


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:4>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Jan 15, 2016, 5:29 AM
Post #6 of 8 (1628 views)
Permalink
#1802: Segfault after VCL change
----------------------+--------------------
Reporter: lkarsten | Owner: phk
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution:
Keywords: |
----------------------+--------------------

Comment (by fgsch):

Side effect of commit 25417c3.

We no longer generate a temporary file so -C ends up clobbering the loaded
so file.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:5>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Jan 18, 2016, 6:07 AM
Post #7 of 8 (1620 views)
Permalink
#1802: Segfault after VCL change
----------------------+---------------------
Reporter: lkarsten | Owner: phk
Type: defect | Status: closed
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution: fixed
Keywords: |
----------------------+---------------------
Changes (by Poul-Henning Kamp <phk@…>):

* status: new => closed
* resolution: => fixed


Comment:

In [00736b717316cac500c9508d6d46ac5fb6e0eca4]:
{{{
#!CommitTicketReference repository=""
revision="00736b717316cac500c9508d6d46ac5fb6e0eca4"
Use a unique (and now reserved) VCL name of -C testing.

Fixes: #1802
}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:6>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1802: Segfault after VCL change [ In reply to ]
varnish-bugs at varnish-cache
Jan 18, 2016, 8:38 AM
Post #8 of 8 (1620 views)
Permalink
#1802: Segfault after VCL change
----------------------+---------------------
Reporter: lkarsten | Owner: phk
Type: defect | Status: closed
Priority: normal | Milestone:
Component: varnishd | Version: 4.1.0
Severity: normal | Resolution: fixed
Keywords: |
----------------------+---------------------

Comment (by Lasse Karstensen <lkarsten@…>):

In [8de208636c4425cbc3257e1ce509794835dfdd73]:
{{{
#!CommitTicketReference repository=""
revision="8de208636c4425cbc3257e1ce509794835dfdd73"
Use a unique (and now reserved) VCL name of -C testing.

Fixes: #1802
}}}

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1802#comment:7>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal