Mailing List Archive

varnish core dump
Can anyone help me on below error. looks like core dump is happening and child is getting restarted. This is happening recently without any change to the VCL and any system configuration.

During this time the requests are getting terminated abruptly.

Thanks in advance.


Varnish version - varnish-4.0.2


Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory 'ccpp-2016-06-24-23:48:19-26526' creation detected
Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to CLI, killing it.
Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to CLI, killing it.
Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6 (core dumped)
Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461: CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98: ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012 0x3c25e079d1: /lib64/libpthread.so.0() [0x3c25e079d1]#012 0x3c25ae8b6d: /lib64/libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012 sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430 {#012 fd = 64, vxid = 34652201,#012 client = 10.254.9.78 61217,#012 step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws = 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} = {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method = 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012 id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012 },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012 "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012 "HTTP/1.1",#012 "host: xxx.xxx.com",#012 "Accept-Encoding: gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP: 49.15.204.238",#012 "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1; 1201 Build/LMY47I)",#012 "Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)",#012 "X-Aka
Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child starts


Thanks,

Mahesh.
Re: varnish core dump [ In reply to ]
Hi

I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.

Please upgrade to the latest release and file a bug if you still run into this.

[1]: https://www.varnish-cache.org/trac/ticket/1539

On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho@outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461:
> CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98:
> ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so.0() [0x3c25e079d1]#012 0x3c25ae8b6d:
> /lib64/libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body
> = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430
> {#012 fd = 64, vxid = 34652201,#012 client = 10.254.9.78 61217,#012
> step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws =
> 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method =
> 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012
> id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012 "host: xxx.xxx.com",#012 "Accept-Encoding:
> gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP:
> 49.15.204.238",#012 "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012 "Via: 1.1 v1-akamaitech.net(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost)
> (AkamaiGHost)",#012 "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: varnish core dump [ In reply to ]
Hi,


Thanks for the response.

We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.


Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012 Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432f13: pan_ic+0x163#012 0x42dec5: http_EstimateWS+0x435#012 0x421874: vbf_beresp2obj+0x84#012 0x422cb5: vbf_stp_error+0x235#012 0x42366b: vbf_fetch_thread+0x1db#012 0x449825: WRK_Thread+0x4a5#012 0x449c5b: pool_thread+0x2b#012 0x3c25e079d1: libpthread.so.0() [0x3c25e079d1]#012 0x3c25ae8b6d: libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012 ws = 0x7f7f50520960 {#012 OVERFLOWED id = \"Bo\",#012 {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012 },#012 refcnt = 2,#012 retries = 0, failed = 1, state = 1,#012 flags = {do_gzip, do_stream, is_gunzip},#012 director_req = 0x1a16130 {#012 vcl_name = farm_anon,#012 type = round-robin {#012 },#012 },#012 director_resp = 0x1a08e98 {#012 vcl_name = PROD_LB_BE_ANON_1,#012 type = backend {#012 display_name = boot.PROD_LB_BE_ANON_1,#012 ipv4 = 10.254.4.254,#012 port = 8080,#012 hosthdr = 10.254.4.254,#012 health=healthy, admin_health=probe, changed=1473664247.5,#012 n_conn = 2,#012 },#012 },#012 http[bereq] = 0x7f7f50520f28 {#012 ws[Bo] = 0x7f7f50520960,#012 hdrs {#012 \"GET\",#012 \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012 \"HTTP/1.1\",#012 \"host: www.example.com<http://www.example.com> \",#012 \"Accept-Encoding: gzip\",#012 \"Akamai-Origin-Hop: 4\",#012 \"True-Client-IP: 183.83.249.187\",#012 \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012 \"Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts


Thanks in advance,

Mahesh.

________________________________
From: Dag Haavi Finstad <daghf@varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc@varnish-cache.org
Subject: Re: varnish core dump

Hi

I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.

Please upgrade to the latest release and file a bug if you still run into this.

[1]: https://www.varnish-cache.org/trac/ticket/1539

On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho@outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461:
> CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98:
> ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so.0() [0x3c25e079d1]#012 0x3c25ae8b6d:
> /lib64/libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body
> = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430
> {#012 fd = 64, vxid = 34652201,#012 client = 10.254.9.78 61217,#012
> step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws =
> 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method =
> 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012
> id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012 "host: xxx.xxx.com",#012 "Accept-Encoding:
> gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP:
> 49.15.204.238",#012 "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012 "Via: 1.1 v1-akamaitech.net(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost)
> (AkamaiGHost)",#012 "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
Re: varnish core dump [ In reply to ]
Thank you Anand.

we have set workspace_backend=262144 and the issue is resolved.


Thanks,

Mahesh.

________________________________
From: anand@rediff-inc.com <anand@rediff-inc.com> on behalf of Anand Shah <anand@rediff-inc.com>
Sent: Monday, September 12, 2016 4:09 PM
To: Mahesh Ollalwar
Cc: Dag Haavi Finstad ; varnish-misc@varnish-cache.org
Subject: Re: varnish core dump

In order to reduce contention between all the threads, Varnish uses workspaces which allow each thread to use or modify memory independently of other threads, which avoids locking. Try increasing session workspace and shm_workspace. Disable any VMOD's that you use.



Regards,
Anand Shah



From: Mahesh Ollalwar <mahesho@outlook.com>
Sent: Mon, 12 Sep 2016 15:16:52
To: Dag Haavi Finstad <daghf@varnish-software.com>, "varnish-misc@varnish-cache.org" <varnish-misc@varnish-cache.org>
Subject: Re: varnish core dump




Hi,



Thanks for the response.

We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.





Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012 Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432f13: pan_ic+0x163#012 0x42dec5: http_EstimateWS+0x435#012 0x421874: vbf_beresp2obj+0x84#012 0x422cb5: vbf_stp_error+0x235#012 0x42366b: vbf_fetch_thread+0x1db#012 0x449825: WRK_Thread+0x4a5#012 0x449c5b: pool_thread+0x2b#012 0x3c25e079d1: libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012 0x3c25ae8b6d: libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012 ws = 0x7f7f50520960 {#012 OVERFLOWED id = \"Bo\",#012 {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012 },#012 refcnt = 2,#012 retries = 0, failed = 1, state = 1,#012 flags = {do_gzip, do_stream, is_gunzip},#012 director_req = 0x1a16130 {#012 vcl_name = farm_anon,#012 type = round-robin {#012 },#012 },#012 director_resp = 0x1a08e98 {#012 vcl_name = PROD_LB_BE_ANON_1,#012 type = backend {#012 display_name = boot.PROD_LB_BE_ANON_1,#012 ipv4 = 10.254.4.254<http://10.254.4.254>,#012 port = 8080,#012 hosthdr = 10.254.4.254<http://10.254.4.254>,#012 health=healthy, admin_health=probe, changed=1473664247.5,#012 n_conn = 2,#012 },#012 },#012 http[bereq] = 0x7f7f50520f28 {#012 ws[Bo] = 0x7f7f50520960,#012 hdrs {#012 \"GET\",#012 \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012 \"HTTP/1.1\",#012 \"host: www.example.com<http://www.example.com> \",#012 \"Accept-Encoding: gzip\",#012 \"Akamai-Origin-Hop: 4\",#012 \"True-Client-IP: 183.83.249.187<http://183.83.249.187>\",#012 \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012 \"Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts





Thanks in advance,

Mahesh.


________________________________
From: Dag Haavi Finstad <daghf@varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc@varnish-cache.org
Subject: Re: varnish core dump

Hi

I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.

Please upgrade to the latest release and file a bug if you still run into this.

[1]: https://www.varnish-cache.org/trac/ticket/1539

On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho@outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461:
> CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98:
> ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012 0x3c25ae8b6d:
> /lib64/libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body
> = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430
> {#012 fd = 64, vxid = 34652201,#012 client = 10.254.9.78<http://10.254.9.78> 61217,#012
> step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws =
> 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method =
> 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012
> id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012 "host: xxx.xxx.com<http://xxx.xxx.com>",#012 "Accept-Encoding:
> gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP:
> 49.15.204.238<http://49.15.204.238>",#012 "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012 "Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 akamai.net<http://akamai.net>(ghost)
> (AkamaiGHost)",#012 "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<//prolinks.rediffmailpro.com/cgi-bin/prored.cgi?red=https%3A%2F%2Fwww%2Evarnish%2Dcache%2Eorg%2Flists%2Fmailman%2Flistinfo%2Fvarnish%2Dmisc&rediffng=0>