Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Varnish>
  3. Misc
Varnish crashes periodically with high disk read and write
hrhosseini at hotmail
Sep 28, 2020, 1:39 AM
Post #1 of 2 (391 views)
Permalink
Hi
I'm using varnish-6.0.5,
1.
3nodes of varnish crashes every day and I should restart it else the whole of my system doesnt work anymore
I don't know the reason but my disks at that time shows that It has 50 MB/s (400Mb/s) Read and 48 MB/s write but my outbound network shows 120Mb/s
I put the link of Netdata picture at that time,
http://uupload.ir/files/c60r_varnish.png
How can I solve this problem?

2.
My next question is that I have another varnish that I set malloc to 68Gb (the server has 86 Gb ram) but it will pass 68g till 86 and server crash, Why does it exceed from 68g through I have adjusted the accurate and permitted number?

But really I'm confused and annoyed with first problem because it destroys my cluster

```

Sep 27 13:46:19 varnish-16 varnishd[121910]: Error: Manager got SIGTERM
Sep 27 13:46:19 varnish-16 varnishd[121910]: Manager got SIGTERM
Sep 27 13:46:19 varnish-16 varnishd[121910]: Debug: Stopping Child
Sep 27 13:46:19 varnish-16 varnishd[121910]: Stopping Child
Sep 27 13:46:30 varnish-16 varnishd[121910]: Error: Child (121932) died signal=15
Sep 27 13:46:30 varnish-16 varnishd[121910]: Child (121932) died signal=15
Sep 27 13:46:30 varnish-16 varnishd[121910]: Debug: Child cleanup complete
Sep 27 13:46:30 varnish-16 varnishd[121910]: Child cleanup complete
Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager stopping child
Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager dies
Sep 27 13:46:30 varnish-16 varnishd[121910]: manager stopping child
Sep 27 13:46:30 varnish-16 varnishd[121910]: manager dies
Sep 27 13:44:46 varnish-16 kernel: lowmem_reserve[]: 0 2934 15972 15972 15972
Sep 27 13:44:46 varnish-16 kernel: Node 0 DMA32 free:55712kB min:2968kB low:3708kB high:4452kB active_anon:0kB inactive_anon:2674740kB active_file:104kB inactive_file:0kB unevictable:15528kB isolated(ano
Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 13037 13037 13037
Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal free:21044kB min:13196kB low:16492kB high:19792kB active_anon:492kB inactive_anon:12208096kB active_file:548kB inactive_file:1300kB unevictable:70304kB is
Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (E) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 0*512kB 1*1024kB (U) 2*2048kB (ME) 2*4096kB (M) = 14332kB
Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA32: 7633*4kB (UME) 2869*8kB (UMEH) 39*16kB (UM) 14*32kB (UM) 12*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55580kB
Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal: 5132*4kB (UMEH) 69*8kB (UMH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 21096kB
Sep 27 13:44:47 varnish-16 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 27 13:44:47 varnish-16 kernel: 3742574 total pagecache pages
Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete 469898786, find 188596514/233894822
Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB
Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete 469898786, find 188596514/233894822
Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB
Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
Sep 27 13:44:47 varnish-16 kernel: 0 pages HighMem/MovableOnly
Sep 27 13:44:47 varnish-16 kernel: 90468 pages reserved
Sep 27 13:44:47 varnish-16 kernel: 0 pages cma reserved
Sep 27 13:44:47 varnish-16 kernel: 0 pages hwpoisoned
Sep 27 13:44:47 varnish-16 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 27 13:44:47 varnish-16 kernel: [ 509] 0 509 10968 358 22 3 607 0 systemd-journal
Sep 27 13:44:47 varnish-16 kernel: [ 546] 0 546 23692 130 16 3 51 0 lvmetad
Sep 27 13:44:47 varnish-16 kernel: [ 557] 0 557 11194 307 22 3 327 -1000 systemd-udevd
Sep 27 13:44:47 varnish-16 kernel: [ 639] 0 639 28742 364 49 3 320 0 vmtoolsd
Sep 27 13:44:47 varnish-16 kernel: [ 703] 100 703 25080 235 19 3 85 0 systemd-timesyn
Sep 27 13:44:47 varnish-16 kernel: [ 969] 0 969 6510 350 18 3 59 0 atd
Sep 27 13:44:47 varnish-16 kernel: [ 983] 107 983 10721 208 28 3 119 -900 dbus-daemon
Sep 27 13:44:47 varnish-16 kernel: [ 1010] 0 1010 6931 401 19 3 94 0 cron
Sep 27 13:44:47 varnish-16 kernel: [ 1012] 104 1012 64097 156 28 3 324 0 rsyslogd
Sep 27 13:44:47 varnish-16 kernel: [ 1013] 0 1013 21359 372 33 3 351 0 VGAuthService
Sep 27 13:44:47 varnish-16 kernel: [ 1019] 0 1019 68649 76 37 3 211 0 accounts-daemon
Sep 27 13:44:47 varnish-16 kernel: [ 1025] 0 1025 1098 288 8 3 35 0 acpid
Sep 27 13:44:47 varnish-16 kernel: [ 1028] 0 1028 7164 366 18 3 86 0 systemd-logind
Sep 27 13:44:47 varnish-16 kernel: [ 1038] 0 1038 40225 209 15 3 114 0 lxcfs
Sep 27 13:44:47 varnish-16 kernel: [ 1100] 0 1100 3342 255 11 3 43 0 mdadm
Sep 27 13:44:47 varnish-16 kernel: [ 1114] 0 1114 69271 223 40 3 180 0 polkitd
Sep 27 13:44:47 varnish-16 kernel: [ 1224] 0 1224 1304 359 8 3 31 0 iscsid
Sep 27 13:44:47 varnish-16 kernel: [ 1225] 0 1225 1429 879 8 3 0 -17 iscsid
Sep 27 13:44:47 varnish-16 kernel: [ 1251] 0 1251 16377 348 36 3 190 -1000 sshd
Sep 27 13:44:47 varnish-16 kernel: [ 1303] 0 1303 3663 312 13 3 39 0 agetty
Sep 27 13:44:47 varnish-16 kernel: [ 1314] 0 1314 4903 314 15 3 125 0 irqbalance
Sep 27 13:44:47 varnish-16 kernel: [ 1358] 999 1358 13541 0 29 2 11886 1000 netdata
Sep 27 13:44:47 varnish-16 kernel: [121910] 0 121910 9067 422 19 3 113 0 varnishd
Sep 27 13:44:47 varnish-16 kernel: [121932] 112 121932 86615131 7577 159392 334 76899838 0 cache-main
Sep 27 13:44:47 varnish-16 kernel: [25236] 999 25236 1253 0 6 2 857 1000 apps.plugin
Sep 27 13:44:47 varnish-16 kernel: [35166] 999 35166 406 0 5 2 91 1000 bash
Sep 27 13:44:47 varnish-16 kernel: [44898] 999 44898 24372 337 35 3 3600 1000 python
Sep 27 13:44:47 varnish-16 kernel: Out of memory: Kill process 1358 (netdata) score 1000 or sacrifice child
Sep 27 13:44:47 varnish-16 kernel: Killed process 25236 (apps.plugin) total-vm:5012kB, anon-rss:0kB, file-rss:0kB


```

```

varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2019 Varnish Software AS

```

Best regards.
Re: Varnish crashes periodically with high disk read and write [ In reply to ]
guillaume at varnish-software
Sep 28, 2020, 11:45 AM
Post #2 of 2 (390 views)
Permalink
Hi,

This post should help understand a couple of points:
https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage
On top of this, you are probably using Transient:
https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage


--
Guillaume Quintard


On Mon, Sep 28, 2020 at 1:40 AM Hamidreza Hosseini <hrhosseini@hotmail.com>
wrote:

>
> Hi
> I'm using varnish-6.0.5,
> 1.
> 3nodes of varnish crashes every day and I should restart it else the whole
> of my system doesnt work anymore
> I don't know the reason but my disks at that time shows that It has 50
> MB/s (400Mb/s) Read and 48 MB/s write but my outbound network shows 120Mb/s
> I put the link of Netdata picture at that time,
> http://uupload.ir/files/c60r_varnish.png
> How can I solve this problem?
>
> 2.
> My next question is that I have another varnish that I set malloc to 68Gb
> (the server has 86 Gb ram) but it will pass 68g till 86 and server crash,
> Why does it exceed from 68g through I have adjusted the accurate and
> permitted number?
>
> But really I'm confused and annoyed with first problem because it destroys
> my cluster
>
> ```
>
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Error: Manager got SIGTERM
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Manager got SIGTERM
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Debug: Stopping Child
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Stopping Child
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Error: Child (121932) died
> signal=15
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Child (121932) died signal=15
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Debug: Child cleanup complete
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Child cleanup complete
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager stopping child
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager dies
> Sep 27 13:46:30 varnish-16 varnishd[121910]: manager stopping child
> Sep 27 13:46:30 varnish-16 varnishd[121910]: manager dies
> Sep 27 13:44:46 varnish-16 kernel: lowmem_reserve[]: 0 2934 15972 15972
> 15972
> Sep 27 13:44:46 varnish-16 kernel: Node 0 DMA32 free:55712kB min:2968kB
> low:3708kB high:4452kB active_anon:0kB inactive_anon:2674740kB
> active_file:104kB inactive_file:0kB unevictable:15528kB isolated(ano
> Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 13037 13037 13037
> Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal free:21044kB min:13196kB
> low:16492kB high:19792kB active_anon:492kB inactive_anon:12208096kB
> active_file:548kB inactive_file:1300kB unevictable:70304kB is
> Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 0 0 0
> Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB
> (U) 1*32kB (E) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 0*512kB 1*1024kB (U)
> 2*2048kB (ME) 2*4096kB (M) = 14332kB
> Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA32: 7633*4kB (UME) 2869*8kB
> (UMEH) 39*16kB (UM) 14*32kB (UM) 12*64kB (U) 2*128kB (U) 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 55580kB
> Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal: 5132*4kB (UMEH) 69*8kB
> (UMH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB
> 0*4096kB = 21096kB
> Sep 27 13:44:47 varnish-16 kernel: Node 0 hugepages_total=0
> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> Sep 27 13:44:47 varnish-16 kernel: 3742574 total pagecache pages
> Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
> Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete
> 469898786, find 188596514/233894822
> Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB
> Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
> Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
> Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
> Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete
> 469898786, find 188596514/233894822
> Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB
> Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
> Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
> Sep 27 13:44:47 varnish-16 kernel: 0 pages HighMem/MovableOnly
> Sep 27 13:44:47 varnish-16 kernel: 90468 pages reserved
> Sep 27 13:44:47 varnish-16 kernel: 0 pages cma reserved
> Sep 27 13:44:47 varnish-16 kernel: 0 pages hwpoisoned
> Sep 27 13:44:47 varnish-16 kernel: [ pid ] uid tgid total_vm rss
> nr_ptes nr_pmds swapents oom_score_adj name
> Sep 27 13:44:47 varnish-16 kernel: [ 509] 0 509 10968 358
> 22 3 607 0 systemd-journal
> Sep 27 13:44:47 varnish-16 kernel: [ 546] 0 546 23692 130
> 16 3 51 0 lvmetad
> Sep 27 13:44:47 varnish-16 kernel: [ 557] 0 557 11194 307
> 22 3 327 -1000 systemd-udevd
> Sep 27 13:44:47 varnish-16 kernel: [ 639] 0 639 28742 364
> 49 3 320 0 vmtoolsd
> Sep 27 13:44:47 varnish-16 kernel: [ 703] 100 703 25080 235
> 19 3 85 0 systemd-timesyn
> Sep 27 13:44:47 varnish-16 kernel: [ 969] 0 969 6510 350
> 18 3 59 0 atd
> Sep 27 13:44:47 varnish-16 kernel: [ 983] 107 983 10721 208
> 28 3 119 -900 dbus-daemon
> Sep 27 13:44:47 varnish-16 kernel: [ 1010] 0 1010 6931 401
> 19 3 94 0 cron
> Sep 27 13:44:47 varnish-16 kernel: [ 1012] 104 1012 64097 156
> 28 3 324 0 rsyslogd
> Sep 27 13:44:47 varnish-16 kernel: [ 1013] 0 1013 21359 372
> 33 3 351 0 VGAuthService
> Sep 27 13:44:47 varnish-16 kernel: [ 1019] 0 1019 68649 76
> 37 3 211 0 accounts-daemon
> Sep 27 13:44:47 varnish-16 kernel: [ 1025] 0 1025 1098 288
> 8 3 35 0 acpid
> Sep 27 13:44:47 varnish-16 kernel: [ 1028] 0 1028 7164 366
> 18 3 86 0 systemd-logind
> Sep 27 13:44:47 varnish-16 kernel: [ 1038] 0 1038 40225 209
> 15 3 114 0 lxcfs
> Sep 27 13:44:47 varnish-16 kernel: [ 1100] 0 1100 3342 255
> 11 3 43 0 mdadm
> Sep 27 13:44:47 varnish-16 kernel: [ 1114] 0 1114 69271 223
> 40 3 180 0 polkitd
> Sep 27 13:44:47 varnish-16 kernel: [ 1224] 0 1224 1304 359
> 8 3 31 0 iscsid
> Sep 27 13:44:47 varnish-16 kernel: [ 1225] 0 1225 1429 879
> 8 3 0 -17 iscsid
> Sep 27 13:44:47 varnish-16 kernel: [ 1251] 0 1251 16377 348
> 36 3 190 -1000 sshd
> Sep 27 13:44:47 varnish-16 kernel: [ 1303] 0 1303 3663 312
> 13 3 39 0 agetty
> Sep 27 13:44:47 varnish-16 kernel: [ 1314] 0 1314 4903 314
> 15 3 125 0 irqbalance
> Sep 27 13:44:47 varnish-16 kernel: [ 1358] 999 1358 13541 0
> 29 2 11886 1000 netdata
> Sep 27 13:44:47 varnish-16 kernel: [121910] 0 121910 9067 422
> 19 3 113 0 varnishd
> Sep 27 13:44:47 varnish-16 kernel: [121932] 112 121932 86615131 7577
> 159392 334 76899838 0 cache-main
> Sep 27 13:44:47 varnish-16 kernel: [25236] 999 25236 1253 0
> 6 2 857 1000 apps.plugin
> Sep 27 13:44:47 varnish-16 kernel: [35166] 999 35166 406 0
> 5 2 91 1000 bash
> Sep 27 13:44:47 varnish-16 kernel: [44898] 999 44898 24372 337
> 35 3 3600 1000 python
> Sep 27 13:44:47 varnish-16 kernel: Out of memory: Kill process 1358
> (netdata) score 1000 or sacrifice child
> Sep 27 13:44:47 varnish-16 kernel: Killed process 25236 (apps.plugin)
> total-vm:5012kB, anon-rss:0kB, file-rss:0kB
>
>
> ```
>
> ```
>
> varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40)
> Copyright (c) 2006 Verdens Gang AS
> Copyright (c) 2006-2019 Varnish Software AS
>
> ```
>
> Best regards.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal