Hi,
Ouch !!
You may get some benefit from examining what settings you have at your disposal in the following areas:
- Whichever database backend you are using, tuning its disk write policies or working memory allocations
- Tuning the filesystem layer in your OS
- Tuning firmware setting in your disk controller and/or disks
HTH
Dariush
From: ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Munroe Sollog
Sent: 17 July 2020 13:02
To: ntop@unipi.it
Subject: Re: [Ntop] Performance pains
ATTENTION: External email - This message has been sent from outside LV=.
I meant to include this as well:
[image.png]
On Fri, Jul 17, 2020 at 7:59 AM Munroe Sollog <mus3@lehigh.edu<mailto:mus3@lehigh.edu>> wrote:
While they may not me NVMe, I currently have 6 ssds in a raid 10 configuration.
On Fri, Jul 17, 2020 at 3:06 AM Marsh-Mossadeghi, Dariush <Dariush.Marsh-Mossadeghi@lv.com<mailto:Dariush.Marsh-Mossadeghi@lv.com>> wrote:
Hi,
I’d guess your disks can’t keep up.
As a rule of thumb, if you’re using any timeseries database and you’re not using solid state storage in 2020 you’re gonna have a bad time ;-)
I’ve had ntopng monitoring a busy 10G pipe with upwards of 8k active hosts on the ‘local’ network and running comfortably at a load average of no more than 3-4 on a 16 core Xeon, using up less than 100G of memory. But it does need some serious disk IO.
Use iostat to check what %iowait looks like, it’s the percentage of the time the CPU’s are waiting around for disk IO to complete. If that’s anything more than 1-2% your bottleneck is most likely disk IO.
HTH
Dariush
From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> [mailto:ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it>] On Behalf Of Munroe Sollog
Sent: 16 July 2020 19:21
To: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>
Subject: [Ntop] Performance pains
ATTENTION: External email - This message has been sent from outside LV=.
looking in the ntopng log I am getting a lot of:
[ts_5min_dump_utils.lua:461] ERROR: [Inside_VPN]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts?
ERROR: [Outside]Cannot complete local hosts timeseries dump. Disk slow or too many local hosts?
I'm assuming the host can't keep up. Our local network is defined as a /16 and it looks like we received about 50-60k flows per minute. For reference my cpu:
CPU(s): 56
On-line CPU(s) list: 0-55
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
# free -h
total used free shared buff/cache available
Mem: 251Gi 186Gi 45Gi 209Mi 20Gi 161Gi
Swap: 29Gi 347Mi 29Gi
--
Munroe Sollog
Senior Network Engineer
munroe@lehigh.edu<mailto:munroe@lehigh.edu>
This email (including any attachment) may contain confidential and/ or legally privileged information. If you are not the intended recipient, please notify us at DPO@lv.com<mailto:DPO@lv.com> and destroy it and any copies. Unauthorised access, use, disclosure, storage or copying of this email is not permitted and, unless you are the intended recipient, you are not entitled to rely on it in any way. Any opinions expressed in this email are those of the individual sending it and not necessarily those of LV=.
This email is believed to be free of any virus or other defect. However, communication by email cannot be guaranteed to be free from defect, error free or secure. If you choose to communicate with us by email you must realise that there can be no guarantee of privacy and you should carry out your own security checks before opening any email or attachment.
LV= accepts no liability for any loss or damage which may be caused by any lack of privacy, software viruses or other defect.
LV= reserves the right to monitor and inspect any email (including any attachment) sent to and/or from LV= for reasons of security and for monitoring internal compliance with our office policies. LV= may use email monitoring or blocking software at its discretion. You are responsible for ensuring that any email you send is appropriate and within the bounds of the law.
LV= and Liverpool Victoria are trademarks of Liverpool Victoria Financial Services Limited and LV= and Liverpool Victoria are trading styles of the Liverpool Victoria group of companies. Liverpool Victoria Financial Services Limited , registered in England with registration number 12383237 is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority, register number 110035.The registered office address for all LV= companies is County Gates, Bournemouth, BH1 2NF. Information about the LV= group of companies can be found via this link www.lv.com/legal/lvcompanies<
http://www.lv.com/legal/lvcompanies/>
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop --
Munroe Sollog
Senior Network Engineer
munroe@lehigh.edu<mailto:munroe@lehigh.edu>
--
Munroe Sollog
Senior Network Engineer
munroe@lehigh.edu<mailto:munroe@lehigh.edu>
