22 Zář
2020
22 Zář
'20
23:11
Hi,
We run around 50 knot servers around the world, with great success - thanks
for the software!
We are just a little curious about seemingly high CPU usage, and how best
to interpret the results, if someone could advise? Specifically, we are
seeing the majority of the cpu time in 'system' and not 'user', what
would
that suggest is happening perhaps?
Any tuning help gratefully received!
Our platform (sample)
24 Core CPU E5-2440 0 @ 2.40
64Gb RAM
SSD
knot-2.9.5, running on Centos 8.2
Currently receiving around 4k pps
Server is authoritative for a 278 zones, the largest zone file being around
600Mb
Output of top
top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47, 12.34,
11.83
Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
%Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7 si,
0.0 st
%Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7 si,
0.0 st
%Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3 si,
0.0 st
%Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3 si,
0.0 st
%Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3 si,
0.0 st
%Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0 si,
0.0 st
%Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0 si,
0.0 st
%Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3 si,
0.0 st
%Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7 si,
0.0 st
%Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3 si,
0.0 st
%Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0 si,
0.0 st
%Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
%Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0 si,
0.0 st
%Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3 si,
0.0 st
%Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3 si,
0.0 st
%Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3 si,
0.0 st
%Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7 si,
0.0 st
%Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0 si,
0.0 st
MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8 buff/cache
MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
23 Zář
23 Zář
8:59
Hi Jonathan,
On 2020-09-22 23:11, Jonathan Hewlett wrote:
Hi,
We run around 50 knot servers around the world, with great success -
thanks for the software!
Thanks for this feedback :-)
We are just a little curious about seemingly high CPU usage, and how
best to interpret the results, if someone could advise? Specifically,
we are seeing the majority of the cpu time in 'system' and not 'user',
what would that suggest is happening perhaps?
Could you provide me the output from `knotc status workers`?
I like htop. Are you able to send me a screenshot when running htop? :-)
[Don't forget to configure:
F2
Display options->Detailed CPU time
(System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
F10
]
Best,
Daniel
Any tuning help gratefully received!
Our platform (sample)
24 Core CPU E5-2440 0 @ 2.40
64Gb RAM
SSD
knot-2.9.5, running on Centos 8.2
Currently receiving around 4k pps
Server is authoritative for a 278 zones, the largest zone file being
around 600Mb
Output of top
top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
12.34, 11.83
Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
%Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7
si, 0.0 st
%Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
si, 0.0 st
%Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
si, 0.0 st
%Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
si, 0.0 st
%Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
si, 0.0 st
%Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
buff/cache
MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
13:18
Daniel
Thank you for your time
# knotc status workers
UDP workers: 24, TCP workers 24, background workers: 10 (running: 0,
pending: 0)
[image: image.png]
Kind regards
Jonathan
On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz> wrote:
Hi Jonathan,
On 2020-09-22 23:11, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
Hi,
We run around 50 knot servers around the world, with great success -
thanks for the software!
Thanks for this feedback :-)
We are just a little curious about seemingly high CPU usage, and how
best to interpret the results, if someone could advise? Specifically,
we are seeing the majority of the cpu time in 'system' and not 'user',
what would that suggest is happening perhaps?
Could you provide me the output from `knotc status workers`?
I like htop. Are you able to send me a screenshot when running htop? :-)
[Don't forget to configure:
F2
Display options->Detailed CPU time
(System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
F10
]
Best,
Daniel
Any tuning help gratefully received!
Our platform (sample)
24 Core CPU E5-2440 0 @ 2.40
64Gb RAM
SSD
knot-2.9.5, running on Centos 8.2
Currently receiving around 4k pps
Server is authoritative for a 278 zones, the largest zone file being
around 600Mb
Output of top
top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
12.34, 11.83
Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
%Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7
si, 0.0 st
%Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
si, 0.0 st
%Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
si, 0.0 st
%Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
si, 0.0 st
%Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
si, 0.0 st
%Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
buff/cache
MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
13:47
Jonathan,
there must be something wrong with the networking. How many `server.listen` configuration
values
do yo have? Is it a physical server or a virtual one? What is the network card used?
Daneil
On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
Daniel
Thank you for your time
# knotc status workers
UDP workers: 24, TCP workers 24, background workers: 10 (running: 0, pending: 0)
image.png
Kind regards
Jonathan
On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>> wrote:
Hi Jonathan,
On 2020-09-22 23:11, Jonathan Hewlett wrote:
Hi,
We run around 50 knot servers around the world, with great success -
thanks for the software!
Thanks for this feedback :-)
We are just a little curious about seemingly high CPU usage, and how
best to interpret the results, if someone could advise? Specifically,
we are seeing the majority of the cpu time in 'system' and not 'user',
what would that suggest is happening perhaps?
Could you provide me the output from `knotc status workers`?
I like htop. Are you able to send me a screenshot when running htop? :-)
[Don't forget to configure:
F2
Display options->Detailed CPU time
(System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
F10
]
Best,
Daniel
Any tuning help gratefully received!
Our platform (sample)
24 Core CPU E5-2440 0 @ 2.40
64Gb RAM
SSD
knot-2.9.5, running on Centos 8.2
Currently receiving around 4k pps
Server is authoritative for a 278 zones, the largest zone file being
around 600Mb
Output of top
top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
12.34, 11.83
Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
%Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7
si, 0.0 st
%Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
si, 0.0 st
%Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
si, 0.0 st
%Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
si, 0.0 st
%Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
si, 0.0 st
%Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
buff/cache
MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number
8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
17:45
Hi Daniel
* Server listens: 967 (multiple v4 and v6 networks)
* Physical server
* NIC details
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: eno1
version: 00
serial: b0:83:fe:c8:f3:ec
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom
ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd
autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3
driverversion=3.137 duplex=full firmware=FFV21.60.2 bc 5720-v1.39
ip=38.88.50.50 latency=0 link=yes multicast=yes port=twisted pair
speed=1Gbit/s
resources: irq:16 memory:d80a0000-d80affff memory:d80b0000-d80bffff
memory:d80c0000-d80cffff memory:dd800000-dd83ffff
Thanks for the continuing support
Jonathan
On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <daniel.salzman(a)nic.cz> wrote:
Jonathan,
there must be something wrong with the networking. How many
`server.listen` configuration values
do yo have? Is it a physical server or a virtual one? What is the network
card used?
Daneil
On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
how
:-)
company
number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
Daniel
Thank you for your time
# knotc status workers
UDP workers: 24, TCP workers 24, background workers: 10 (running: 0,
pending: 0)
image.png
Kind regards
Jonathan
On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz <mailto:
daniel.salzman(a)nic.cz>> wrote:
Hi Jonathan,
On 2020-09-22 23:11, Jonathan Hewlett wrote:
> Hi,
>
> We run around 50 knot servers around the world, with great success
-
thanks
for the software!
Thanks for this feedback :-)
>
> We are just a little curious about seemingly high CPU usage, and > best to interpret the results, if
someone could advise?
Specifically,
> we are seeing the majority of the cpu
time in 'system' and not
'user',
what
would that suggest is happening perhaps?
Could you provide me the output from `knotc status workers`?
I like htop. Are you able to send me a screenshot when running htop? [Don't forget to configure:
F2
Display options->Detailed CPU time
(System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
F10
]
Best,
Daniel
>
> Any tuning help gratefully received!
>
> Our platform (sample)
>
> 24 Core CPU E5-2440 0 @ 2.40
> 64Gb RAM
> SSD
>
> knot-2.9.5, running on Centos 8.2
>
> Currently receiving around 4k pps
>
> Server is authoritative for a 278 zones, the largest zone file
being
> around 600Mb
>
> Output of top
>
> top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
> 12.34, 11.83
> Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1
zombie
> %Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3
id, 0.3 wa, 0.7 hi, 0.7
> si, 0.0 st
> %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
> si, 0.0 st
> %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
> si, 0.0 st
> %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
> si, 0.0 st
> %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
> si, 0.0 st
> %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
> si, 0.0 st
> %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
> si, 0.0 st
> %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
> si, 0.0 st
> %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
> si, 0.0 st
> %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
> si, 0.0 st
> %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
> si, 0.0 st
> %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
> si, 0.0 st
> %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
> si, 0.0 st
> %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
> si, 0.0 st
> %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
> si, 0.0 st
> %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
> si, 0.0 st
> %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
> si, 0.0 st
> MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
> buff/cache
> MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0
avail
> Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> 213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3
1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
24 Zář
24 Zář
12:14
Hi Jonathan,
If you have just one 1GbE port, I would recommend reducing the number of udp-workers. What
about 12 or less?
How do you install Knot DNS (repository or source code)?
What do say these commands?
# sudo ethtool -l eno1
# knotc status configure
Daniel
On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
Hi Daniel
* Server listens: 967 (multiple v4 and v6 networks)
* Physical server
* NIC details
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: eno1
version: 00
serial: b0:83:fe:c8:f3:ec
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical
tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=3.137
duplex=full firmware=FFV21.60.2 bc 5720-v1.39 ip=38.88.50.50 latency=0 link=yes
multicast=yes port=twisted pair
speed=1Gbit/s
resources: irq:16 memory:d80a0000-d80affff memory:d80b0000-d80bffff
memory:d80c0000-d80cffff memory:dd800000-dd83ffff
Thanks for the continuing support
Jonathan
On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>> wrote:
Jonathan,
there must be something wrong with the networking. How many `server.listen`
configuration values
do yo have? Is it a physical server or a virtual one? What is the network card used?
Daneil
On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
Daniel
Thank you for your time
# knotc status workers
UDP workers: 24, TCP workers 24, background workers: 10 (running: 0, pending: 0)
image.png
Kind regards
Jonathan
On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>>> wrote:
Hi Jonathan,
On 2020-09-22 23:11, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number
8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE. Hi,
We run around 50 knot servers around the world, with great success -
thanks for the software!
Thanks for this feedback :-)
We are just a little curious about seemingly high CPU usage, and how
best to interpret the results, if someone could advise? Specifically,
we are seeing the majority of the cpu time in 'system' and not 'user',
what would that suggest is happening perhaps?
Could you provide me the output from `knotc status workers`?
I like htop. Are you able to send me a screenshot when running htop? :-)
[Don't forget to configure:
F2
Display options->Detailed CPU time
(System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
F10
]
Best,
Daniel
Any tuning help gratefully received!
Our platform (sample)
24 Core CPU E5-2440 0 @ 2.40
64Gb RAM
SSD
knot-2.9.5, running on Centos 8.2
Currently receiving around 4k pps
Server is authoritative for a 278 zones, the largest zone file being
around 600Mb
Output of top
top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
12.34, 11.83
Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
%Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7
si, 0.0 st
%Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
si, 0.0 st
%Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
si, 0.0 st
%Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
si, 0.0 st
%Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
si, 0.0 st
%Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
buff/cache
MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number
8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
8 Říj
8 Říj
2:12
Hi Danel,
I'm sorry for the late reply; I've been on holiday/vacation and playing
catch-up! ;)
ethtool:
Channel parameters for eno1:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 0
Current hardware settings:
RX: 4
TX: 1
Other: 0
Combined: 0
knotc output:
Knot DNS 2.9.5
Target: linux-gnu x86_64
Compiler: gcc
CFLAGS: -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-DNDEBUG -Wno-unused -Wall -Wshadow -Werror=format-security
-Werror=implicit -Wstrict-prototypes
LIBS: -Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
LibURCU: -lurcu
GnuTLS: -lgnutls -I/usr/include/p11-kit-1
Libedit: -ledit -I/usr/include/editline
LMDB: shared -llmdb
Config: 500 MiB default mapsize
Prefix: /usr
Run dir: /run/knot
Storage dir: /var/lib/knot
Config dir: /etc/knot
Module dir:
Static modules: cookies dnsproxy dnstap geoip noudp onlinesign queryacl
rrl stats synthrecord whoami
Shared modules:
Knot DNS libraries: yes
Knot DNS daemon: yes
Knot DNS utilities: yes
Knot DNS documentation: yes
Use recvmmsg: yes
Use SO_REUSEPORT(_LB): yes
Memory allocator: auto
Fast zone parser: yes
Utilities with IDN: libidn2
Utilities with Dnstap: yes
MaxMind DB support: yes
Systemd integration: yes
POSIX capabilities: yes
PKCS #11 support: yes
Ed25519 support: yes
Ed448 support: no
Code coverage: no
Sanitizer: no
LibFuzzer: no
OSS-Fuzz: no
Continued thanks! ;)
Jonathan
On Thu, 24 Sep 2020 at 11:14, Daniel Salzman <daniel.salzman(a)nic.cz> wrote:
Hi Jonathan,
If you have just one 1GbE port, I would recommend reducing the number of
udp-workers. What about 12 or less?
How do you install Knot DNS (repository or source code)?
What do say these commands?
# sudo ethtool -l eno1
# knotc status configure
Daniel
On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
Hi Daniel
* Server listens: 967 (multiple v4 and v6 networks)
* Physical server
* NIC details
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: eno1
version: 00
serial: b0:83:fe:c8:f3:ec
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom
ethernet
physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd
autonegotiation
configuration: autonegotiation=on
broadcast=yes driver=tg3
driverversion=3.137 duplex=full firmware=FFV21.60.2 bc
5720-v1.39
ip=38.88.50.50 latency=0 link=yes multicast=yes port=twisted pair
speed=1Gbit/s
resources: irq:16 memory:d80a0000-d80affff
memory:d80b0000-d80bffff
memory:d80c0000-d80cffff memory:dd800000-dd83ffff
Thanks for the continuing support
Jonathan
On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>> wrote:
Jonathan,
there must be something wrong with the networking. How many
`server.listen`
configuration values
do yo have? Is it a physical server or a
virtual one? What is the
network card used?
Daneil
On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
> Daniel
> Thank you for your time
>
> # knotc status workers
> UDP workers: 24, TCP workers 24, background workers: 10 (running:
0,
pending: 0)
>
> image.png
>
> Kind regards
> Jonathan
>
>
> On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz <mailto:
daniel.salzman(a)nic.cz> <mailto:daniel.salzman@nic.cz <mailto:
daniel.salzman(a)nic.cz>>> wrote:
>
> Hi Jonathan,
>
> On 2020-09-22 23:11, Jonathan Hewlett wrote:
> > Hi,
> >
> > We run around 50 knot servers around the world, with great
success -
> > thanks for the software!
>
> Thanks for this feedback :-)
>
> >
> > We are just a little curious about seemingly high CPU usage,
and
how
> > best to interpret the results,
if someone could advise?
Specifically,
> > we are seeing the majority of
the cpu time in 'system' and
not 'user',
> > what would that suggest is
happening perhaps?
>
> Could you provide me the output from `knotc status workers`?
>
> I like htop. Are you able to send me a screenshot when running
htop?
:-)
> [Don't forget to configure:
> F2
> Display options->Detailed CPU time
> (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
> F10
> ]
>
> Best,
> Daniel
>
> >
> > Any tuning help gratefully received!
> >
> > Our platform (sample)
> >
> > 24 Core CPU E5-2440 0 @ 2.40
> > 64Gb RAM
> > SSD
> >
> > knot-2.9.5, running on Centos 8.2
> >
> > Currently receiving around 4k pps
> >
> > Server is authoritative for a 278 zones, the largest zone
file
being
> > around 600Mb
> >
> > Output of top
> >
> > top - 20:50:38 up 6 days, 4:49, 2 users, load average:
13.47,
> > 12.34, 11.83
> > Tasks: 321 total, 2 running, 318 sleeping, 0 stopped,
1
zombie
> > %Cpu0 : 4.0 us, 34.1 sy, 0.0
ni, 60.3 id, 0.3 wa, 0.7
hi, 0.7
> > si, 0.0 st
> > %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3
hi,
0.7
> > si, 0.0 st
> > %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7
hi,
0.3
> > si, 0.0 st
> > %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3
hi,
0.3
> > si, 0.0 st
> > %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3
hi,
0.3
> > si, 0.0 st
> > %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7
hi,
0.0
> > si, 0.0 st
> > %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7
hi,
0.0
> > si, 0.0 st
> > %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7
hi,
0.3
> > si, 0.0 st
> > %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7
hi,
3.7
> > si, 0.0 st
> > %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3
hi,
0.3
> > si, 0.0 st
> > %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0
hi,
4.0
> > si, 0.0 st
> > %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7
hi,
4.0
> > si, 0.0 st
> > %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3
hi,
0.3
> > si, 0.0 st
> > %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7
hi,
1.3
> > si, 0.0 st
> > %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3
hi,
0.3
> > si, 0.0 st
> > %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7
hi,
3.7
> > si, 0.0 st
> > %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3
hi,
0.0
> > si, 0.0 st
> > MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used,
5608.8
> > buff/cache
> > MiB Swap: 32248.0 total, 32248.0 free, 0.0 used.
51670.0
avail
> > Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+
> > COMMAND
> > 213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3
1319:12
> > /usr/sbin/knotd
> >
> > Thanks again for this great product.
> >
> > Jonathan
>
>
>
> --
> --\--
> Jonathan Hewlett
> ^Senior\ Systems\ Engineer\n$
> CentralNic Group plc (LSE:CNIC)
> https://www.centralnic.com <https://www.centralnic.com/>
> Tel: +44.7548243020
> CentralNic Group plc is a company registered in England and Wales
with
company number 8576358. Registered Offices: Saddlers House, Gutter
Lane, London EC2V 6AE.
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company
number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
9:29
Hi Jonathan
Try increasing the number or transmit channels:
ethtool -L eno1 rx 4 tx 4
And reduce the number of udp-workers
Daniel
On 10/8/20 2:12 AM, Jonathan Hewlett wrote:
Hi Danel,
I'm sorry for the late reply; I've been on holiday/vacation and playing catch-up!
;)
ethtool:
Channel parameters for eno1:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 0
Current hardware settings:
RX: 4
TX: 1
Other: 0
Combined: 0
knotc output:
Knot DNS 2.9.5
Target: linux-gnu x86_64
Compiler: gcc
CFLAGS: -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -DNDEBUG
-Wno-unused -Wall -Wshadow -Werror=format-security -Werror=implicit -Wstrict-prototypes
LIBS: -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
LibURCU: -lurcu
GnuTLS: -lgnutls -I/usr/include/p11-kit-1
Libedit: -ledit -I/usr/include/editline
LMDB: shared -llmdb
Config: 500 MiB default mapsize
Prefix: /usr
Run dir: /run/knot
Storage dir: /var/lib/knot
Config dir: /etc/knot
Module dir:
Static modules: cookies dnsproxy dnstap geoip noudp onlinesign queryacl rrl stats
synthrecordwhoami
Shared modules:
Knot DNS libraries: yes
Knot DNS daemon: yes
Knot DNS utilities: yes
Knot DNS documentation: yes
Use recvmmsg: yes
Use SO_REUSEPORT(_LB): yes
Memory allocator: auto
Fast zone parser: yes
Utilities with IDN: libidn2
Utilities with Dnstap: yes
MaxMind DB support: yes
Systemd integration: yes
POSIX capabilities: yes
PKCS #11 support: yes
Ed25519 support: yes
Ed448 support: no
Code coverage: no
Sanitizer: no
LibFuzzer: no
OSS-Fuzz: no
Continued thanks! ;)
Jonathan
On Thu, 24 Sep 2020 at 11:14, Daniel Salzman <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>> wrote:
Hi Jonathan,
If you have just one 1GbE port, I would recommend reducing the number of udp-workers.
What about 12 or less?
How do you install Knot DNS (repository or source code)?
What do say these commands?
# sudo ethtool -l eno1
# knotc status configure
Daniel
On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number
8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
Hi Daniel
* Server listens: 967 (multiple v4 and v6 networks)
* Physical server
* NIC details
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: eno1
version: 00
serial: b0:83:fe:c8:f3:ec
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical
tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=3.137
duplex=full firmware=FFV21.60.2 bc 5720-v1.39 ip=38.88.50.50 latency=0 link=yes
multicast=yes port=twisted pair
speed=1Gbit/s
resources: irq:16 memory:d80a0000-d80affff memory:d80b0000-d80bffff
memory:d80c0000-d80cffff memory:dd800000-dd83ffff
Thanks for the continuing support
Jonathan
On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>>> wrote:
Jonathan,
there must be something wrong with the networking. How many `server.listen`
configuration values
do yo have? Is it a physical server or a virtual one? What is the network card used?
Daneil
On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
> Daniel
> Thank you for your time
>
> # knotc status workers
> UDP workers: 24, TCP workers 24, background workers: 10 (running: 0, pending:
0)
>
> image.png
>
> Kind regards
> Jonathan
>
>
> On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>>>> wrote:
Hi Jonathan,
On 2020-09-22 23:11, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number
8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE. Hi,
We run around 50 knot servers around the world, with great success -
thanks for the software!
Thanks for this feedback :-)
We are just a little curious about seemingly high CPU usage, and how
best to interpret the results, if someone could advise? Specifically,
we are seeing the majority of the cpu time in 'system' and not 'user',
what would that suggest is happening perhaps?
Could you provide me the output from `knotc status workers`?
I like htop. Are you able to send me a screenshot when running htop? :-)
[Don't forget to configure:
F2
Display options->Detailed CPU time
(System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
F10
]
Best,
Daniel
Any tuning help gratefully received!
Our platform (sample)
24 Core CPU E5-2440 0 @ 2.40
64Gb RAM
SSD
knot-2.9.5, running on Centos 8.2
Currently receiving around 4k pps
Server is authoritative for a 278 zones, the largest zone file being
around 600Mb
Output of top
top - 20:50:38 up 6 days, 4:49, 2 users, load average: 13.47,
12.34, 11.83
Tasks: 321 total, 2 running, 318 sleeping, 0 stopped, 1 zombie
%Cpu0 : 4.0 us, 34.1 sy, 0.0 ni, 60.3 id, 0.3 wa, 0.7 hi, 0.7
si, 0.0 st
%Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa, 0.3 hi, 0.7
si, 0.0 st
%Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa, 1.0 hi, 4.0
si, 0.0 st
%Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
%Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa, 0.7 hi, 4.0
si, 0.0 st
%Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa, 0.7 hi, 1.3
si, 0.0 st
%Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa, 0.3 hi, 0.3
si, 0.0 st
%Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa, 0.7 hi, 3.7
si, 0.0 st
%Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa, 0.3 hi, 0.0
si, 0.0 st
MiB Mem : 64107.4 total, 46764.9 free, 11733.6 used, 5608.8
buff/cache
MiB Swap: 32248.0 total, 32248.0 free, 0.0 used. 51670.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
213370 knot 20 0 34.0g 9.6g 41784 S 644.9 15.3 1319:12
/usr/sbin/knotd
Thanks again for this great product.
Jonathan
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number
8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
20:04
Hi Daniel
I've made the suggested changes, a slight improvement I think, but load is
still fairly high:
[image: image.png]
Any further suggestions greatly appreciated.
Thanks
Jonathan
On Thu, 8 Oct 2020 at 08:29, Daniel Salzman <daniel.salzman(a)nic.cz> wrote:
Hi Jonathan
Try increasing the number or transmit channels:
ethtool -L eno1 rx 4 tx 4
And reduce the number of udp-workers
Daniel
On 10/8/20 2:12 AM, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
Hi Danel,
I'm sorry for the late reply; I've been on holiday/vacation and playing
catch-up! ;)
ethtool:
Channel parameters for eno1:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 0
Current hardware settings:
RX: 4
TX: 1
Other: 0
Combined: 0
knotc output:
Knot DNS 2.9.5
Target: linux-gnu x86_64
Compiler: gcc
CFLAGS: -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-DNDEBUG
-Wno-unused -Wall -Wshadow
-Werror=format-security -Werror=implicit
-Wstrict-prototypes
LIBS: -Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
LibURCU: -lurcu
GnuTLS: -lgnutls -I/usr/include/p11-kit-1
Libedit: -ledit -I/usr/include/editline
LMDB: shared -llmdb
Config: 500 MiB default mapsize
Prefix: /usr
Run dir: /run/knot
Storage dir: /var/lib/knot
Config dir: /etc/knot
Module dir:
Static modules: cookies dnsproxy dnstap geoip noudp onlinesign
queryacl rrl
stats synthrecordwhoami
Shared modules:
Knot DNS libraries: yes
Knot DNS daemon: yes
Knot DNS utilities: yes
Knot DNS documentation: yes
Use recvmmsg: yes
Use SO_REUSEPORT(_LB): yes
Memory allocator: auto
Fast zone parser: yes
Utilities with IDN: libidn2
Utilities with Dnstap: yes
MaxMind DB support: yes
Systemd integration: yes
POSIX capabilities: yes
PKCS #11 support: yes
Ed25519 support: yes
Ed448 support: no
Code coverage: no
Sanitizer: no
LibFuzzer: no
OSS-Fuzz: no
Continued thanks! ;)
Jonathan
On Thu, 24 Sep 2020 at 11:14, Daniel Salzman <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>> wrote:
Hi Jonathan,
If you have just one 1GbE port, I would recommend reducing the
number of
udp-workers. What about 12 or less?
How do you install Knot DNS (repository or source code)?
What do say these commands?
# sudo ethtool -l eno1
# knotc status configure
Daniel
On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
> Hi Daniel
>
> * Server listens: 967 (multiple v4 and v6 networks)
> * Physical server
> * NIC details
> *-network:0
> description: Ethernet interface
> product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
> vendor: Broadcom Inc. and subsidiaries
> physical id: 0
> bus info: pci@0000:02:00.0
> logical name: eno1
> version: 00
> serial: b0:83:fe:c8:f3:ec
> size: 1Gbit/s
> capacity: 1Gbit/s
> width: 64 bits
> clock: 33MHz
> capabilities: pm vpd msi msix pciexpress bus_master
cap_list rom
ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt
1000bt-fd autonegotiation
> configuration: autonegotiation=on
broadcast=yes driver=tg3
driverversion=3.137 duplex=full firmware=FFV21.60.2 bc
5720-v1.39
ip=38.88.50.50 latency=0 link=yes multicast=yes port=twisted pair
> speed=1Gbit/s
> resources: irq:16 memory:d80a0000-d80affff
memory:d80b0000-d80bffff memory:d80c0000-d80cffff memory:dd800000-dd83ffff
>
> Thanks for the continuing support
> Jonathan
>
>
> On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <
daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:
daniel.salzman(a)nic.cz <mailto:daniel.salzman@nic.cz>>> wrote:
>
> Jonathan,
>
> there must be something wrong with the networking. How many
`server.listen` configuration values
> do yo have? Is it a physical server
or a virtual one? What is
the network card used?
>
> Daneil
>
> On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
> > Daniel
> > Thank you for your time
> >
> > # knotc status workers
> > UDP workers: 24, TCP workers 24, background workers: 10
(running: 0, pending: 0)
> >
> > image.png
> >
> > Kind regards
> > Jonathan
> >
> >
> > On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz <mailto:
daniel.salzman(a)nic.cz>> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>
<mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>>>>
wrote:
> >
> > Hi Jonathan,
> >
> > On 2020-09-22 23:11, Jonathan Hewlett wrote:
> > > Hi,
> > >
> > > We run around 50 knot servers around the world, with
great success -
> > > thanks for the
software!
> >
> > Thanks for this feedback :-)
> >
> > >
> > > We are just a little curious about seemingly high CPU
usage, and how
> > > best to interpret the
results, if someone could
advise? Specifically,
> > > we are seeing the
majority of the cpu time in 'system'
and not 'user',
> > > what would that
suggest is happening perhaps?
> >
> > Could you provide me the output from `knotc status
workers`?
> >
> > I like htop. Are you able to send me a screenshot when
running htop? :-)
> > [Don't forget to
configure:
> > F2
> > Display options->Detailed CPU time
> > (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
> > F10
> > ]
> >
> > Best,
> > Daniel
> >
> > >
> > > Any tuning help gratefully received!
> > >
> > > Our platform (sample)
> > >
> > > 24 Core CPU E5-2440 0 @ 2.40
> > > 64Gb RAM
> > > SSD
> > >
> > > knot-2.9.5, running on Centos 8.2
> > >
> > > Currently receiving around 4k pps
> > >
> > > Server is authoritative for a 278 zones, the largest
zone file being
> > > around 600Mb
> > >
> > > Output of top
> > >
> > > top - 20:50:38 up 6 days, 4:49, 2 users, load
average: 13.47,
> > > 12.34, 11.83
> > > Tasks: 321 total, 2 running, 318 sleeping, 0
stopped, 1 zombie
> > > %Cpu0 : 4.0 us, 34.1
sy, 0.0 ni, 60.3 id, 0.3 wa,
0.7 hi, 0.7
> > > si, 0.0 st
> > > %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id, 0.0 wa,
0.3 hi, 0.7
> > > si, 0.0 st
> > > %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id, 0.0 wa,
0.7 hi, 0.3
> > > si, 0.0 st
> > > %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id, 0.0 wa,
0.3 hi, 0.3
> > > si, 0.0 st
> > > %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id, 0.0 wa,
0.3 hi, 0.3
> > > si, 0.0 st
> > > %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id, 0.0 wa,
0.7 hi, 0.0
> > > si, 0.0 st
> > > %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id, 0.0 wa,
0.7 hi, 0.0
> > > si, 0.0 st
> > > %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id, 0.0 wa,
0.7 hi, 0.3
> > > si, 0.0 st
> > > %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id, 0.0 wa,
0.7 hi, 3.7
> > > si, 0.0 st
> > > %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id, 0.0 wa,
0.3 hi, 0.3
> > > si, 0.0 st
> > > %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id, 0.0 wa,
1.0 hi, 4.0
> > > si, 0.0 st
> > > %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id, 0.0 wa,
0.7 hi, 4.0
> > > si, 0.0 st
> > > %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id, 0.0 wa,
0.3 hi, 0.3
> > > si, 0.0 st
> > > %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id, 0.0 wa,
0.7 hi, 1.3
> > > si, 0.0 st
> > > %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id, 0.0 wa,
0.3 hi, 0.3
> > > si, 0.0 st
> > > %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id, 0.0 wa,
0.7 hi, 3.7
> > > si, 0.0 st
> > > %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id, 0.0 wa,
0.3 hi, 0.0
> > > si, 0.0 st
> > > MiB Mem : 64107.4 total, 46764.9 free, 11733.6
used,
5608.8
> > > buff/cache
> > > MiB Swap: 32248.0 total, 32248.0 free, 0.0
used.
51670.0 avail
> > > Mem
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU
%MEM TIME+
> > > COMMAND
> > > 213370 knot 20 0 34.0g 9.6g 41784 S 644.9
15.3 1319:12
> > > /usr/sbin/knotd
> > >
> > > Thanks again for this great product.
> > >
> > > Jonathan
> >
> >
> >
> > --
> > --\--
> > Jonathan Hewlett
> > ^Senior\ Systems\ Engineer\n$
> > CentralNic Group plc (LSE:CNIC)
> > https://www.centralnic.com <https://www.centralnic.com/>
> > Tel: +44.7548243020
> > CentralNic Group plc is a company registered in England and
Wales with company number 8576358. Registered Offices: Saddlers House,
Gutter Lane, London EC2V 6AE.
>
>
>
> --
> --\--
> Jonathan Hewlett
> ^Senior\ Systems\ Engineer\n$
> CentralNic Group plc (LSE:CNIC)
> https://www.centralnic.com <https://www.centralnic.com/>
> Tel: +44.7548243020
> CentralNic Group plc is a company registered in England and Wales
with
company number 8576358. Registered Offices: Saddlers House, Gutter
Lane, London EC2V 6AE.
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company
number 8576358. Registered Offices: Saddlers House, Gutter Lane,
London EC2V 6AE.
20:10
What is the new number of udp workers? Have tried 4 or 8?
On 2020-10-08 20:04, Jonathan Hewlett wrote:
Hi Daniel
I've made the suggested changes, a slight improvement I think, but
load is still fairly high:
Any further suggestions greatly appreciated.
Thanks
Jonathan
On Thu, 8 Oct 2020 at 08:29, Daniel Salzman <daniel.salzman(a)nic.cz>
wrote:
Hi Jonathan
Try increasing the number or transmit channels:
ethtool -L eno1 rx 4 tx 4
And reduce the number of udp-workers
Daniel
On 10/8/20 2:12 AM, Jonathan Hewlett wrote:
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com [1]
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with
company number 8576358. Registered Offices: Saddlers House, Gutter
Lane, London EC2V 6AE.
Links:
------
[1] https://www.centralnic.com/ Hi Danel,
I'm sorry for the late reply; I've been on holiday/vacation and
playing
catch-up! ;)
ethtool:
Channel parameters for eno1:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 0
Current hardware settings:
RX: 4
TX: 1
Other: 0
Combined: 0
knotc output:
Knot DNS 2.9.5
Target: linux-gnu x86_64
Compiler: gcc
CFLAGS: -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection -DNDEBUG
-Wno-unused -Wall -Wshadow
-Werror=format-security
-Werror=implicit -Wstrict-prototypes
LIBS: -Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
LibURCU: -lurcu
GnuTLS: -lgnutls -I/usr/include/p11-kit-1
Libedit: -ledit -I/usr/include/editline
LMDB: shared -llmdb
Config: 500 MiB default mapsize
Prefix: /usr
Run dir: /run/knot
Storage dir: /var/lib/knot
Config dir: /etc/knot
Module dir:
Static modules: cookies dnsproxy dnstap geoip noudp onlinesign
queryacl rrl stats
synthrecordwhoami
Shared modules:
Knot DNS libraries: yes
Knot DNS daemon: yes
Knot DNS utilities: yes
Knot DNS documentation: yes
Use recvmmsg: yes
Use SO_REUSEPORT(_LB): yes
Memory allocator: auto
Fast zone parser: yes
Utilities with IDN: libidn2
Utilities with Dnstap: yes
MaxMind DB support: yes
Systemd integration: yes
POSIX capabilities: yes
PKCS #11 support: yes
Ed25519 support: yes
Ed448 support: no
Code coverage: no
Sanitizer: no
LibFuzzer: no
OSS-Fuzz: no
Continued thanks! ;)
Jonathan
On Thu, 24 Sep 2020 at 11:14, Daniel Salzman
<daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>> wrote:
Hi Jonathan,
If you have just one 1GbE port, I would recommend reducing the
number of
udp-workers. What about 12 or less?
How do you install Knot DNS (repository or source code)?
What do say these commands?
# sudo ethtool -l eno1
# knotc status configure
Daniel
On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
> Hi Daniel
>
> * Server listens: 967 (multiple v4 and v6 networks)
> * Physical server
> * NIC details
> *-network:0
> description: Ethernet interface
> product: NetXtreme BCM5720 2-port Gigabit Ethernet
PCIe
> vendor: Broadcom Inc. and
subsidiaries
> physical id: 0
> bus info: pci@0000:02:00.0
> logical name: eno1
> version: 00
> serial: b0:83:fe:c8:f3:ec
> size: 1Gbit/s
> capacity: 1Gbit/s
> width: 64 bits
> clock: 33MHz
> capabilities: pm vpd msi msix pciexpress bus_master
cap_list rom
ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt
1000bt-fd autonegotiation
> configuration: autonegotiation=on
broadcast=yes
driver=tg3 driverversion=3.137 duplex=full firmware=FFV21.60.2 bc
5720-v1.39 ip=38.88.50.50 latency=0 link=yes multicast=yes
port=twisted pair
> speed=1Gbit/s
> resources: irq:16 memory:d80a0000-d80affff
memory:d80b0000-d80bffff
memory:d80c0000-d80cffff
memory:dd800000-dd83ffff
>
> Thanks for the continuing support
> Jonathan
>
>
> On Wed, 23 Sep 2020 at 12:47, Daniel Salzman
<daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz>
<mailto:daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>>>
wrote:
>
> Jonathan,
>
> there must be something wrong with the networking. How
many
`server.listen` configuration values
> do yo have? Is it a physical server or a
virtual one?
What is the network card used?
>
> Daneil
>
> On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
> > Daniel
> > Thank you for your time
> >
> > # knotc status workers
> > UDP workers: 24, TCP workers 24, background workers:
10 (running: 0,
pending: 0)
> >
> > image.png
> >
> > Kind regards
> > Jonathan
> >
> >
> > On Wed, 23 Sep 2020 at 07:59, <daniel.salzman(a)nic.cz
<mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>> <mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>
<mailto:daniel.salzman@nic.cz
<mailto:daniel.salzman@nic.cz>>>>
wrote:
> >
> > Hi Jonathan,
> >
> > On 2020-09-22 23:11, Jonathan Hewlett wrote:
> > > Hi,
> > >
> > > We run around 50 knot servers around the world,
with great
success -
> > > thanks for the software!
> >
> > Thanks for this feedback :-)
> >
> > >
> > > We are just a little curious about seemingly
high CPU usage,
and how
> > > best to interpret the
results, if someone could
advise? Specifically,
> > > we are seeing the majority
of the cpu time in
'system' and not 'user',
> > > what would that suggest is
happening perhaps?
> >
> > Could you provide me the output from `knotc status
workers`?
> >
> > I like htop. Are you able to send me a screenshot
when running
htop? :-)
> > [Don't forget to configure:
> > F2
> > Display options->Detailed CPU time
> > (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
> > F10
> > ]
> >
> > Best,
> > Daniel
> >
> > >
> > > Any tuning help gratefully received!
> > >
> > > Our platform (sample)
> > >
> > > 24 Core CPU E5-2440 0 @ 2.40
> > > 64Gb RAM
> > > SSD
> > >
> > > knot-2.9.5, running on Centos 8.2
> > >
> > > Currently receiving around 4k pps
> > >
> > > Server is authoritative for a 278 zones, the
largest zone
file being
> > > around 600Mb
> > >
> > > Output of top
> > >
> > > top - 20:50:38 up 6 days, 4:49, 2 users, load
average:
13.47,
> > > 12.34, 11.83
> > > Tasks: 321 total, 2 running, 318 sleeping, 0
stopped,
1 zombie
> > > %Cpu0 : 4.0 us, 34.1 sy,
0.0 ni, 60.3 id,
0.3 wa, 0.7 hi, 0.7
> > > si, 0.0 st
> > > %Cpu1 : 4.3 us, 19.4 sy, 0.0 ni, 75.3 id,
0.0 wa, 0.3
hi, 0.7
> > > si, 0.0 st
> > > %Cpu2 : 4.3 us, 33.8 sy, 0.0 ni, 60.9 id,
0.0 wa, 0.7
hi, 0.3
> > > si, 0.0 st
> > > %Cpu3 : 5.0 us, 21.2 sy, 0.0 ni, 73.2 id,
0.0 wa, 0.3
hi, 0.3
> > > si, 0.0 st
> > > %Cpu4 : 3.3 us, 34.9 sy, 0.0 ni, 61.5 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu5 : 3.0 us, 20.2 sy, 0.0 ni, 76.2 id,
0.0 wa, 0.3
hi, 0.3
> > > si, 0.0 st
> > > %Cpu6 : 4.0 us, 35.2 sy, 0.0 ni, 60.4 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu7 : 5.0 us, 18.7 sy, 0.0 ni, 76.0 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu8 : 2.0 us, 35.0 sy, 0.0 ni, 62.3 id,
0.0 wa, 0.7
hi, 0.0
> > > si, 0.0 st
> > > %Cpu9 : 2.0 us, 20.0 sy, 0.0 ni, 77.7 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu10 : 3.7 us, 34.4 sy, 0.0 ni, 61.2 id,
0.0 wa, 0.7
hi, 0.0
> > > si, 0.0 st
> > > %Cpu11 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu12 : 2.3 us, 34.6 sy, 0.0 ni, 62.1 id,
0.0 wa, 0.7
hi, 0.3
> > > si, 0.0 st
> > > %Cpu13 : 3.3 us, 20.7 sy, 0.0 ni, 75.7 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu14 : 2.3 us, 33.6 sy, 0.0 ni, 59.7 id,
0.0 wa, 0.7
hi, 3.7
> > > si, 0.0 st
> > > %Cpu15 : 3.3 us, 19.7 sy, 0.0 ni, 76.3 id,
0.0 wa, 0.3
hi, 0.3
> > > si, 0.0 st
> > > %Cpu16 : 2.3 us, 34.0 sy, 0.0 ni, 58.7 id,
0.0 wa, 1.0
hi, 4.0
> > > si, 0.0 st
> > > %Cpu17 : 2.3 us, 21.5 sy, 0.0 ni, 75.8 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > %Cpu18 : 3.3 us, 34.8 sy, 0.0 ni, 57.2 id,
0.0 wa, 0.7
hi, 4.0
> > > si, 0.0 st
> > > %Cpu19 : 5.0 us, 19.3 sy, 0.0 ni, 75.1 id,
0.0 wa, 0.3
hi, 0.3
> > > si, 0.0 st
> > > %Cpu20 : 2.7 us, 35.3 sy, 0.0 ni, 60.0 id,
0.0 wa, 0.7
hi, 1.3
> > > si, 0.0 st
> > > %Cpu21 : 1.7 us, 20.5 sy, 0.0 ni, 77.2 id,
0.0 wa, 0.3
hi, 0.3
> > > si, 0.0 st
> > > %Cpu22 : 4.7 us, 32.4 sy, 0.0 ni, 58.5 id,
0.0 wa, 0.7
hi, 3.7
> > > si, 0.0 st
> > > %Cpu23 : 2.3 us, 19.0 sy, 0.0 ni, 78.3 id,
0.0 wa, 0.3
hi, 0.0
> > > si, 0.0 st
> > > MiB Mem : 64107.4 total, 46764.9 free,
11733.6 used,
5608.8
> > > buff/cache
> > > MiB Swap: 32248.0 total, 32248.0 free,
0.0 used. 51670.0
avail
> > > Mem
> > >
> > > PID USER PR NI VIRT RES SHR S
%CPU %MEM
TIME+
> > > COMMAND
> > > 213370 knot 20 0 34.0g 9.6g 41784 S
644.9 15.3
1319:12
> > > /usr/sbin/knotd
> > >
> > > Thanks again for this great product.
> > >
> > > Jonathan
> >
> >
> >
> > --
> > --\--
> > Jonathan Hewlett
> > ^Senior\ Systems\ Engineer\n$
> > CentralNic Group plc (LSE:CNIC)
> > https://www.centralnic.com
<https://www.centralnic.com/>
> > Tel: +44.7548243020
> > CentralNic Group plc is a company registered in
England and Wales
with company number 8576358. Registered Offices:
Saddlers House, Gutter Lane, London EC2V 6AE.
>
>
>
> --
> --\--
> Jonathan Hewlett
> ^Senior\ Systems\ Engineer\n$
> CentralNic Group plc (LSE:CNIC)
> https://www.centralnic.com <https://www.centralnic.com/>
> Tel: +44.7548243020
> CentralNic Group plc is a company registered in England and
Wales with company
number 8576358. Registered Offices: Saddlers
House, Gutter Lane, London EC2V 6AE.
--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com <https://www.centralnic.com/>
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales
with company
number 8576358. Registered Offices: Saddlers House,
Gutter Lane, London EC2V 6AE.
1302
days inactive
1318
days old
9 comments
3 participants
participants (3)
-
Daniel Salzman -
daniel.salzman＠nic.cz
-
Jonathan Hewlett