3 Dub
2019
3 Dub
'19
10:45
I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
as a workload client. Knot server has 32 cores. Interrupts from 10Gb
network card are spreaded across all 32 cores. Knot configured with
64 udp-workers. Each knot thread assigned to one core. So there are at
least two knot threads assigned to one core. Then i start dnsperf with
command
./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
htop on knot server shows 3-4 cores completly unused. Then i restart
dnsperf unused cores are changes.
That is the reason for unused core?
3 Dub
3 Dub
10:52
On 03. 04. 19 10:45, Sergey Petrov wrote:
I perfoms benchmarks with knot-dns as a authoritative
server and dnsperf
as a workload client. Knot server has 32 cores. Interrupts from 10Gb
network card are spreaded across all 32 cores. Knot configured with
64 udp-workers. Each knot thread assigned to one core. So there are at
least two knot threads assigned to one core. Then i start dnsperf with
command
./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
htop on knot server shows 3-4 cores completly unused. Then i restart
dnsperf unused cores are changes.
That is the reason for unused core?
Well, sometimes dnsperf is too slow :-)
I recommend to check this:
- Make sure dnsperf ("source machine") is not 100 % utilized.
- Try to increase number of sockets used by dnsperf, i.e. -c parameter.
I would try also values like 500 and 1000 to see if it makes any
difference. It might change results significantly because Linux kernel
is using hashes over some packet fields and low number of sockets might
result in uneven query distribution.
Please let us know what are your new results.
--
Petr Špaček @ CZ.NIC
11:53
On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
On 03. 04. 19 10:45, Sergey Petrov wrote:
The source machne is about 15% utilized.
./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
get us some performance penalty (260000 rps VS 310000 rps) and more even
distribution across all cores with 100% usages of all eight cores on
last CPU socket. While other CPU socket cores are aproximately 60%
loaded.
Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
core load distribution and even more performance penalty.
Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
utilized cores and 2 50% utilized cores with about 300000 rps
The question is that prevents knot thread on 0% used core to serve a
packet arrived with IRQ bounded to another core? May be you have some
developer guide can answer this question?
I perfoms benchmarks with knot-dns as a
authoritative server and dnsperf
as a workload client. Knot server has 32 cores. Interrupts from 10Gb
network card are spreaded across all 32 cores. Knot configured with
64 udp-workers. Each knot thread assigned to one core. So there are at
least two knot threads assigned to one core. Then i start dnsperf with
command
./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
htop on knot server shows 3-4 cores completly unused. Then i restart
dnsperf unused cores are changes.
That is the reason for unused core?
Well, sometimes dnsperf is too slow :-)
I recommend to check this:
- Make sure dnsperf ("source machine") is not 100 % utilized.
- Try to increase number of sockets used by dnsperf, i.e. -c parameter.
I would try also values like 500 and 1000 to see if it makes any
difference. It might change results significantly because Linux kernel
is using hashes over some packet fields and low number of sockets might
result in uneven query distribution.
Please let us know what are your new results.
12:28
Hi,
We don't have any developer guide. In general, every network card has different
characteristic and features, so it's almost impossible to provide a comprehensive
manual for that.
Daniel
On 4/3/19 11:53 AM, Sergey Petrov wrote:
On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
On 03. 04. 19 10:45, Sergey Petrov wrote:
The source machne is about 15% utilized.
./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
get us some performance penalty (260000 rps VS 310000 rps) and more even
distribution across all cores with 100% usages of all eight cores on
last CPU socket. While other CPU socket cores are aproximately 60%
loaded.
Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
core load distribution and even more performance penalty.
Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
utilized cores and 2 50% utilized cores with about 300000 rps
The question is that prevents knot thread on 0% used core to serve a
packet arrived with IRQ bounded to another core? May be you have some
developer guide can answer this question?
I perfoms benchmarks with knot-dns as a
authoritative server and dnsperf
as a workload client. Knot server has 32 cores. Interrupts from 10Gb
network card are spreaded across all 32 cores. Knot configured with
64 udp-workers. Each knot thread assigned to one core. So there are at
least two knot threads assigned to one core. Then i start dnsperf with
command
./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
htop on knot server shows 3-4 cores completly unused. Then i restart
dnsperf unused cores are changes.
That is the reason for unused core?
Well, sometimes dnsperf is too slow :-)
I recommend to check this:
- Make sure dnsperf ("source machine") is not 100 % utilized.
- Try to increase number of sockets used by dnsperf, i.e. -c parameter.
I would try also values like 500 and 1000 to see if it makes any
difference. It might change results significantly because Linux kernel
is using hashes over some packet fields and low number of sockets might
result in uneven query distribution.
Please let us know what are your new results.
16:37
I reverse the client and the server. So the server now is 36-cores intel
box (72 HT-core)
Starting with small loads i see knot use lower cores except core-0.
When adding more load i see cores 0-17 AND 37-54 are used but not to
100% level. At maximum load i see all cores are about 100% used.
It seems to me as system scheduler feature. First it starts with lower
number cores, then add cores from second CPU socket and after all
HT-cores.
On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
On 03. 04. 19 10:45, Sergey Petrov wrote:
The source machne is about 15% utilized.
./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
get us some performance penalty (260000 rps VS 310000 rps) and more even
distribution across all cores with 100% usages of all eight cores on
last CPU socket. While other CPU socket cores are aproximately 60%
loaded.
Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
core load distribution and even more performance penalty.
Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
utilized cores and 2 50% utilized cores with about 300000 rps
The question is that prevents knot thread on 0% used core to serve a
packet arrived with IRQ bounded to another core? May be you have some
developer guide can answer this question?
I perfoms benchmarks with knot-dns as a
authoritative server and dnsperf
as a workload client. Knot server has 32 cores. Interrupts from 10Gb
network card are spreaded across all 32 cores. Knot configured with
64 udp-workers. Each knot thread assigned to one core. So there are at
least two knot threads assigned to one core. Then i start dnsperf with
command
./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
htop on knot server shows 3-4 cores completly unused. Then i restart
dnsperf unused cores are changes.
That is the reason for unused core?
Well, sometimes dnsperf is too slow :-)
I recommend to check this:
- Make sure dnsperf ("source machine") is not 100 % utilized.
- Try to increase number of sockets used by dnsperf, i.e. -c parameter.
I would try also values like 500 and 1000 to see if it makes any
difference. It might change results significantly because Linux kernel
is using hashes over some packet fields and low number of sockets might
result in uneven query distribution.
Please let us know what are your new results.
16:56
Hello,
as you already found out it is complicated ;-)
Linux kernel has its own magic algorithms to schedule work on
multi-core/multi-socket/NUMA machines and it DNS benchmarking also very
much depends on network card, its drivers etc.
If we were going to fine-tune your setup we would have to go into details:
What is your CPU architecture? Number of sockets, CPU in them etc.?
How is operating memory connected to CPUs?
Is it NUMA?
Do you have irqbalance enabled?
Have you somehow configured IRQ affinity?
What is your network card (how many IO queues it has)?
Did you configure network card queues and other driver settings explicitly?
etc.
Fine tunning always has to take into account your specific environment
and it is hard to provide general advice.
If you find specific reproducible problem please report it to our Gitlab:
https://gitlab.labs.nic.cz/knot/knot-dns/issues/
Please understand that amount of time and hardware we can allocate for
free support is limited. In case you require fine-tunning for your
specific deployment please consider byuing professional support:
https://www.knot-dns.cz/support/
Thank you for understanding.
Petr Špaček @ CZ.NIC
On 03. 04. 19 16:37, Sergey Petrov wrote:
I reverse the client and the server. So the server now
is 36-cores intel
box (72 HT-core)
Starting with small loads i see knot use lower cores except core-0.
When adding more load i see cores 0-17 AND 37-54 are used but not to
100% level. At maximum load i see all cores are about 100% used.
It seems to me as system scheduler feature. First it starts with lower
number cores, then add cores from second CPU socket and after all
HT-cores.
On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
> On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
>> On 03. 04. 19 10:45, Sergey Petrov wrote:
>>> I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
>>> as a workload client. Knot server has 32 cores. Interrupts from 10Gb
>>> network card are spreaded across all 32 cores. Knot configured with
>>> 64 udp-workers. Each knot thread assigned to one core. So there are at
>>> least two knot threads assigned to one core. Then i start dnsperf with
>>> command
>>>
>>> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
>>>
>>> htop on knot server shows 3-4 cores completly unused. Then i restart
>>> dnsperf unused cores are changes.
>>>
>>> That is the reason for unused core?
>>
>> Well, sometimes dnsperf is too slow :-)
>>
>> I recommend to check this:
>> - Make sure dnsperf ("source machine") is not 100 % utilized.
>> - Try to increase number of sockets used by dnsperf, i.e. -c parameter.
>> I would try also values like 500 and 1000 to see if it makes any
>> difference. It might change results significantly because Linux kernel
>> is using hashes over some packet fields and low number of sockets might
>> result in uneven query distribution.
>>
>> Please let us know what are your new results.
>>
>
> The source machne is about 15% utilized.
>
> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
>
> get us some performance penalty (260000 rps VS 310000 rps) and more even
> distribution across all cores with 100% usages of all eight cores on
> last CPU socket. While other CPU socket cores are aproximately 60%
> loaded.
>
> Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
> core load distribution and even more performance penalty.
>
> Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
> utilized cores and 2 50% utilized cores with about 300000 rps
>
> The question is that prevents knot thread on 0% used core to serve a
> packet arrived with IRQ bounded to another core? May be you have some
> developer guide can answer this question?
>
17:05
Thanks a lot for you attention!
I will answer next day. The answer will be long.
On Wed, 2019-04-03 at 16:56 +0200, Petr Špaček wrote:
Hello,
as you already found out it is complicated ;-)
Linux kernel has its own magic algorithms to schedule work on
multi-core/multi-socket/NUMA machines and it DNS benchmarking also very
much depends on network card, its drivers etc.
If we were going to fine-tune your setup we would have to go into details:
What is your CPU architecture? Number of sockets, CPU in them etc.?
How is operating memory connected to CPUs?
Is it NUMA?
Do you have irqbalance enabled?
Have you somehow configured IRQ affinity?
What is your network card (how many IO queues it has)?
Did you configure network card queues and other driver settings explicitly?
etc.
Fine tunning always has to take into account your specific environment
and it is hard to provide general advice.
If you find specific reproducible problem please report it to our Gitlab:
https://gitlab.labs.nic.cz/knot/knot-dns/issues/
Please understand that amount of time and hardware we can allocate for
free support is limited. In case you require fine-tunning for your
specific deployment please consider byuing professional support:
https://www.knot-dns.cz/support/
Thank you for understanding.
Petr Špaček @ CZ.NIC
On 03. 04. 19 16:37, Sergey Petrov wrote:
I reverse the client and the server. So the
server now is 36-cores intel
box (72 HT-core)
Starting with small loads i see knot use lower cores except core-0.
When adding more load i see cores 0-17 AND 37-54 are used but not to
100% level. At maximum load i see all cores are about 100% used.
It seems to me as system scheduler feature. First it starts with lower
number cores, then add cores from second CPU socket and after all
HT-cores.
On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
> On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
>> On 03. 04. 19 10:45, Sergey Petrov wrote:
>>> I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
>>> as a workload client. Knot server has 32 cores. Interrupts from 10Gb
>>> network card are spreaded across all 32 cores. Knot configured with
>>> 64 udp-workers. Each knot thread assigned to one core. So there are at
>>> least two knot threads assigned to one core. Then i start dnsperf with
>>> command
>>>
>>> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
>>>
>>> htop on knot server shows 3-4 cores completly unused. Then i restart
>>> dnsperf unused cores are changes.
>>>
>>> That is the reason for unused core?
>>
>> Well, sometimes dnsperf is too slow :-)
>>
>> I recommend to check this:
>> - Make sure dnsperf ("source machine") is not 100 % utilized.
>> - Try to increase number of sockets used by dnsperf, i.e. -c parameter.
>> I would try also values like 500 and 1000 to see if it makes any
>> difference. It might change results significantly because Linux kernel
>> is using hashes over some packet fields and low number of sockets might
>> result in uneven query distribution.
>>
>> Please let us know what are your new results.
>>
>
> The source machne is about 15% utilized.
>
> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
>
> get us some performance penalty (260000 rps VS 310000 rps) and more even
> distribution across all cores with 100% usages of all eight cores on
> last CPU socket. While other CPU socket cores are aproximately 60%
> loaded.
>
> Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
> core load distribution and even more performance penalty.
>
> Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
> utilized cores and 2 50% utilized cores with about 300000 rps
>
> The question is that prevents knot thread on 0% used core to serve a
> packet arrived with IRQ bounded to another core? May be you have some
> developer guide can answer this question?
>
4 Dub
4 Dub
11:19
Lets start step-by-step
On Wed, 2019-04-03 at 16:56 +0200, Petr Špaček wrote:
Hello,
as you already found out it is complicated ;-)
Linux kernel has its own magic algorithms to schedule work on
multi-core/multi-socket/NUMA machines and it DNS benchmarking also very
much depends on network card, its drivers etc.
If we were going to fine-tune your setup we would have to go into details:
What is your CPU architecture? Number of sockets, CPU in them etc.?
It's russian Elbrus CPU, i have a little info about it architecture. 4
socket motherboard with 8-core CPU at each socket.
elbrus01 ~/src/dnsperf # uname -a
Linux elbrus01 4.9.0-2.2-e8c #1 SMP Mon Nov 12 10:52:48 GMT 2018 e2k E8C
E8C-SWTX GNU/Linux
elbrus01 ~/src/dnsperf # cat /etc/mcst_version
4.0-rc2
elbrus01 ~/src/dnsperf #
How is operating memory connected to CPUs?
Is it NUMA?
I think it is NUMA. I can see some memory scew across numa nodes
elbrus01 ~/src/dnsperf # numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 32 33 34 35 36 37
38 39 48 49 50 51 52 53 54 55
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3
elbrus01 ~/src/dnsperf # numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64433 MB
node 0 free: 63261 MB
node 1 cpus: 16 17 18 19 20 21 22 23
node 1 size: 64467 MB
node 1 free: 62363 MB
node 2 cpus: 32 33 34 35 36 37 38 39
node 2 size: 64467 MB
node 2 free: 63768 MB
node 3 cpus: 48 49 50 51 52 53 54 55
node 3 size: 64467 MB
node 3 free: 63811 MB
node distances:
node 0 1 2 3
0: 10 20 20 20
1: 20 10 20 20
2: 20 20 10 20
3: 20 20 20 10
elbrus01 ~/src/dnsperf #
Do you have irqbalance enabled?
I try to use irqbalance on previous OperatinSystem based on 3.11 Linux
kernel. New one OS based on 4.9 Linux kernel has no irqbalance binary at
all.
Have you somehow configured IRQ affinity?
What is your network card (how many IO queues it has)?
Network card is Intel 540 10Gbe PCI card. PIC lines are connected to
CPU0 socket. NSD prefer all the IRQ bounded to CPU0, memory allocated at
numa-node0 AND workers bound to CPU0/CPU1. KNOT prefer IRQ spreaded
across all 4 CPUs and workers using all CPUs.
Did you configure network card queues and other driver
settings explicitly?
etc.
I spread incoming UDP across all 32 RX-queues
elbrus01 ~/src/dnsperf # ethtool -N eth4 rx-flow-hash udp4 sdfn
And affine each net4 IRQ to its personal CPU core
elbrus01 ~/src/dnsperf # echo 00800000,00000000
/proc/irq/120/smp_affinity
elbrus01
~/src/dnsperf # echo 00400000,00000000
/proc/irq/119/smp_affinity
elbrus01
~/src/dnsperf # echo 00200000,00000000
/proc/irq/118/smp_affinity
elbrus01
~/src/dnsperf # echo 00100000,00000000
/proc/irq/117/smp_affinity
elbrus01
~/src/dnsperf # echo 00080000,00000000
/proc/irq/116/smp_affinity
elbrus01
~/src/dnsperf # echo 00040000,00000000
/proc/irq/115/smp_affinity
elbrus01
~/src/dnsperf # echo 00020000,00000000
/proc/irq/114/smp_affinity
elbrus01
~/src/dnsperf # echo 00010000,00000000
/proc/irq/113/smp_affinity
elbrus01
~/src/dnsperf # echo 00000080,00000000
/proc/irq/112/smp_affinity
elbrus01
~/src/dnsperf # echo 00000040,00000000
/proc/irq/111/smp_affinity
elbrus01
~/src/dnsperf # echo 00000020,00000000
/proc/irq/110/smp_affinity
elbrus01
~/src/dnsperf # echo 00000010,00000000
/proc/irq/109/smp_affinity
elbrus01
~/src/dnsperf # echo 00000008,00000000
/proc/irq/108/smp_affinity
elbrus01
~/src/dnsperf # echo 00000004,00000000
/proc/irq/107/smp_affinity
elbrus01
~/src/dnsperf # echo 00000002,00000000
/proc/irq/106/smp_affinity
elbrus01
~/src/dnsperf # echo 00000001,00000000
/proc/irq/105/smp_affinity
elbrus01
~/src/dnsperf # echo 00800000 > /proc/irq/104/smp_affinity
elbrus01 ~/src/dnsperf # echo 00400000 > /proc/irq/103/smp_affinity
elbrus01 ~/src/dnsperf # echo 00200000 > /proc/irq/102/smp_affinity
elbrus01 ~/src/dnsperf # echo 00100000 > /proc/irq/101/smp_affinity
elbrus01 ~/src/dnsperf # echo 00080000 > /proc/irq/100/smp_affinity
elbrus01 ~/src/dnsperf # echo 00040000 > /proc/irq/99/smp_affinity
elbrus01 ~/src/dnsperf # echo 00020000 > /proc/irq/98/smp_affinity
elbrus01 ~/src/dnsperf # echo 00010000 > /proc/irq/97/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000080 > /proc/irq/96/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000040 > /proc/irq/95/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000020 > /proc/irq/94/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000010 > /proc/irq/93/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000008 > /proc/irq/92/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000004 > /proc/irq/91/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000002 > /proc/irq/90/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000001 > /proc/irq/89/smp_affinity
A /proc/interrupt output is attached
The load are starts with
[nikor@kaa5 dnsperf]$ ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T72 -t
500 -S 1 -q 1000 -D
Different runs shows random numer of unused cores from 4 to 1. The
request-per-seconds changes accordingly. Less unused cores get more
performance.
Fine tunning always has to take into account your specific environment
and it is hard to provide general advice.
If you find specific reproducible problem please report it to our Gitlab:
https://gitlab.labs.nic.cz/knot/knot-dns/issues/
Please understand that amount of time and hardware we can allocate for
free support is limited. In case you require fine-tunning for your
specific deployment please consider byuing professional support:
https://www.knot-dns.cz/support/
It is reproducible case but in very specific environment. The overall
result may be good enought for production usage. At this case
professional support will be good option.
Myself interested in such strange behavior. Hope this case may be
usefull for you to. I will resend this thread to OS developers. May be
they can clear this issue.
Thank you for you attention.
Thank you for understanding.
Petr Špaček @ CZ.NIC
On 03. 04. 19 16:37, Sergey Petrov wrote:
I reverse the client and the server. So the
server now is 36-cores intel
box (72 HT-core)
Starting with small loads i see knot use lower cores except core-0.
When adding more load i see cores 0-17 AND 37-54 are used but not to
100% level. At maximum load i see all cores are about 100% used.
It seems to me as system scheduler feature. First it starts with lower
number cores, then add cores from second CPU socket and after all
HT-cores.
On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
> On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
>> On 03. 04. 19 10:45, Sergey Petrov wrote:
>>> I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
>>> as a workload client. Knot server has 32 cores. Interrupts from 10Gb
>>> network card are spreaded across all 32 cores. Knot configured with
>>> 64 udp-workers. Each knot thread assigned to one core. So there are at
>>> least two knot threads assigned to one core. Then i start dnsperf with
>>> command
>>>
>>> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
>>>
>>> htop on knot server shows 3-4 cores completly unused. Then i restart
>>> dnsperf unused cores are changes.
>>>
>>> That is the reason for unused core?
>>
>> Well, sometimes dnsperf is too slow :-)
>>
>> I recommend to check this:
>> - Make sure dnsperf ("source machine") is not 100 % utilized.
>> - Try to increase number of sockets used by dnsperf, i.e. -c parameter.
>> I would try also values like 500 and 1000 to see if it makes any
>> difference. It might change results significantly because Linux kernel
>> is using hashes over some packet fields and low number of sockets might
>> result in uneven query distribution.
>>
>> Please let us know what are your new results.
>>
>
> The source machne is about 15% utilized.
>
> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
>
> get us some performance penalty (260000 rps VS 310000 rps) and more even
> distribution across all cores with 100% usages of all eight cores on
> last CPU socket. While other CPU socket cores are aproximately 60%
> loaded.
>
> Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
> core load distribution and even more performance penalty.
>
> Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
> utilized cores and 2 50% utilized cores with about 300000 rps
>
> The question is that prevents knot thread on 0% used core to serve a
> packet arrived with IRQ bounded to another core? May be you have some
> developer guide can answer this question?
>
15:22
One more interesting point.
It seems a code at start of udp_master() function
knot-dns/src/knot/server/udp-handler.c
unsigned cpu = dt_online_cpus();
if (cpu > 1) {
unsigned cpu_mask = (dt_get_id(thread) % cpu);
dt_setaffinity(thread, &cpu_mask, 1);
}
is not effective at all for Elbrus architecture.
Without thread-to-cpu bound by shell "taskset" command i see no empty
cores.
On Thu, 2019-04-04 at 12:19 +0300, Sergey Petrov wrote:
Lets start step-by-step
On Wed, 2019-04-03 at 16:56 +0200, Petr Špaček wrote:
Hello,
as you already found out it is complicated ;-)
Linux kernel has its own magic algorithms to schedule work on
multi-core/multi-socket/NUMA machines and it DNS benchmarking also very
much depends on network card, its drivers etc.
If we were going to fine-tune your setup we would have to go into details:
What is your CPU architecture? Number of sockets, CPU in them etc.?
It's russian Elbrus CPU, i have a little info about it architecture. 4
socket motherboard with 8-core CPU at each socket.
elbrus01 ~/src/dnsperf # uname -a
Linux elbrus01 4.9.0-2.2-e8c #1 SMP Mon Nov 12 10:52:48 GMT 2018 e2k E8C
E8C-SWTX GNU/Linux
elbrus01 ~/src/dnsperf # cat /etc/mcst_version
4.0-rc2
elbrus01 ~/src/dnsperf #
How is operating memory connected to CPUs?
Is it NUMA?
I think it is NUMA. I can see some memory scew across numa nodes
elbrus01 ~/src/dnsperf # numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 32 33 34 35 36 37
38 39 48 49 50 51 52 53 54 55
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3
elbrus01 ~/src/dnsperf # numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64433 MB
node 0 free: 63261 MB
node 1 cpus: 16 17 18 19 20 21 22 23
node 1 size: 64467 MB
node 1 free: 62363 MB
node 2 cpus: 32 33 34 35 36 37 38 39
node 2 size: 64467 MB
node 2 free: 63768 MB
node 3 cpus: 48 49 50 51 52 53 54 55
node 3 size: 64467 MB
node 3 free: 63811 MB
node distances:
node 0 1 2 3
0: 10 20 20 20
1: 20 10 20 20
2: 20 20 10 20
3: 20 20 20 10
elbrus01 ~/src/dnsperf #
Do you have irqbalance enabled?
I try to use irqbalance on previous OperatinSystem based on 3.11 Linux
kernel. New one OS based on 4.9 Linux kernel has no irqbalance binary at
all.
Have you somehow configured IRQ affinity?
What is your network card (how many IO queues it has)?
Network card is Intel 540 10Gbe PCI card. PIC lines are connected to
CPU0 socket. NSD prefer all the IRQ bounded to CPU0, memory allocated at
numa-node0 AND workers bound to CPU0/CPU1. KNOT prefer IRQ spreaded
across all 4 CPUs and workers using all CPUs.
Did you configure network card queues and other
driver settings explicitly?
etc.
I spread incoming UDP across all 32 RX-queues
elbrus01 ~/src/dnsperf # ethtool -N eth4 rx-flow-hash udp4 sdfn
And affine each net4 IRQ to its personal CPU core
elbrus01 ~/src/dnsperf # echo 00800000,00000000
/proc/irq/120/smp_affinity
elbrus01
~/src/dnsperf # echo 00400000,00000000
/proc/irq/119/smp_affinity
elbrus01
~/src/dnsperf # echo 00200000,00000000
/proc/irq/118/smp_affinity
elbrus01
~/src/dnsperf # echo 00100000,00000000
/proc/irq/117/smp_affinity
elbrus01
~/src/dnsperf # echo 00080000,00000000
/proc/irq/116/smp_affinity
elbrus01
~/src/dnsperf # echo 00040000,00000000
/proc/irq/115/smp_affinity
elbrus01
~/src/dnsperf # echo 00020000,00000000
/proc/irq/114/smp_affinity
elbrus01
~/src/dnsperf # echo 00010000,00000000
/proc/irq/113/smp_affinity
elbrus01
~/src/dnsperf # echo 00000080,00000000
/proc/irq/112/smp_affinity
elbrus01
~/src/dnsperf # echo 00000040,00000000
/proc/irq/111/smp_affinity
elbrus01
~/src/dnsperf # echo 00000020,00000000
/proc/irq/110/smp_affinity
elbrus01
~/src/dnsperf # echo 00000010,00000000
/proc/irq/109/smp_affinity
elbrus01
~/src/dnsperf # echo 00000008,00000000
/proc/irq/108/smp_affinity
elbrus01
~/src/dnsperf # echo 00000004,00000000
/proc/irq/107/smp_affinity
elbrus01
~/src/dnsperf # echo 00000002,00000000
/proc/irq/106/smp_affinity
elbrus01
~/src/dnsperf # echo 00000001,00000000
/proc/irq/105/smp_affinity
elbrus01
~/src/dnsperf # echo 00800000 > /proc/irq/104/smp_affinity
elbrus01 ~/src/dnsperf # echo 00400000 > /proc/irq/103/smp_affinity
elbrus01 ~/src/dnsperf # echo 00200000 > /proc/irq/102/smp_affinity
elbrus01 ~/src/dnsperf # echo 00100000 > /proc/irq/101/smp_affinity
elbrus01 ~/src/dnsperf # echo 00080000 > /proc/irq/100/smp_affinity
elbrus01 ~/src/dnsperf # echo 00040000 > /proc/irq/99/smp_affinity
elbrus01 ~/src/dnsperf # echo 00020000 > /proc/irq/98/smp_affinity
elbrus01 ~/src/dnsperf # echo 00010000 > /proc/irq/97/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000080 > /proc/irq/96/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000040 > /proc/irq/95/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000020 > /proc/irq/94/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000010 > /proc/irq/93/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000008 > /proc/irq/92/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000004 > /proc/irq/91/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000002 > /proc/irq/90/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000001 > /proc/irq/89/smp_affinity
A /proc/interrupt output is attached
The load are starts with
[nikor@kaa5 dnsperf]$ ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T72 -t
500 -S 1 -q 1000 -D
Different runs shows random numer of unused cores from 4 to 1. The
request-per-seconds changes accordingly. Less unused cores get more
performance.
Fine tunning always has to take into account your specific environment
and it is hard to provide general advice.
If you find specific reproducible problem please report it to our Gitlab:
https://gitlab.labs.nic.cz/knot/knot-dns/issues/
Please understand that amount of time and hardware we can allocate for
free support is limited. In case you require fine-tunning for your
specific deployment please consider byuing professional support:
https://www.knot-dns.cz/support/
It is reproducible case but in very specific environment. The overall
result may be good enought for production usage. At this case
professional support will be good option.
Myself interested in such strange behavior. Hope this case may be
usefull for you to. I will resend this thread to OS developers. May be
they can clear this issue.
Thank you for you attention.
Thank you for understanding.
Petr Špaček @ CZ.NIC
On 03. 04. 19 16:37, Sergey Petrov wrote:
I reverse the client and the server. So the
server now is 36-cores intel
box (72 HT-core)
Starting with small loads i see knot use lower cores except core-0.
When adding more load i see cores 0-17 AND 37-54 are used but not to
100% level. At maximum load i see all cores are about 100% used.
It seems to me as system scheduler feature. First it starts with lower
number cores, then add cores from second CPU socket and after all
HT-cores.
On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
> On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
>> On 03. 04. 19 10:45, Sergey Petrov wrote:
>>> I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
>>> as a workload client. Knot server has 32 cores. Interrupts from 10Gb
>>> network card are spreaded across all 32 cores. Knot configured with
>>> 64 udp-workers. Each knot thread assigned to one core. So there are at
>>> least two knot threads assigned to one core. Then i start dnsperf with
>>> command
>>>
>>> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
>>>
>>> htop on knot server shows 3-4 cores completly unused. Then i restart
>>> dnsperf unused cores are changes.
>>>
>>> That is the reason for unused core?
>>
>> Well, sometimes dnsperf is too slow :-)
>>
>> I recommend to check this:
>> - Make sure dnsperf ("source machine") is not 100 % utilized.
>> - Try to increase number of sockets used by dnsperf, i.e. -c parameter.
>> I would try also values like 500 and 1000 to see if it makes any
>> difference. It might change results significantly because Linux kernel
>> is using hashes over some packet fields and low number of sockets might
>> result in uneven query distribution.
>>
>> Please let us know what are your new results.
>>
>
> The source machne is about 15% utilized.
>
> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
>
> get us some performance penalty (260000 rps VS 310000 rps) and more even
> distribution across all cores with 100% usages of all eight cores on
> last CPU socket. While other CPU socket cores are aproximately 60%
> loaded.
>
> Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
> core load distribution and even more performance penalty.
>
> Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
> utilized cores and 2 50% utilized cores with about 300000 rps
>
> The question is that prevents knot thread on 0% used core to serve a
> packet arrived with IRQ bounded to another core? May be you have some
> developer guide can answer this question?
>
15:36
Hi,
it's interesting! Unfortunately, we cannot debug this architecture :-(
Do you think you are able to create a reasonable patch?
Daniel
On 4/4/19 3:22 PM, Sergey Petrov wrote:
One more interesting point.
It seems a code at start of udp_master() function
knot-dns/src/knot/server/udp-handler.c
unsigned cpu = dt_online_cpus();
if (cpu > 1) {
unsigned cpu_mask = (dt_get_id(thread) % cpu);
dt_setaffinity(thread, &cpu_mask, 1);
}
is not effective at all for Elbrus architecture.
Without thread-to-cpu bound by shell "taskset" command i see no empty
cores.
On Thu, 2019-04-04 at 12:19 +0300, Sergey Petrov wrote:
Lets start step-by-step
On Wed, 2019-04-03 at 16:56 +0200, Petr Špaček wrote:
Hello,
as you already found out it is complicated ;-)
Linux kernel has its own magic algorithms to schedule work on
multi-core/multi-socket/NUMA machines and it DNS benchmarking also very
much depends on network card, its drivers etc.
If we were going to fine-tune your setup we would have to go into details:
What is your CPU architecture? Number of sockets, CPU in them etc.?
It's russian Elbrus CPU, i have a little info about it architecture. 4
socket motherboard with 8-core CPU at each socket.
elbrus01 ~/src/dnsperf # uname -a
Linux elbrus01 4.9.0-2.2-e8c #1 SMP Mon Nov 12 10:52:48 GMT 2018 e2k E8C
E8C-SWTX GNU/Linux
elbrus01 ~/src/dnsperf # cat /etc/mcst_version
4.0-rc2
elbrus01 ~/src/dnsperf #
How is operating memory connected to CPUs?
Is it NUMA?
I think it is NUMA. I can see some memory scew across numa nodes
elbrus01 ~/src/dnsperf # numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 32 33 34 35 36 37
38 39 48 49 50 51 52 53 54 55
cpubind: 0 1 2 3
nodebind: 0 1 2 3
membind: 0 1 2 3
elbrus01 ~/src/dnsperf # numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64433 MB
node 0 free: 63261 MB
node 1 cpus: 16 17 18 19 20 21 22 23
node 1 size: 64467 MB
node 1 free: 62363 MB
node 2 cpus: 32 33 34 35 36 37 38 39
node 2 size: 64467 MB
node 2 free: 63768 MB
node 3 cpus: 48 49 50 51 52 53 54 55
node 3 size: 64467 MB
node 3 free: 63811 MB
node distances:
node 0 1 2 3
0: 10 20 20 20
1: 20 10 20 20
2: 20 20 10 20
3: 20 20 20 10
elbrus01 ~/src/dnsperf #
Do you have irqbalance enabled?
I try to use irqbalance on previous OperatinSystem based on 3.11 Linux
kernel. New one OS based on 4.9 Linux kernel has no irqbalance binary at
all.
Have you somehow configured IRQ affinity?
What is your network card (how many IO queues it has)?
Network card is Intel 540 10Gbe PCI card. PIC lines are connected to
CPU0 socket. NSD prefer all the IRQ bounded to CPU0, memory allocated at
numa-node0 AND workers bound to CPU0/CPU1. KNOT prefer IRQ spreaded
across all 4 CPUs and workers using all CPUs.
Did you configure network card queues and other
driver settings explicitly?
etc.
I spread incoming UDP across all 32 RX-queues
elbrus01 ~/src/dnsperf # ethtool -N eth4 rx-flow-hash udp4 sdfn
And affine each net4 IRQ to its personal CPU core
elbrus01 ~/src/dnsperf # echo 00800000,00000000
/proc/irq/120/smp_affinity
elbrus01
~/src/dnsperf # echo 00400000,00000000
/proc/irq/119/smp_affinity
elbrus01
~/src/dnsperf # echo 00200000,00000000
/proc/irq/118/smp_affinity
elbrus01
~/src/dnsperf # echo 00100000,00000000
/proc/irq/117/smp_affinity
elbrus01
~/src/dnsperf # echo 00080000,00000000
/proc/irq/116/smp_affinity
elbrus01
~/src/dnsperf # echo 00040000,00000000
/proc/irq/115/smp_affinity
elbrus01
~/src/dnsperf # echo 00020000,00000000
/proc/irq/114/smp_affinity
elbrus01
~/src/dnsperf # echo 00010000,00000000
/proc/irq/113/smp_affinity
elbrus01
~/src/dnsperf # echo 00000080,00000000
/proc/irq/112/smp_affinity
elbrus01
~/src/dnsperf # echo 00000040,00000000
/proc/irq/111/smp_affinity
elbrus01
~/src/dnsperf # echo 00000020,00000000
/proc/irq/110/smp_affinity
elbrus01
~/src/dnsperf # echo 00000010,00000000
/proc/irq/109/smp_affinity
elbrus01
~/src/dnsperf # echo 00000008,00000000
/proc/irq/108/smp_affinity
elbrus01
~/src/dnsperf # echo 00000004,00000000
/proc/irq/107/smp_affinity
elbrus01
~/src/dnsperf # echo 00000002,00000000
/proc/irq/106/smp_affinity
elbrus01
~/src/dnsperf # echo 00000001,00000000
/proc/irq/105/smp_affinity
elbrus01
~/src/dnsperf # echo 00800000 > /proc/irq/104/smp_affinity
elbrus01 ~/src/dnsperf # echo 00400000 > /proc/irq/103/smp_affinity
elbrus01 ~/src/dnsperf # echo 00200000 > /proc/irq/102/smp_affinity
elbrus01 ~/src/dnsperf # echo 00100000 > /proc/irq/101/smp_affinity
elbrus01 ~/src/dnsperf # echo 00080000 > /proc/irq/100/smp_affinity
elbrus01 ~/src/dnsperf # echo 00040000 > /proc/irq/99/smp_affinity
elbrus01 ~/src/dnsperf # echo 00020000 > /proc/irq/98/smp_affinity
elbrus01 ~/src/dnsperf # echo 00010000 > /proc/irq/97/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000080 > /proc/irq/96/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000040 > /proc/irq/95/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000020 > /proc/irq/94/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000010 > /proc/irq/93/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000008 > /proc/irq/92/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000004 > /proc/irq/91/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000002 > /proc/irq/90/smp_affinity
elbrus01 ~/src/dnsperf # echo 00000001 > /proc/irq/89/smp_affinity
A /proc/interrupt output is attached
The load are starts with
[nikor@kaa5 dnsperf]$ ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T72 -t
500 -S 1 -q 1000 -D
Different runs shows random numer of unused cores from 4 to 1. The
request-per-seconds changes accordingly. Less unused cores get more
performance.
Fine tunning always has to take into account your specific environment
and it is hard to provide general advice.
If you find specific reproducible problem please report it to our Gitlab:
https://gitlab.labs.nic.cz/knot/knot-dns/issues/
Please understand that amount of time and hardware we can allocate for
free support is limited. In case you require fine-tunning for your
specific deployment please consider byuing professional support:
https://www.knot-dns.cz/support/
It is reproducible case but in very specific environment. The overall
result may be good enought for production usage. At this case
professional support will be good option.
Myself interested in such strange behavior. Hope this case may be
usefull for you to. I will resend this thread to OS developers. May be
they can clear this issue.
Thank you for you attention.
Thank you for understanding.
Petr Špaček @ CZ.NIC
On 03. 04. 19 16:37, Sergey Petrov wrote:
I reverse the client and the server. So the
server now is 36-cores intel
box (72 HT-core)
Starting with small loads i see knot use lower cores except core-0.
When adding more load i see cores 0-17 AND 37-54 are used but not to
100% level. At maximum load i see all cores are about 100% used.
It seems to me as system scheduler feature. First it starts with lower
number cores, then add cores from second CPU socket and after all
HT-cores.
On Wed, 2019-04-03 at 12:53 +0300, Sergey Petrov wrote:
> On Wed, 2019-04-03 at 10:52 +0200, Petr Špaček wrote:
>> On 03. 04. 19 10:45, Sergey Petrov wrote:
>>> I perfoms benchmarks with knot-dns as a authoritative server and dnsperf
>>> as a workload client. Knot server has 32 cores. Interrupts from 10Gb
>>> network card are spreaded across all 32 cores. Knot configured with
>>> 64 udp-workers. Each knot thread assigned to one core. So there are at
>>> least two knot threads assigned to one core. Then i start dnsperf with
>>> command
>>>
>>> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
>>>
>>> htop on knot server shows 3-4 cores completly unused. Then i restart
>>> dnsperf unused cores are changes.
>>>
>>> That is the reason for unused core?
>>
>> Well, sometimes dnsperf is too slow :-)
>>
>> I recommend to check this:
>> - Make sure dnsperf ("source machine") is not 100 % utilized.
>> - Try to increase number of sockets used by dnsperf, i.e. -c parameter.
>> I would try also values like 500 and 1000 to see if it makes any
>> difference. It might change results significantly because Linux kernel
>> is using hashes over some packet fields and low number of sockets might
>> result in uneven query distribution.
>>
>> Please let us know what are your new results.
>>
>
> The source machne is about 15% utilized.
>
> ./dnsperf -s 10.0.0.4 -d out -n 20 -c 512 -T 512 -t 500 -S 1 -q 1000 -D
>
> get us some performance penalty (260000 rps VS 310000 rps) and more even
> distribution across all cores with 100% usages of all eight cores on
> last CPU socket. While other CPU socket cores are aproximately 60%
> loaded.
>
> Using "-c 1000 -T 1000" parameters of dnsperf i see practicaly the same
> core load distribution and even more performance penalty.
>
> Using "-c 16 -T 16" parameters i see 14 0% utilized cores, 16 100%
> utilized cores and 2 50% utilized cores with about 300000 rps
>
> The question is that prevents knot thread on 0% used core to serve a
> packet arrived with IRQ bounded to another core? May be you have some
> developer guide can answer this question?
>
3 Dub
3 Dub
11:05
Hi,
I have no experience with dnsperf, but based on my experience, it's difficult
to saturate a server with one client. We must bypass the network stack using netmap
during our benchmarking!
Btw, to get maximum performance, the number of udp workers should equal the
number of available CPU cores (including HT if enabled). Also if well configured,
all CPU cores should be loaded equally.
Best,
Daniel
On 4/3/19 10:45 AM, Sergey Petrov wrote:
I perfoms benchmarks with knot-dns as a authoritative
server and dnsperf
as a workload client. Knot server has 32 cores. Interrupts from 10Gb
network card are spreaded across all 32 cores. Knot configured with
64 udp-workers. Each knot thread assigned to one core. So there are at
least two knot threads assigned to one core. Then i start dnsperf with
command
./dnsperf -s 10.0.0.4 -d out -n 20 -c 103 -T 64 -t 500 -S 1 -q 1000 -D
htop on knot server shows 3-4 cores completly unused. Then i restart
dnsperf unused cores are changes.
That is the reason for unused core?
2058
days inactive
2059
days old
10 comments
3 participants
participants (3)
-
Daniel Salzman -
Petr Špaček -
Sergey Petrov