Hi Danel,
I'm sorry for the late reply; I've been on holiday/vacation and playing catch-up! ;)

ethtool:
Channel parameters for eno1:
Pre-set maximums:
RX: 4
TX: 4
Other: 0
Combined: 0
Current hardware settings:
RX: 4
TX: 1
Other: 0
Combined: 0

knotc output:

  Knot DNS 2.9.5

    Target:   linux-gnu x86_64
    Compiler: gcc
    CFLAGS:   -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -DNDEBUG -Wno-unused -Wall -Wshadow -Werror=format-security -Werror=implicit -Wstrict-prototypes
    LIBS:      -Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
    LibURCU:  -lurcu  
    GnuTLS:   -lgnutls  -I/usr/include/p11-kit-1
    Libedit:  -ledit  -I/usr/include/editline
    LMDB:     shared  -llmdb
    Config:   500 MiB default mapsize

    Prefix:      /usr
    Run dir:     /run/knot
    Storage dir: /var/lib/knot
    Config dir:  /etc/knot
    Module dir:  

    Static modules: cookies dnsproxy dnstap geoip noudp onlinesign queryacl rrl stats synthrecord whoami
    Shared modules:

    Knot DNS libraries:     yes
    Knot DNS daemon:        yes
    Knot DNS utilities:     yes
    Knot DNS documentation: yes

    Use recvmmsg:           yes
    Use SO_REUSEPORT(_LB):  yes
    Memory allocator:       auto
    Fast zone parser:       yes
    Utilities with IDN:     libidn2
    Utilities with Dnstap:  yes
    MaxMind DB support:     yes
    Systemd integration:    yes
    POSIX capabilities:     yes
    PKCS #11 support:       yes
    Ed25519 support:        yes
    Ed448 support:          no
    Code coverage:          no
    Sanitizer:              no
    LibFuzzer:              no
    OSS-Fuzz:               no

Continued thanks! ;)

Jonathan

On Thu, 24 Sep 2020 at 11:14, Daniel Salzman <daniel.salzman@nic.cz> wrote:
Hi Jonathan,

If you have just one 1GbE port, I would recommend reducing the number of udp-workers. What about 12 or less?

How do you install Knot DNS (repository or source code)?

What do say these commands?
# sudo ethtool -l eno1
# knotc status configure

Daniel

On 9/23/20 5:45 PM, Jonathan Hewlett wrote:
> Hi Daniel
>
> * Server listens: 967 (multiple v4 and v6 networks)
> * Physical server
> * NIC details
>   *-network:0
>        description: Ethernet interface
>        product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
>        vendor: Broadcom Inc. and subsidiaries
>        physical id: 0
>        bus info: pci@0000:02:00.0
>        logical name: eno1
>        version: 00
>        serial: b0:83:fe:c8:f3:ec
>        size: 1Gbit/s
>        capacity: 1Gbit/s
>        width: 64 bits
>        clock: 33MHz
>        capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
>        configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=3.137 duplex=full firmware=FFV21.60.2 bc 5720-v1.39 ip=38.88.50.50 latency=0 link=yes multicast=yes port=twisted pair
> speed=1Gbit/s
>        resources: irq:16 memory:d80a0000-d80affff memory:d80b0000-d80bffff memory:d80c0000-d80cffff memory:dd800000-dd83ffff
>
> Thanks for the continuing support
> Jonathan
>
>
> On Wed, 23 Sep 2020 at 12:47, Daniel Salzman <daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>> wrote:
>
>     Jonathan,
>
>     there must be something wrong with the networking. How many `server.listen` configuration values
>     do yo have? Is it a physical server or a virtual one? What is the network card used?
>
>     Daneil
>
>     On 9/23/20 1:18 PM, Jonathan Hewlett wrote:
>     > Daniel
>     > Thank you for your time
>     >
>     > # knotc status workers
>     > UDP workers: 24, TCP workers 24, background workers: 10 (running: 0, pending: 0)
>     >
>     > image.png
>     >
>     > Kind regards
>     > Jonathan
>     >
>     >
>     > On Wed, 23 Sep 2020 at 07:59, <daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz> <mailto:daniel.salzman@nic.cz <mailto:daniel.salzman@nic.cz>>> wrote:
>     >
>     >     Hi Jonathan,
>     >
>     >     On 2020-09-22 23:11, Jonathan Hewlett wrote:
>     >     > Hi,
>     >     >
>     >     > We run around 50 knot servers around the world, with great success -
>     >     > thanks for the software!
>     >
>     >     Thanks for this feedback :-)
>     >
>     >     >
>     >     > We are just a little curious about seemingly high CPU usage, and how
>     >     > best to interpret the results, if someone could advise?  Specifically,
>     >     > we are seeing the majority of the cpu time in 'system' and not 'user',
>     >     > what would that suggest is happening perhaps?
>     >
>     >     Could you provide me the output from `knotc status workers`?
>     >
>     >     I like htop. Are you able to send me a screenshot when running htop? :-)
>     >     [Don't forget to configure:
>     >       F2
>     >       Display options->Detailed CPU time
>     >     (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
>     >       F10
>     >     ]
>     >
>     >     Best,
>     >     Daniel
>     >
>     >     >
>     >     > Any tuning help gratefully received!
>     >     >
>     >     > Our platform (sample)
>     >     >
>     >     > 24 Core CPU E5-2440 0 @ 2.40
>     >     > 64Gb RAM
>     >     > SSD
>     >     >
>     >     > knot-2.9.5, running on Centos 8.2
>     >     >
>     >     > Currently receiving around 4k pps
>     >     >
>     >     > Server is authoritative for a 278 zones, the largest zone file being
>     >     > around 600Mb
>     >     >
>     >     > Output of top
>     >     >
>     >     > top - 20:50:38 up 6 days,  4:49,  2 users,  load average: 13.47,
>     >     > 12.34, 11.83
>     >     > Tasks: 321 total,   2 running, 318 sleeping,   0 stopped,   1 zombie
>     >     > %Cpu0  :  4.0 us, 34.1 sy,  0.0 ni, 60.3 id,  0.3 wa,  0.7 hi,  0.7
>     >     > si,  0.0 st
>     >     > %Cpu1  :  4.3 us, 19.4 sy,  0.0 ni, 75.3 id,  0.0 wa,  0.3 hi,  0.7
>     >     > si,  0.0 st
>     >     > %Cpu2  :  4.3 us, 33.8 sy,  0.0 ni, 60.9 id,  0.0 wa,  0.7 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu3  :  5.0 us, 21.2 sy,  0.0 ni, 73.2 id,  0.0 wa,  0.3 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu4  :  3.3 us, 34.9 sy,  0.0 ni, 61.5 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu5  :  3.0 us, 20.2 sy,  0.0 ni, 76.2 id,  0.0 wa,  0.3 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu6  :  4.0 us, 35.2 sy,  0.0 ni, 60.4 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu7  :  5.0 us, 18.7 sy,  0.0 ni, 76.0 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu8  :  2.0 us, 35.0 sy,  0.0 ni, 62.3 id,  0.0 wa,  0.7 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu9  :  2.0 us, 20.0 sy,  0.0 ni, 77.7 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu10 :  3.7 us, 34.4 sy,  0.0 ni, 61.2 id,  0.0 wa,  0.7 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu11 :  3.3 us, 20.7 sy,  0.0 ni, 75.7 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu12 :  2.3 us, 34.6 sy,  0.0 ni, 62.1 id,  0.0 wa,  0.7 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu13 :  3.3 us, 20.7 sy,  0.0 ni, 75.7 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu14 :  2.3 us, 33.6 sy,  0.0 ni, 59.7 id,  0.0 wa,  0.7 hi,  3.7
>     >     > si,  0.0 st
>     >     > %Cpu15 :  3.3 us, 19.7 sy,  0.0 ni, 76.3 id,  0.0 wa,  0.3 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu16 :  2.3 us, 34.0 sy,  0.0 ni, 58.7 id,  0.0 wa,  1.0 hi,  4.0
>     >     > si,  0.0 st
>     >     > %Cpu17 :  2.3 us, 21.5 sy,  0.0 ni, 75.8 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > %Cpu18 :  3.3 us, 34.8 sy,  0.0 ni, 57.2 id,  0.0 wa,  0.7 hi,  4.0
>     >     > si,  0.0 st
>     >     > %Cpu19 :  5.0 us, 19.3 sy,  0.0 ni, 75.1 id,  0.0 wa,  0.3 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu20 :  2.7 us, 35.3 sy,  0.0 ni, 60.0 id,  0.0 wa,  0.7 hi,  1.3
>     >     > si,  0.0 st
>     >     > %Cpu21 :  1.7 us, 20.5 sy,  0.0 ni, 77.2 id,  0.0 wa,  0.3 hi,  0.3
>     >     > si,  0.0 st
>     >     > %Cpu22 :  4.7 us, 32.4 sy,  0.0 ni, 58.5 id,  0.0 wa,  0.7 hi,  3.7
>     >     > si,  0.0 st
>     >     > %Cpu23 :  2.3 us, 19.0 sy,  0.0 ni, 78.3 id,  0.0 wa,  0.3 hi,  0.0
>     >     > si,  0.0 st
>     >     > MiB Mem :  64107.4 total,  46764.9 free,  11733.6 used,   5608.8
>     >     > buff/cache
>     >     > MiB Swap:  32248.0 total,  32248.0 free,      0.0 used.  51670.0 avail
>     >     > Mem
>     >     >
>     >     > PID USER      PR   NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>     >     > COMMAND
>     >     > 213370 knot  20   0      34.0g   9.6g  41784 S 644.9  15.3   1319:12
>     >     > /usr/sbin/knotd
>     >     >
>     >     > Thanks again for this great product.
>     >     >
>     >     > Jonathan
>     >
>     >
>     >
>     > --
>     > --\--
>     > Jonathan Hewlett
>     > ^Senior\ Systems\ Engineer\n$
>     > CentralNic Group plc (LSE:CNIC)
>     > https://www.centralnic.com <https://www.centralnic.com/>
>     > Tel: +44.7548243020
>     > CentralNic Group plc is a company registered in England and Wales with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.
>
>
>
> --
> --\--
> Jonathan Hewlett
> ^Senior\ Systems\ Engineer\n$
> CentralNic Group plc (LSE:CNIC)
> https://www.centralnic.com <https://www.centralnic.com/>
> Tel: +44.7548243020
> CentralNic Group plc is a company registered in England and Wales with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.


--
--\--
Jonathan Hewlett
^Senior\ Systems\ Engineer\n$
CentralNic Group plc (LSE:CNIC)
https://www.centralnic.com
Tel: +44.7548243020
CentralNic Group plc is a company registered in England and Wales with company number 8576358. Registered Offices: Saddlers House, Gutter Lane, London EC2V 6AE.