1 Bře
2022
1 Bře
'22
14:49
Hello,
I'd like to kindly ask for help with following issue.
I am considering to deploy knot-resolver to the DNS solution where it
should coexist with other DNS daemons namely Unbound. I am running
dnsdist <https://dnsdist.org/> in front of the pool of resolvers where I
have just added also the latest release (5.5.4) of knot-resolver. It
receives a portion of requests at the rate about 500qps. There are 6
kresd processes on a VM running with cache in tmpfs. Cache size is 2GB
(8GB mounted to tmpfs). Resolver is running Debian Bullseye. The
solution is serving real customers - the traffic is not artificial.
The configuration enables some modules:
modules = {
'hints > iterate', -- Load /etc/hosts and allow custom root hints
'stats', -- Track internal statistics
'predict', -- Prefetch expiring/frequent records
'bogus_log', -- DNSSEC validation failure logging
'nsid',
'prefill',
'rebinding < iterate'
}
cache.size = cache.fssize() - 6*GB
modules = {
predict = {
window = 15, -- 15 minutes sampling window
period = 6*(60/15) -- track last 6 hours
}
}
policy.add(
policy.rpz(policy.DENY,
'/var/cache/unbound/db.rpz.xxx.cz',
true)
)
And the Carbon protocol reporting is enabled with the sample rate of 5s.
The performance in terms of response time is equal or even better when
comparing with Unbound. I am measuring the performance of the dnsdist's
backend servers (daemons) - the latency, dropped requests and the
requests rate using Carbon protocol with sample rate of 5s. The backend
servers kresd included are receiving requests from just several clients
at the moment (source IPs of the balancers).
What is wrong here is some kind of repeated packet drops reported for
kresd instance by dnsdist. It appears in about 15 min. interval. When it
occurs the drop rate increases from about 0.5 req/s (standard behavior)
to 5-10 req/s which causes a positive feedback and increased packet
rate. There are such peaks every 15 min. When I restart the kresd
instances drops go away and everything is stellar for couple of hours.
I have no explanation for that. It is apparently not related to the
requests - I have tried to simulate this by replaying the traffic
captured when the problem occurs several times without success. kresd
can even process 20k qps on this instance with no increase of drop rate
from the balancer's point of view. But after a while... The fact that
restarting kresd helps immediately shows that there might be something
wrong inside.
I have tried to increase the number of kresd processes, cache size and
disabling the cache persistence. Nothing helps here. I am pretty sure
there are no network issues on the VM or surrounding network.
I can provide some graphical representations of this behavior of it is
needed or a packet capture of the real traffic.
Many thanks
With best regards
Ales Rygl