3 Úno
2021
3 Úno
'21
15:29
Hello,
I have just run into a weird problem and since I have failed to look
this up on the Web myself, I wonder if anyone here could give me a
pointer regarding what to do with it.
I run a Linux/arm DNS server based on knot-3.0.4. It used to serve as a
slave in a zone whose master was a knot-2.9, with automatic DNSSEC
signing _on master only_ set up and working. The zone configuration
(minus "master" and "acl" keywords) looked as follows:
zonefile-load: difference-no-serial
journal-content: all
semantic-checks: on
# dnssec-signing: on
# dnssec-policy: nsec3
where "nsec3" is a policy also defined in Knot configuration, identical
to the one used on the master. In this role everything worked fine.
Recently it has been decided to retire the original master server and
replace it with the erstwhile slave. I removed the "master" line from
the zone configuration in knot.conf (yes, I did leave the DNSSEC line
commented out) in addition to setting up replication to a new slave
server, copied the key files + the KASP dump from the old master and
installed them in the right places, then restarted knot. Again, so far
so good.
I then attempted to re-enable DNSSEC by uncommenting those two lines -
and immediately after I'd run 'knotc reload' Knot began spamming the
system log with lines like
info: [example.com.] zone file parsed, serial corrected 2021020301 ->
2021020303
info: [example.com.] DNSSEC, key, tag 4533, algorithm
RSASHA1_NSEC3_SHA1, KSK, public, active
info: [example.com.] DNSSEC, key, tag 10532, algorithm
RSASHA1_NSEC3_SHA1, public, active
info: [example.com.] DNSSEC, signing started
info: [example.com.] DNSSEC, successfully signed
info: [example.com.] loaded, serial 2021020302 -> 2021020303 ->
2021020302, 113233 bytes
info: [example.com.] DNSSEC, next signing at 2021-02-10T13:21:14+0000
info: [example.com.] zone file parsed, serial corrected 2021020301 ->
2021020303
info: [example.com.] DNSSEC, key, tag 4533, algorithm
RSASHA1_NSEC3_SHA1, KSK, public, active
info: [example.com.] DNSSEC, key, tag 10532, algorithm
RSASHA1_NSEC3_SHA1, public, active
info: [example.com.] DNSSEC, signing started
info: [example.com.] DNSSEC, successfully signed
info: [example.com.] loaded, serial 2021020302 -> 2021020303 ->
2021020302, 113233 bytes
info: [example.com.] DNSSEC, next signing at 2021-02-10T13:21:14+0000
info: [example.com.] zone file parsed, serial corrected 2021020301 ->
2021020303
[...and so on...]
This was accompanied by knot consistently requiring much more CPU power
than usual. I also noticed that unlike before, running 'knotc
zone-flush' or shutting Knot down does NOT update the plain-text zone files.
From what I could see by removing old DNSSEC signatures and the NSEC3
chain from the zone file, restarting Knot and then querying the server
with dig, the server does indeed sign the zone. Therefore, my current
working hypothesis is that somehow Knot does not update its internal
zone state following the successful signing, thus continuing to think
the outdated zone file found on the file system is the latest available
version. The problem is, I do not know why it does it or how I could
make it stop.
Any and all suggestions will be very much appreciated!
Cheers,
--
MS
3 Úno
3 Úno
17:55
Hi Marek,
thank you for reporting your troubles with Knot.
This seems like an effect of a bug. I must admit that the scenario of
transforming a slave into signing master has not been tested too much ;)
We would really appreciate if you helped us debug it.
In order to attempt to mitigate the issue, you may try purging zone
timers. Please back up the timers database first (by simply copying the
directory <storage>/timers somewhere), and then either `knotc zone-purge
example.com. +timers` or delete the directory (I assume that you have
just one zone configured at your server?).
Please let us know if that helped. In the meantime, I will try to
reproduce your issue.
Z poważaniem,
Libor
Dne 03. 02. 21 v 15:29 Marek Szuba napsal(a):
Hello,
I have just run into a weird problem and since I have failed to look
this up on the Web myself, I wonder if anyone here could give me a
pointer regarding what to do with it.
I run a Linux/arm DNS server based on knot-3.0.4. It used to serve as
a slave in a zone whose master was a knot-2.9, with automatic DNSSEC
signing _on master only_ set up and working. The zone configuration
(minus "master" and "acl" keywords) looked as follows:
zonefile-load: difference-no-serial
journal-content: all
semantic-checks: on
# dnssec-signing: on
# dnssec-policy: nsec3
where "nsec3" is a policy also defined in Knot configuration,
identical to the one used on the master. In this role everything
worked fine.
Recently it has been decided to retire the original master server and
replace it with the erstwhile slave. I removed the "master" line from
the zone configuration in knot.conf (yes, I did leave the DNSSEC line
commented out) in addition to setting up replication to a new slave
server, copied the key files + the KASP dump from the old master and
installed them in the right places, then restarted knot. Again, so far
so good.
I then attempted to re-enable DNSSEC by uncommenting those two lines -
and immediately after I'd run 'knotc reload' Knot began spamming the
system log with lines like
info: [example.com.] zone file parsed, serial corrected 2021020301 ->
2021020303
info: [example.com.] DNSSEC, key, tag 4533, algorithm
RSASHA1_NSEC3_SHA1, KSK, public, active
info: [example.com.] DNSSEC, key, tag 10532, algorithm
RSASHA1_NSEC3_SHA1, public, active
info: [example.com.] DNSSEC, signing started
info: [example.com.] DNSSEC, successfully signed
info: [example.com.] loaded, serial 2021020302 -> 2021020303 ->
2021020302, 113233 bytes
info: [example.com.] DNSSEC, next signing at 2021-02-10T13:21:14+0000
info: [example.com.] zone file parsed, serial corrected 2021020301 ->
2021020303
info: [example.com.] DNSSEC, key, tag 4533, algorithm
RSASHA1_NSEC3_SHA1, KSK, public, active
info: [example.com.] DNSSEC, key, tag 10532, algorithm
RSASHA1_NSEC3_SHA1, public, active
info: [example.com.] DNSSEC, signing started
info: [example.com.] DNSSEC, successfully signed
info: [example.com.] loaded, serial 2021020302 -> 2021020303 ->
2021020302, 113233 bytes
info: [example.com.] DNSSEC, next signing at 2021-02-10T13:21:14+0000
info: [example.com.] zone file parsed, serial corrected 2021020301 ->
2021020303
[...and so on...]
This was accompanied by knot consistently requiring much more CPU
power than usual. I also noticed that unlike before, running 'knotc
zone-flush' or shutting Knot down does NOT update the plain-text zone
files.
From what I could see by removing old DNSSEC signatures and the NSEC3
chain from the zone file, restarting Knot and then querying the server
with dig, the server does indeed sign the zone. Therefore, my current
working hypothesis is that somehow Knot does not update its internal
zone state following the successful signing, thus continuing to think
the outdated zone file found on the file system is the latest
available version. The problem is, I do not know why it does it or how
I could make it stop.
Any and all suggestions will be very much appreciated!
Cheers,
4 Úno
4 Úno
11:41
On 03/02/2021 16:55, libor.peltan wrote:
In order to attempt to mitigate the issue, you may try
purging zone
timers. Please back up the timers database first (by simply copying the
directory <storage>/timers somewhere), and then either `knotc zone-purge
example.com. +timers` or delete the directory (I assume that you have
just one zone configured at your server?).
There are other zones on that server but they are all from the same set
so I can manipulate the server with no major disruption.
Anyway, here is what I have tried (in this order) since my previous e-mail:
- purging the timers
- downgrading knot to 2.9
- following the downgrade, deleting both the timers and the journal
- commenting out "journal-content: all" (the old master didn't use this)
Unfortunately none of these actions have helped.
Hmm. Could it be that something has gone wrong with the import of the
KASP database from the old master?
--
MS
12:05
Hi Marek,
I tried to reproduce your issue, but after following the steps you
described, nothing wrong happened in my environment.
Could you please share some more log snippets produced by Knot after the
changes you performed? It would be best, whenever the "dead loop"
occurs, to share the log from the first (and second) run of this loop.
I guess there is nothing wrong with copying the KASP database. The
signing process itself seems to be working fine, based on the log
snippet you shared earlier.
This issue is really suspicious. I can see no code path that might lead
to repeated zone 'load' event.
Thanks,
Libor
Dne 04. 02. 21 v 11:41 Marek Szuba napsal(a):
On 03/02/2021 16:55, libor.peltan wrote:
In order to attempt to mitigate the issue, you
may try purging zone
timers. Please back up the timers database first (by simply copying
the directory <storage>/timers somewhere), and then either `knotc
zone-purge example.com. +timers` or delete the directory (I assume
that you have just one zone configured at your server?).
There are other zones on that server but they are all from the same
set so I can manipulate the server with no major disruption.
Anyway, here is what I have tried (in this order) since my previous
e-mail:
- purging the timers
- downgrading knot to 2.9
- following the downgrade, deleting both the timers and the journal
- commenting out "journal-content: all" (the old master didn't use this)
Unfortunately none of these actions have helped.
Hmm. Could it be that something has gone wrong with the import of the
KASP database from the old master?
17:46
On 2021-02-04 02:41, Marek Szuba wrote:
On 03/02/2021 16:55, libor.peltan wrote:
I am by no means an expert here. But from my
experience. If the DB is somehow
not supported. The log will be filled with messages indicating that. Another
thought that comes to mind;
You indicated you transferred data from one machine to another. Did the perms
change in any way? IOW Can unbound read/write to the transferred data files?
Are they owned by the user unbound was/is installed as?
Just some thoughts/observations. :-)
--Chris
In order to attempt to mitigate the issue, you
may try purging zone timers.
Please back up the timers database first (by simply copying the directory
<storage>/timers somewhere), and then either `knotc zone-purge example.com.
+timers` or delete the directory (I assume that you have just one zone
configured at your server?).
There are other zones on that server but they are all from the same set so I
can
manipulate the server with no major disruption.
Anyway, here is what I have tried (in this order) since my previous e-mail:
- purging the timers
- downgrading knot to 2.9
- following the downgrade, deleting both the timers and the journal
- commenting out "journal-content: all" (the old master didn't use this)
Unfortunately none of these actions have helped.
Hmm. Could it be that something has gone wrong with the import of the KASP
database from the old master?
--
MS
5 Úno
5 Úno
13:01
On 04/02/2021 16:46, Chris wrote:
I am by no means an expert here. But from my
experience. If the DB is
somehow not supported. The log will be filled with messages
indicating that.
I thought it might have been something like that, especially given this
server runs on arm and the old master was an amd64 box; as a matter of
fact I *had* to use the dump-and-restore procedure for the KASP database
because the copied LMDB files had not been recognised as valid, by
either knotd or lmdb-utils.
That said, this also happens when Knot is launched on that server with
NO databases present. Yesterday evening I tried shutting knotd down,
deleting everything except the plain-text zone files from /var/lib/knot,
restarted - and as soon as Knot had generated new KSK and ZSK the log
spam started anew.
I honestly don't know what is going on here. It's as if there was
something wrong with this specifically on arm...
BTW. All the systems discussed here run Debian Buster and use Knot .deb
packages from deb.knot-dns.cz.
Another thought that comes to mind; You indicated you
transferred
data from one machine to another. Did the perms change in any way?
IOW Can unbound read/write to the transferred data files? Are they
owned by the user unbound was/is installed as?
I take it you meant knot and not unbound here :-) I've just checked
again and all the file-system permissions are exactly the same, apart
from the fact that the user and the group "knot" resolve to different
UIDs/GIDs on the two machines.
--
MS
17:16
On 2021-02-05 04:01, Marek Szuba wrote:
On 04/02/2021 16:46, Chris wrote:
That is of course possible. I felt qualified to
pose a possible solution.
As I had to move a couple of zones from a server we were retiring to a
different server. Which ran a newer version of knot by more than a couple
of points. As memory serves. I froze the zones prior to dumping. Tarred
them up, and then transferred them. Again, as I recall. I got errors
indicating
that the DB was incompatible. So I *think* I simply purged *all* history on
the
transferred zones. shut down knot. Deleted the zone DBs. Ensured that the
(transferred) zone algos were consistent in the newer (knot) version. Then
leaving the keys in the zone files. Asked knot to load them on restart. And
that was that. Your mileage may vary. ;-)
I am by no means an expert here. But from my
experience. If the DB is
somehow not supported. The log will be filled with messages
indicating that.
I thought it might have been something like that, especially given this
server
runs on arm and the old master was an amd64 box; as a matter of fact I *had*
to
use the dump-and-restore procedure for the KASP database because the copied
LMDB
files had not been recognised as valid, by either knotd or lmdb-utils.
That said, this also happens when Knot is launched on that server with NO
databases present. Yesterday evening I tried shutting knotd down, deleting
everything except the plain-text zone files from /var/lib/knot, restarted -
and as
soon as Knot had generated new KSK and ZSK the log spam started anew.
I honestly don't know what is going on here. It's as if there was something
wrong
with this specifically on arm...
BTW. All the systems discussed here run Debian Buster and use Knot .deb
packages
from deb.knot-dns.cz.
LOL. Yep. I did. I was pondering
a problem someone mentioned on another
mailing
list I'm subscribed to regarding unbound, when I was answering you -- DO'H!
;-)
Another thought that comes to mind; You indicated
you transferred
data from one machine to another. Did the perms change in any way?
IOW Can unbound read/write to the transferred data files? Are they
owned by the user unbound was/is installed as?
I take it you meant knot and not unbound here :-) I've just checked again and all the file-system
permissions are exactly the
same,
apart from the fact that the user and the group "knot" resolve to different
UIDs/GIDs on the two machines.
1187
days inactive
1189
days old
6 comments
3 participants
participants (3)
-
Chris -
libor.peltan -
Marek Szuba