12 Úno
2024
12 Úno
'24
22:33
Hello,
today I had a long power outage that stopped my primary server for a
dozen hours
When I re-started the server, some zones refused to start :
Feb 12 21:55:17 arrakeen knotd[20728]: info: [geekwu.org.] DNSSEC, signing zone
Feb 12 21:55:17 arrakeen knotd[20728]: error: [geekwu.org.] zone event 're-sign'
failed (invalid parameter)
the invalid parameter was there was no active KSK for these zones, as a
keymgr list show
b37b6c2[...] 39945 KSK ECDSAP384SHA384 publish=1636966605 active=1637053005
7cc8622[...] 20799 ZSK ECDSAP384SHA384 created=1706695010
After manually setting published & active status in keygmr, reloading
these zones succeeded, and knot restarted to serve them.
Do you know how these zone could has failed as this ? They run fine on
auto-sign for years now. should I monitor closely the ones that need a
re-sign tomorrow ?
Regards,
--
Bastien
13 Úno
13 Úno
10:03
Hi Bastien,
could you please have a deeper look into the history of the zone in the
log file (or share it) ? There should be the answer hidden somewhere...
Thanks!
Libor
Dne 12. 02. 24 v 22:33 Bastien Durel napsal(a):
Hello,
today I had a long power outage that stopped my primary server for a
dozen hours
When I re-started the server, some zones refused to start :
Feb 12 21:55:17 arrakeen knotd[20728]: info: [geekwu.org.] DNSSEC, signing zone
Feb 12 21:55:17 arrakeen knotd[20728]: error: [geekwu.org.] zone event 're-sign'
failed (invalid parameter)
the invalid parameter was there was no active KSK for these zones, as a
keymgr list show
b37b6c2[...] 39945 KSK ECDSAP384SHA384 publish=1636966605 active=1637053005
7cc8622[...] 20799 ZSK ECDSAP384SHA384 created=1706695010
After manually setting published & active status in keygmr, reloading
these zones succeeded, and knot restarted to serve them.
Do you know how these zone could has failed as this ? They run fine on
auto-sign for years now. should I monitor closely the ones that need a
re-sign tomorrow ?
Regards,
22:00
Le 13/02/2024 à 10:03, libor.peltan a écrit :
Hi Bastien,
could you please have a deeper look into the history of the zone in the
log file (or share it) ? There should be the answer hidden somewhere...
Thanks!
Hello,
Looks like the problem was lying for a little while :
# journalctl -u knot | grep geekwu.org
janv. 26 22:50:10 arrakeen knotd[3061]: info:
[geekwu.org.] zone will be loaded
janv. 26 22:50:10 arrakeen knotd[3061]: info: [geekwu.org.] zone file parsed, serial
2024021105
janv. 26 22:50:27 arrakeen knotd[3061]: notice: [geekwu.org.] DNSSEC, cleared future
timers of auto-managed key 20414
janv. 26 22:50:27 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, key, tag 39945,
algorithm ECDSAP384SHA384, KSK, public, active
janv. 26 22:50:27 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, key, tag 20414,
algorithm ECDSAP384SHA384
janv. 26 22:50:27 arrakeen knotd[3061]: 2024-01-26T22:50:27+0100 error: [geekwu.org.]
DNSSEC, keys validation failed (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: 2024-01-26T22:50:27+0100 error: [geekwu.org.]
DNSSEC, failed to load keys (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: error: [geekwu.org.] DNSSEC, keys validation
failed (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: error: [geekwu.org.] DNSSEC, failed to load keys
(missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: 2024-01-26T22:50:27+0100 error: [geekwu.org.]
zone event 'load' failed (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: error: [geekwu.org.] zone event 'load'
failed (missing active KSK or ZSK)
févr. 12 21:38:02 arrakeen knotd[3061]: 2024-02-12T21:38:02+0100 error: [geekwu.org.]
zone event 're-sign' failed (invalid parameter)
févr. 12 21:38:02 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, signing zone
févr. 12 21:38:02 arrakeen knotd[3061]: error: [geekwu.org.] zone event 're-sign'
failed (invalid parameter)
févr. 12 22:08:57 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, dropping previous
signatures, re-signing zone
Nothing to do with the outage, then, but there's few info. No logs
before the 26/01 in journalctl, syslog mixes my 2 knot instances, so
it's difficult to read :/
--
Bastien Durel
20 Úno
20 Úno
23:13
Bastien Durel <bastien(a)durel.org> writes:
> could you please have a deeper look into the
history of the zone in
> the log file (or share it) ? There should be the answer hidden
> somewhere...
FYI, I hit this exact same problem recently. One of my zones stopped
signing because the KSK was marked as not active. I used the same
solution to redeploy it. And it only happened with one zone. You can
see the effects in this graph showing that all the other zones kept
resigning on a regular basis but one had a slow downward trend toward
expiring (which I caught 4 days out):
https://capturedonearth.com/temp/dnssec-days-remaining.png
Note that I also had a power failure a few days before (on the night of
the 4th/5th). I have a hard time seeing why it would be related but in
theory I supposed it could be.
--
Wes Hardaker
21 Úno
21 Úno
9:27
Hi Bastien,
What's the Knot version and your signing policy configuration?
Daniel
On 2/13/24 22:00, Bastien Durel wrote:
Le 13/02/2024 à 10:03, libor.peltan a écrit :
Hi Bastien,
could you please have a deeper look into the history of the zone in the log file (or
share it) ? There should be the answer hidden somewhere...
Thanks!
Hello,
Looks like the problem was lying for a little while :
# journalctl -u knot | grep geekwu.org
janv. 26 22:50:10 arrakeen knotd[3061]: info:
[geekwu.org.] zone will be loaded
janv. 26 22:50:10 arrakeen knotd[3061]: info: [geekwu.org.] zone file parsed, serial
2024021105
janv. 26 22:50:27 arrakeen knotd[3061]: notice: [geekwu.org.] DNSSEC, cleared future
timers of auto-managed key 20414
janv. 26 22:50:27 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, key, tag 39945,
algorithm ECDSAP384SHA384, KSK, public, active
janv. 26 22:50:27 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, key, tag 20414,
algorithm ECDSAP384SHA384
janv. 26 22:50:27 arrakeen knotd[3061]: 2024-01-26T22:50:27+0100 error: [geekwu.org.]
DNSSEC, keys validation failed (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: 2024-01-26T22:50:27+0100 error: [geekwu.org.]
DNSSEC, failed to load keys (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: error: [geekwu.org.] DNSSEC, keys validation
failed (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: error: [geekwu.org.] DNSSEC, failed to load keys
(missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: 2024-01-26T22:50:27+0100 error: [geekwu.org.]
zone event 'load' failed (missing active KSK or ZSK)
janv. 26 22:50:27 arrakeen knotd[3061]: error: [geekwu.org.] zone event 'load'
failed (missing active KSK or ZSK)
févr. 12 21:38:02 arrakeen knotd[3061]: 2024-02-12T21:38:02+0100 error: [geekwu.org.]
zone event 're-sign' failed (invalid parameter)
févr. 12 21:38:02 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, signing zone
févr. 12 21:38:02 arrakeen knotd[3061]: error: [geekwu.org.] zone event 're-sign'
failed (invalid parameter)
févr. 12 22:08:57 arrakeen knotd[3061]: info: [geekwu.org.] DNSSEC, dropping previous
signatures, re-signing zone
Nothing to do with the outage, then, but there's few info. No logs before the 26/01
in journalctl, syslog mixes my 2 knot instances, so it's difficult to read :/
9:35
Hi Wes,
Do you see the `DNSSEC, cleared future timers of auto-managed key` log message?
Daniel
On 2/20/24 23:13, Wes Hardaker wrote:
Bastien Durel <bastien(a)durel.org> writes:
> could you please have a deeper look into the
history of the zone in
> the log file (or share it) ? There should be the answer hidden
> somewhere...
FYI, I hit this exact same problem recently. One of my zones stopped
signing because the KSK was marked as not active. I used the same
solution to redeploy it. And it only happened with one zone. You can
see the effects in this graph showing that all the other zones kept
resigning on a regular basis but one had a slow downward trend toward
expiring (which I caught 4 days out):
https://capturedonearth.com/temp/dnssec-days-remaining.png
Note that I also had a power failure a few days before (on the night of
the 4th/5th). I have a hard time seeing why it would be related but in
theory I supposed it could be.
9:39
Le mercredi 21 février 2024 à 09:27 +0100, Daniel Salzman via knot-dns-
users a écrit :
Hi Bastien,
What's the Knot version and your signing policy configuration?
Hello,
I'm using the current version (3.3.4-cznic.1~bookworm)
The policy I use for the affected zones is this one :
policy:
- id: default_hsm
keystore: nitrokey
algorithm: ECDSAP384SHA384
ksk-size: 384
zsk-size: 384
nsec3: on
nsec3-salt-lifetime: 4d
ksk-submission: validating-resolver
nsec3-iterations: 0
keystore:
- id: nitrokey
backend: pkcs11
config: "pkcs11:pin-value=x /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so"
Regards,
--
Bastien
10:31
Hi Bastien and Wes,
one of the points to understand the issue is this log message
janv. 26 22:50:27 arrakeen knotd[3061]: notice:
[geekwu.org.] DNSSEC,
cleared future timers of auto-managed key 20414
In recent versions, we added a feature to Knot, that when Automatic key
management is enabled, any key timers that are scheduled in the future
are cleared. The reason was that auto-managed keys imported from Bind9
often had those and it lead to a mess in Knot's automatic key management.
It is hard to imagine for me how it could happen that this code cleared
your keys' "normal" timers that ought not be in the future. Is the issue
somehow reproducible for you so that we could be able to see the keys'
states just before this appears? Could you at least dig a bit deeper
down the logs to see some more history before this?
Could you explain if you routinely or occasionally do some manual
adjustments of the keys with keymgr?
Thank you!
Libor
Dne 20. 02. 24 v 23:13 Wes Hardaker napsal(a):
Bastien Durel <bastien(a)durel.org> writes:
> could you please have a deeper look into the
history of the zone in
> the log file (or share it) ? There should be the answer hidden
> somewhere...
FYI, I hit this exact same problem recently. One of my zones
stopped
signing because the KSK was marked as not active. I used the same
solution to redeploy it. And it only happened with one zone. You can
see the effects in this graph showing that all the other zones kept
resigning on a regular basis but one had a slow downward trend toward
expiring (which I caught 4 days out):
https://capturedonearth.com/temp/dnssec-days-remaining.png
Note that I also had a power failure a few days before (on the night of
the 4th/5th). I have a hard time seeing why it would be related but in
theory I supposed it could be.
11:30
Le mercredi 21 février 2024 à 10:31 +0100, libor.peltan via knot-dns-
users a écrit :
Hi Bastien and Wes,
one of the points to understand the issue is this log message
Hello,
I don´t usually use keymgr, only when I'm fixing things, or if I want
to make a KSK or alg. rollover. (according to git logs, the last
knot.conf modification related to dnssec was on June 22 2023)
I found this message only once in logs since august :
# grep 'cleared future timers of auto-managed' knotd.log
Jan 26 22:50:26 arrakeen knotd[3061]: notice: [durel.eu.] DNSSEC, cleared future timers of
auto-managed key 7402
Jan 26 22:50:26 arrakeen knotd[3061]: notice: [merlusina.eu.] DNSSEC, cleared future
timers of auto-managed key 46783
Jan 26 22:50:27 arrakeen knotd[3061]: notice: [geekwu.org.] DNSSEC, cleared future timers
of auto-managed key 20414
Jan 26 22:50:43 arrakeen knotd[6154]: notice: [durel.eu.] DNSSEC, cleared future timers of
auto-managed key 64035
Jan 26 22:50:43 arrakeen knotd[6154]: notice: [geekwu.org.] DNSSEC, cleared future timers
of auto-managed key 20799
knotd[6154] is my internal-view instance, while knotd[3061] is the
public one
only merlusina.eu. & geekwu.org. was broken, durel.eu. was not.
at 22:50 on Jan 26 I was probably playing boadgames in my local games
association
But actually, my logs are very strange : they jump from Feb 12 13:01 to
Jan 26 22:50
Feb 12 13:01:37 arrakeen freshclam[2243]: Mon Feb 12 13:01:37 2024 -> daily.cld
database is up-to-date (version: 27183, sigs: 2053128, f-level: 90, builder: raynman)
Feb 12 13:01:37 arrakeen freshclam[2243]: Mon Feb 12 13:01:37 2024 -> main.cvd database
is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Feb 12 13:01:37 arrakeen freshclam[2243]: Mon Feb 12 13:01:37 2024 -> bytecode.cld
database is up-to-date (version: 334, sigs: 91, f-level: 90, builder: anvilleg)
Jan 26 22:50:00 arrakeen systemd-modules-load[580]: Inserted module
'firewire_sbp2'
Jan 26 22:50:00 arrakeen systemd-modules-load[580]: Inserted module 'dummy'
Jan 26 22:50:00 arrakeen systemd-modules-load[580]: Inserted module 'coretemp'
Jan 26 22:50:00 arrakeen systemd-sysctl[598]: Couldn't write '1' to
'net/netfilter/nf_conntrack_acct', ignoring: No such file or directory
Jan 26 22:50:00 arrakeen kernel: [ 0.000000] microcode: microcode updated early to
revision 0x1d, date = 2018-05-11
Jan 26 22:50:00 arrakeen systemd-sysctl[598]: Couldn't write '1' to
'net/ipv4/conf/pacserve/mc_forwarding', ignoring: No such file or directory
Jan 26 22:50:00 arrakeen kernel: [ 0.000000] Linux version 6.1.0-18-amd64
(debian-kernel(a)lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils
for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01)
Jan 26 22:50:00 arrakeen kernel: [ 0.000000] Command line:
BOOT_IMAGE=/vmlinuz-6.1.0-18-amd64 root=LABEL=SSDROOT ro intremap=off reboot=cold
raid=noautodetect quiet
[...]
Jan 26 22:50:11 arrakeen Tor[3150]: Received local state file with skewed time
(/var/lib/tor/state): It seems that our clock is behind by 15 days, 23 hours, 23 minutes,
or that theirs is ahead. Tor requires an accurate clock to work: please check your time,
timezone, and date settings.
[...]
Jan 26 22:50:12 arrakeen vnstatd[2450]: Info: Latest database update is in the future (db:
2024-02-12 13:00:00 > now: 2024-01-26 22:50:12). Giving the system clock up to 5
minutes to sync before continuing.
[...]
Jan 26 22:59:48 arrakeen ntpd[3104]: IO: new interface(s) found: waking up resolver
Feb 12 21:38:02 arrakeen ntpd[3104]: CLOCK: time stepped by 1463893.399700
Feb 12 21:38:02 arrakeen ntpd[3104]: CLOCK: time changed from 2024-01-26 to 2024-02-12
So I guess the power cut messed with the computer date in a way or
another, and this is probably the thing that got timer to be purged.
Maybe I should make sure knotd starts after ntpd ^^
Regards,
--
Bastien
janv. 26 22:50:27 arrakeen knotd[3061]: notice:
[geekwu.org.]
DNSSEC,
cleared future timers of auto-managed key 20414
In recent versions, we added a feature to Knot, that when Automatic
key
management is enabled, any key timers that are scheduled in the
future
are cleared. The reason was that auto-managed keys imported from
Bind9
often had those and it lead to a mess in Knot's automatic key
management.
It is hard to imagine for me how it could happen that this code
cleared
your keys' "normal" timers that ought not be in the future. Is the
issue
somehow reproducible for you so that we could be able to see the
keys'
states just before this appears? Could you at least dig a bit deeper
down the logs to see some more history before this?
Could you explain if you routinely or occasionally do some manual
adjustments of the keys with keymgr?
Thank you!
Libor
21:50
"libor.peltan" <libor.peltan(a)nic.cz> writes:
Could you explain if you routinely or occasionally do
some manual
adjustments of the keys with keymgr?
I'll look into logs (give me a bit as I'm away from the machine in
question at the moment).
But some important points:
1. everything is set up for auto-signing and has been for quite some time
2. the keys were indeed imported from bind but otherwise I don't use
keymgr manually
3. it is possible I may have backup files of the keys predating the
event and will need to check. That might be helpful as well.
The irony; the domain in question? dnssec-tools.org (which used to be
signed and run by bind tools, but hasn't been for a while -- I *think*
since near Q1 2022).
--
Wes Hardaker
78
days inactive
87
days old
9 comments
4 participants
participants (4)
-
Bastien Durel -
Daniel Salzman -
libor.peltan -
Wes Hardaker