Hi Thomas, could you please send us you configuration (at least the algorithm currently set) and the output of `keymgr -c <knot_conf> <your.zone> list` ? Thanks, Libor Dne 20. 01. 21 v 20:21 Thomas napsal(a): > Hi, > > we performed successfully an algorithm rollover. After changing the > algorithm in the configuration file everything worked as expected. All > zones have been updated to the new algorithm. > > When I now sign a new zone the zone is being signed with the old > algorithm's key and an algorithm rollover is being triggered immediately. > > Is this an expected behavior? How can I avoid this? Do I have to delete > the old key? > > > Thanks a lot, > Thomas

Hi Thomas,auffälligke the key with tag 32875, having only the "ready" timer set, is obviously wrong and is causing the observed misbehavior. Do you have any record of how this situation emerged? For example, a log file and/or personal memoirs? Anyway, a side notice: as you probably know, when the configured (in policy) and current (in keyset) algorithms differs, Knot initiates an algorithm rollover. That is currently blocked for the specific zone because it looks like another key rollover is running (the ready key is present). As a consequence, once (if) you delete the ready key, Knot immediately starts with algorithm rollover to RSASHA512. BR, Libor Dne 02. 03. 21 v 17:53 Thomas E. napsal(a): > Hi Libor, > > sorry for the late reply. We noticed another strange behaviour in our > signer, which might be related to the previously stated issues. > > We signed a new zone, I'll refer to it as example.com in the examples, > and it looks like the internal key management got mixed up in the > process. > > E.g. the following CDS/CDNSKEY combination is published for the zone: > > $ dig cds example.com +short > 32875 8 2 7E476D751F633B44C1D1ED3A78F986EDCD588E58415CFA10B1256FCB > 10336E96 > > > $ dig cdnskey example.com +multiline > > example.com.            0 IN CDNSKEY 257 3 8 ( > > AwEAAdKaHEdWflf.......bW5fiKHe6sQYUjaXj/YCQcn18= > ) ; KSK; alg = RSASHA256 ; key id = 32875 > > > So then as usual our process starts and picks up the new CDS/CDNSKEY > records and attempts to upload them to the registry. Then the registry > tells us, that the key is actually not yet published and therefor they > won't accept our domain update. > > > A quick dig request does in fact support the issue raised by the > registry: > > > $ dig dnskey example.com +multiline > > example.com.            7170 IN DNSKEY 257 3 8 ( > > AwEAAb1nUFjsy.....cueakqu/h4JoqiGRbH0gS77Ylp > ) ; KSK; alg = RSASHA256 ; key id = 10051 > > example.com.            7170 IN DNSKEY 256 3 8 ( > > wEAAfTyql02Gh................CCp+DOxYYkKrYRdDEc > ) ; ZSK; alg = RSASHA256 ; key id = 6123 > > > Using the keymgr Tool we see the following entries for the zone: > > ~# keymgr -c /etc/knot/knot.conf example.com list > 87472c54eb669c66353503caa0a386b1e95602de ksk=yes zsk=no  tag=32875 > algorithm=8  size=2048 public-only=no  pre-active=0 publish=0 > ready=1551269424 active=0 retire-active=0 retire=0 post-active=0 > revoke=0 remove=0 > > 9304559cacf1f8a6e44a32591c5f6ba6a2ef002b ksk=yes zsk=no  tag=10051 > algorithm=8  size=2048 public-only=no  pre-active=0 publish=1551269695 > ready=0 active=1551269803 retire-active=0 retire=0 post-active=0 > revoke=0 remove=0 > > c2bd0743de6b4b07837e4936e983cae059adc928 ksk=no  zsk=yes tag=06123 > algorithm=8  size=1024 public-only=no  pre-active=0 publish=1551269424 > ready=0 active=1551269424 retire-active=0 retire=0 post-active=0 > revoke=0 remove=0 > > > Two KSK with Alg 8 have been created. Why 2 of them and why with Alg 8? > We don't have Alg 8 in our policy anymore. > Furthermore the key 32875 seems to be ready, but is not published in the > zone, but it has been published as CDNSKEY. The one that is published in > the zone has id 10051. There is something mixed up somehow. > > > > Our redacted knot.conf looks like this: > > server: >    listen: 192.168.1.123@53 >    user: knot:knot > > log: >    - target: /var/log/knot.log >      any: info > > control: >      listen: /var/run/knot/knot.sock > > remote: >    - id: local-resolver >      address: 192.168.1.2 > > submission: >    - id: resolver >      parent: local-resolver > > policy: >    - id: shared >      algorithm: RSASHA512 >      ksk-size: 2048 >      zsk-size: 1024 >      zsk-lifetime: 30d >      ksk-lifetime: 365d >      ksk-shared: true >      ksk-submission: resolver >      nsec3: true >      cds-cdnskey-publish: always > >    - id: unsigned >      algorithm: RSASHA512 >      ksk-size: 2048 >      zsk-size: 1024 >      zsk-lifetime: 30d >      ksk-lifetime: 365d >      ksk-shared: true >      nsec3: true >      cds-cdnskey-publish: delete-dnssec > > key: >    - id: <keya> >      algorithm: hmac-sha256 >      secret: <secreta> >    - id: <keyb> >      algorithm: hmac-sha256 >      secret: <secretb> > > remote: >    - id: <remotea> >      address: <ip1> >      address: <ip2> >      key: <keyb> >    - id: <remoteb> >      address: <ip3> >      key: <keya> > > acl: >    - id: <remotea> >      address: <ip1> >      address: <ip2> >      key: <keyb> >      action: notify > >    - id: <remoteb> >      address: <ip3> >      key: <keya> >      action: transfer > > > template: >    - id: default >      storage: "/var/lib/knot/" >      semantic-checks: on >      global-module: mod-stats >      serial-policy: unixtime >      dnssec-signing: off >      zonefile-load: difference > >    - id: signed >      storage: "/var/lib/knot/" >      dnssec-signing: on >      dnssec-policy: shared >      serial-policy: unixtime >      zonefile-load: difference > >    - id: unsigned >      storage: "/var/lib/knot/" >      dnssec-signing: on >      dnssec-policy: unsigned >      serial-policy: unixtime >      zonefile-load: difference > > zone: >    - domain: example.com. >      template: signed >      master: [<remotea>] >      notify: [<remoteb>] >      acl: [<remotea>, <remoteb>] >      acl: [<remotea>, <remoteb>] > > > > We have a few thousand zones set up in this way, with only a handful of > them being signed yet. > > We know that the algorithm RSASHA512 should not be used, but we wanted > to complete all key rollovers, before attempting another key rollover to > ECDSAP256SHA256. > > > Thanks a lot, > Thomas > > > On 20.01.21 21:41, libor.peltan wrote: >> Hi Thomas, >> >> could you please send us you configuration (at least the algorithm >> currently set) and the output of `keymgr -c <knot_conf> <your.zone> >> list` ? >> >> Thanks, >> >> Libor >> >> Dne 20. 01. 21 v 20:21 Thomas napsal(a): >>> Hi, >>> >>> we performed successfully an algorithm rollover. After changing the >>> algorithm in the configuration file everything worked as expected. All >>> zones have been updated to the new algorithm. >>> >>> When I now sign a new zone the zone is being signed with the old >>> algorithm's key and an algorithm rollover is being triggered >>> immediately. >>> >>> Is this an expected behavior? How can I avoid this? Do I have to delete >>> the old key? >>> >>> >>> Thanks a lot, >>> Thomas

A KSK and ZSK with Alg RSASHA256 have been created and the zone was signed. An algorithm rollover is triggered right after signing. I don't understand why RSASHA256 is still being used.

>>> policy: >>>    - id: shared >>>      algorithm: RSASHA512 >>>      ksk-size: 2048 >>>      zsk-size: 1024 >>>      zsk-lifetime: 30d >>>      ksk-lifetime: 365d >>>      ksk-shared: true >>>      ksk-submission: resolver >>>      nsec3: true >>>      cds-cdnskey-publish: always

On Mon, 8 Mar 2021 17:09:54 +0100 "Thomas E." &lt;list(a)crashcom.info&gt; wrote: This is a very wild guess but I'd suspect this has something to do with ksk-shared: true, note the config below. Btw. cds-cdnskey-publish: always is against instructions in rfc. those should only be published for rollover only. Is there some reason for using shared ksk?

Hi Thomas, yes, it shall be very smooth to simply remove the 'share' option from configuration and continue as before. The idea of shared KSK was generally good and a surprising enhancement, but you might have fallen to a random bug in rarely deployed configuration combo. I'm curious to hear if your issue disappears once you change the config. Please let me know, Libor Dne 09. 03. 21 v 13:15 Thomas E. napsal(a): > Thanks for the hint with the "cds-cdnskey-publish: always" issue. > > We are using shared ksk policy beause we thought it might be a good idea > to reduce the amount of keys in the database. We are going to sing more > than 10K zones and thought it might be a good thing to use shared keys > to speed up backups and restores. > But if this causes the actual problem I will change it of course. Do you > know if it's safe to remove the shared policy attribute from the config? > > Thanks, > Thomas > > On 08.03.21 17:55, Tuomo Soini wrote: >> On Mon, 8 Mar 2021 17:09:54 +0100 >> "Thomas E." &lt;list(a)crashcom.info&gt; wrote: >> >>> A KSK and ZSK with Alg RSASHA256 have been created and the zone was >>> signed. An  algorithm rollover is triggered right after signing. I >>> don't understand why RSASHA256 is still being used. >> This is a very wild guess but I'd suspect this has something to do with >> ksk-shared: true, note the config below. >> >>>>>> policy: >>>>>>     - id: shared >>>>>>       algorithm: RSASHA512 >>>>>>       ksk-size: 2048 >>>>>>       zsk-size: 1024 >>>>>>       zsk-lifetime: 30d >>>>>>       ksk-lifetime: 365d >>>>>>       ksk-shared: true >>>>>>       ksk-submission: resolver >>>>>>       nsec3: true >>>>>>       cds-cdnskey-publish: always >> Btw. cds-cdnskey-publish: always is against instructions in rfc. those >> should only be published for rollover only. >> >> Is there some reason for using shared ksk? >>

Hi Thomas, You could try purging possibly orphaned data: # knotc -f zone-purge -- +orphan Please make a backup of the defective situation first! Do you think that you could prepare and share (just with us) a minimal reproducer of the issue? It would help a lot with the debugging. Best, Daniel On 3/14/21 12:14 AM, Thomas wrote: > Hi Libor, > > we experience the same behavior after changing the config. Still 2 KSK > (and 2 ZSK) are being created after signing a zone. One with Alg 8 and > one with Alg 10. The one with Alg 8 is being used for signing and a > rollover is started immediately after signing. > > Don't now how to get out of the situation. Somehow there must be a > reference to the old Alg 8 somewhere. No idea why a new zone is singed > with Alg 8. > > Is there a way to clean up all the DNSSEC related data somehow? Actually > I would be able to un-sign and purge all zones, we are only signing a > few until now. But I don't know if this will solve the issue. > > Thanks, > Thomas > > > > On 09.03.21 13:40, libor.peltan wrote: >> Hi Thomas, >> >> yes, it shall be very smooth to simply remove the 'share' option from >> configuration and continue as before. >> >> The idea of shared KSK was generally good and a surprising enhancement, >> but you might have fallen to a random bug in rarely deployed >> configuration combo. >> >> I'm curious to hear if your issue disappears once you change the config. >> >> Please let me know, >> >> Libor >> >> Dne 09. 03. 21 v 13:15 Thomas E. napsal(a): >>> Thanks for the hint with the "cds-cdnskey-publish: always" issue. >>> >>> We are using shared ksk policy beause we thought it might be a good idea >>> to reduce the amount of keys in the database. We are going to sing more >>> than 10K zones and thought it might be a good thing to use shared keys >>> to speed up backups and restores. >>> But if this causes the actual problem I will change it of course. Do you >>> know if it's safe to remove the shared policy attribute from the config? >>> >>> Thanks, >>> Thomas >>> >>> On 08.03.21 17:55, Tuomo Soini wrote: >>>> On Mon, 8 Mar 2021 17:09:54 +0100 >>>> "Thomas E." &lt;list(a)crashcom.info&gt; wrote: >>>> >>>>> A KSK and ZSK with Alg RSASHA256 have been created and the zone was >>>>> signed. An  algorithm rollover is triggered right after signing. I >>>>> don't understand why RSASHA256 is still being used. >>>> This is a very wild guess but I'd suspect this has something to do with >>>> ksk-shared: true, note the config below. >>>> >>>>>>>> policy: >>>>>>>>     - id: shared >>>>>>>>       algorithm: RSASHA512 >>>>>>>>       ksk-size: 2048 >>>>>>>>       zsk-size: 1024 >>>>>>>>       zsk-lifetime: 30d >>>>>>>>       ksk-lifetime: 365d >>>>>>>>       ksk-shared: true >>>>>>>>       ksk-submission: resolver >>>>>>>>       nsec3: true >>>>>>>>       cds-cdnskey-publish: always >>>> Btw. cds-cdnskey-publish: always is against instructions in rfc. those >>>> should only be published for rollover only. >>>> >>>> Is there some reason for using shared ksk? >>>>

Hi Libor, hi Daniel, today we were finally able to isolate and to reproduce the problem. We made an issue in gitlab because it's much easier to read when everything is formatted. https://gitlab.nic.cz/knot/knot-dns/-/issues/721 BR, Thomas On 18.03.21 10:32, libor.peltan wrote: > Hi Thomas, > > do you please have any updates to your issue? > > Could you perhaps try to somehow clone your environment so that you had > the issue reproduced in a non-production setup ready for some more > experiments? > > First of all, I wonder if after you add a new zone to your configuration > and instead of re-configuring the server, sign the zone first with > `kzonesign -r`, the same behavior would be observed... > > Sorry for not helping you yet, but we are struggling understanding > what's going on at your setup :( > > BR, > > Libor > > Dne 14. 03. 21 v 8:53 Daniel Salzman napsal(a): >> Hi Thomas, >> >> You could try purging possibly orphaned data: >> # knotc -f zone-purge -- +orphan >> >> Please make a backup of the defective situation first! >> >> Do you think that you could prepare and share (just with us) a minimal >> reproducer of the issue? >> It would help a lot with the debugging. >> >> Best, >> Daniel >> >> On 3/14/21 12:14 AM, Thomas wrote: >>> Hi Libor, >>> >>> we experience the same behavior after changing the config. Still 2 KSK >>> (and 2 ZSK) are being created after signing a zone. One with Alg 8 and >>> one with Alg 10. The one with Alg 8 is being used for signing and a >>> rollover is started immediately after signing. >>> >>> Don't now how to get out of the situation. Somehow there must be a >>> reference to the old Alg 8 somewhere. No idea why a new zone is singed >>> with Alg 8. >>> >>> Is there a way to clean up all the DNSSEC related data somehow? Actually >>> I would be able to un-sign and purge all zones, we are only signing a >>> few until now. But I don't know if this will solve the issue. >>> >>> Thanks, >>> Thomas >>> >>> >>> >>> On 09.03.21 13:40, libor.peltan wrote: >>>> Hi Thomas, >>>> >>>> yes, it shall be very smooth to simply remove the 'share' option from >>>> configuration and continue as before. >>>> >>>> The idea of shared KSK was generally good and a surprising enhancement, >>>> but you might have fallen to a random bug in rarely deployed >>>> configuration combo. >>>> >>>> I'm curious to hear if your issue disappears once you change the >>>> config. >>>> >>>> Please let me know, >>>> >>>> Libor >>>> >>>> Dne 09. 03. 21 v 13:15 Thomas E. napsal(a): >>>>> Thanks for the hint with the "cds-cdnskey-publish: always" issue. >>>>> >>>>> We are using shared ksk policy beause we thought it might be a good >>>>> idea >>>>> to reduce the amount of keys in the database. We are going to sing >>>>> more >>>>> than 10K zones and thought it might be a good thing to use shared keys >>>>> to speed up backups and restores. >>>>> But if this causes the actual problem I will change it of course. >>>>> Do you >>>>> know if it's safe to remove the shared policy attribute from the >>>>> config? >>>>> >>>>> Thanks, >>>>> Thomas >>>>> >>>>> On 08.03.21 17:55, Tuomo Soini wrote: >>>>>> On Mon, 8 Mar 2021 17:09:54 +0100 >>>>>> "Thomas E." &lt;list(a)crashcom.info&gt; wrote: >>>>>> >>>>>>> A KSK and ZSK with Alg RSASHA256 have been created and the zone was >>>>>>> signed. An  algorithm rollover is triggered right after signing. I >>>>>>> don't understand why RSASHA256 is still being used. >>>>>> This is a very wild guess but I'd suspect this has something to do >>>>>> with >>>>>> ksk-shared: true, note the config below. >>>>>> >>>>>>>>>> policy: >>>>>>>>>>      - id: shared >>>>>>>>>>        algorithm: RSASHA512 >>>>>>>>>>        ksk-size: 2048 >>>>>>>>>>        zsk-size: 1024 >>>>>>>>>>        zsk-lifetime: 30d >>>>>>>>>>        ksk-lifetime: 365d >>>>>>>>>>        ksk-shared: true >>>>>>>>>>        ksk-submission: resolver >>>>>>>>>>        nsec3: true >>>>>>>>>>        cds-cdnskey-publish: always >>>>>> Btw. cds-cdnskey-publish: always is against instructions in rfc. >>>>>> those >>>>>> should only be published for rollover only. >>>>>> >>>>>> Is there some reason for using shared ksk? >>>>>>