12 Jun
2024
12 Jun
'24
7:17 a.m.
Hi folks,
Forwarding on behalf of Randy Bush since his mail server/DNS are being
DDoSed right now. Trying to troubleshoot, but any knot DNS expertise
would be greatly appreciated.
Regards
debian 12
knotc (Knot DNS), version 3.2.6
(aside: the server is under serious TCP and UDP DDoS)
all looks reasonable
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] control, received command
'zone-sign'
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, dropping previous
signatures, re-signing zone
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, key, tag 53567,
algorithm RSASHA256, KSK, public, active
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, key, tag 25843,
algorithm RSASHA256
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, key, tag 59161,
algorithm RSASHA256
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, key, tag 5489,
algorithm RSASHA256, KSK, public, active+
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, key, tag 22090,
algorithm RSASHA256, public
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, signing started
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, successfully
signed
Jun 12 03:36:17 rip.psg.com knotd[22495]: info: [psg.com.] DNSSEC, next signing at
2024-06-25T02:36:17+0000
# keymgr psg.com list
649b0d43d1493dd4ad30f8043ca4561c33c38b5a 53567 KSK RSASHA256/2048 publish=1078099200
active=1078099200
173597db4b4f2f072b568cb637710e891ac52246 25843 ZSK RSASHA256/2048 publish=1709251200
active=1709251200 retire=1717977600 remove=1717977600
3194d896f2a64f10b103991e5018b72cd3f1cd28 59161 ZSK RSASHA256/2048 publish=1709251200
active=1709251200 retire=1717977600 remove=1717977600
7b1bf414b34f605c68f9ddb7b52c32c6b53da8d3 5489 KSK RSASHA256/2048 publish=1718161132
902b8e02a5e75754bd69791735e76cb11c3e37af 22090 ZSK RSASHA256/2048 publish=1718161132
but no rrsig
# dig @localhost +vc +dnssec +norec -t dnskey psg.com
; <<>> DiG 9.18.24-1-Debian <<>> @localhost +vc +dnssec
+norec -t dnskey psg.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42269
;; flags: qr aa; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; COOKIE: 7f45da4f7305fdd5010000006669234c9ce14bdf78917f58 (good)
;; QUESTION SECTION:
;psg.com. IN DNSKEY
;; ANSWER SECTION:
psg.com. 86400 IN DNSKEY 256 3 8
AwEAAZfG8Y++ZmGXwa1sgmHpruUSPljDwMR2pY5bUjjOaJNyUBeLlEAP
Fyya3MNAKryW26yTxFmwYmyt0UtXyc4L7Ib5/J/Ew+putYpjRfslwPlS
5TWblvnbiqGcY/ZMuGrtLeZkvK/o39vXM+Hy5y3xbG4Qu4ySiuW03xMM
pN50cr8+VcM2RDQn6/W6kESdiY8WaXyD1DT9eIgIyi5zTaOfhSB7u/g7
H+7LltCAiCZIcIF08CGbS1VEh0YUyw3Th1I6jiQmYeGG6OSGaci5SkjV
fGTDpHrJOjFlCnUVfg+cYc1YPEojbmo90qO/nG+VB5I+qDYtkU1IR8EB +qXNi7ZbBt8=
psg.com. 86400 IN DNSKEY 257 3 8
AwEAAaCgMhvfatdo1jeqr0AsHJY+QB/QVv2O+9W62Sfj+xKCbV5nGgvu
XqPq2A8tXKT1lG1YF0pe3/ABH2iYNZs7v/a6QAb1wEAYasNz6ZlvRca2
bDs6KXz/n2B/Oeb2JoWBJ6OqdNtzkDl6CYEOkQoDWRnbR9jlyINOQ0mN
xfTu2wbXMngSIz78yTadpieyuG/B/TsLQ1SlTUSf436G5NMdxzQ8r7j4
5nW7mEORzvvk5Z1mGtfX8v8taw4qFfoIlaf226N06lZ90jpnEHTOGSTA
T/ii5WVqjBZGFWFYWrNcHR51zHm4QAGKlZ5hzr6lrGZaXqgY7jE3GaOc 86mZhSlyYIs=
psg.com. 86400 IN DNSKEY 257 3 8
AwEAAcXR1VhQRarToAaewor9xoQhrXbmUd9Ob0ruqOpDs5TO/jZLTOFE
W/g4V1yllr9t7tyLVJWA5jdZyJZO3otyu6S+OKvSLD8er3alStkgqI2Q
bLF3gUjtGxmcd/yIci4srWj401tv/6uWigN50+9Df0ClgUpmdjQ/ePq8
51DKtK51qGgc4vHwSYoWKQaGTofELiMDDXpzZSkQqAUveZYRzVTScUCQ
woVjQANSmio/u6JZtkwRnnUF9bChN51ydUY+uVH9NuoY6jEKJ27ZlIAP
4UgQ0h9epWy5JYV9bNQlhV+qpW1G3Zg/l58Yz5mWwh107HQIUefCgVP+ TTIusTwWH0E=
;; Query time: 0 msec
;; SERVER: ::1#53(localhost) (TCP)
;; WHEN: Wed Jun 12 04:25:48 UTC 2024
;; MSG SIZE rcvd: 892
maybe 35 zones with same policy and tenplate. one has RRSIGs no others
do.
mod-rrl:
- id: default
rate-limit: 12 # Allow 200 resp/s for each flow
slip: 2 # Approximately every other response slips
table-size: 900241
mod-cookies:
- id: default
secret-lifetime: 30h
badcookie-slip: 6
Wrong Cookie
Policy:
- Id: Pol-256-256
Algorithm: Rsasha256 # Was Ecdsap256sha256 Sra Uses Ecdsap384sha384
Manual: On
Delete-Delay: 30d
Unsafe-Operation: No-Check-Keyset
...
template:
- id: default
storage: /var/lib/knot/primary
semantic-checks: on
file: %s
global-module: mod-rrl/default
global-module: mod-cookies/default
- id: signed
storage: /var/lib/knot/signed
dnssec-signing: on
dnssec-policy: pol-256-256
semantic-checks: on
zonefile-sync: -1
zonefile-load: difference
journal-content: all
serial-policy: unixtime
sra and i have been beating our heads on this for two days. and there
are significant zones breaking
randy
--
Korry Luke
ルーク, コリー
koluke(a)wide.ad.jp
Graduate School of Media and Governance
Keio University Shonan Fujisawa Campus
慶應義塾大学湘南藤沢キャンパス 政策・メディア研究科
12 Jun
12 Jun
8:52 a.m.
Hi Korry, Randy,
> Manual: On
> Delete-Delay: 30d
> Unsafe-Operation: No-Check-Keyset
> ...
What is the original point in using unsafe operation within your zones?
In general, your key set is probably broken. As a result of the
unsafe-operation settings, Knot ignores the issue and tries to sign the
zone somehow, which however does not lead to good result in this situation.
(Anyway, the delete-delay has no effect in manual policy.)
>
> # keymgr psg.com list
> 649b0d43d1493dd4ad30f8043ca4561c33c38b5a 53567 KSK
> RSASHA256/2048 publish=1078099200 active=1078099200
> 173597db4b4f2f072b568cb637710e891ac52246 25843 ZSK
> RSASHA256/2048 publish=1709251200 active=1709251200 retire=1717977600
> remove=1717977600
> 3194d896f2a64f10b103991e5018b72cd3f1cd28 59161 ZSK
> RSASHA256/2048 publish=1709251200 active=1709251200 retire=1717977600
> remove=1717977600
> 7b1bf414b34f605c68f9ddb7b52c32c6b53da8d3 5489 KSK
> RSASHA256/2048 publish=1718161132
> 902b8e02a5e75754bd69791735e76cb11c3e37af 22090 ZSK
> RSASHA256/2048 publish=1718161132
>
>
There are two ZSKs already retired by far, no problem with them. Than
there is a KSK in active state. Do you actually expect this a to be a
CSK, signing both the DNSKEY and the rest of the zone? Besides this,
there is one KSK and one ZSK in "published" state, which means that they
are not used for signing.
I'd recommend you to clean up you keyset. The most straightforward way
would be to make the existing published ZSK active (keymgr psg.com. set
902b8e02a5e75754bd69791735e76cb11c3e37af active=+0). Than decide what to
do with the second published KSK (delete or what?). And finally, maybe
deleting the old keys might be good idea, depending on their purpose.
Please let us know how you proceed with this issue.
Anyway, we are somewhat interested in your experience from under-DDoS
operation. How severe they are and how they are affecting your
authoritative DNS service? Maybe there would be good point in trying out
XDP if the circumstances allow it?
Thanks,
Libor
5:41 p.m.
Hi Libor,
Thanks for the advice, that appears to have done the trick. Much
appreciated!
The original logic for the unsafe setting transition, as I understand,
was due to the effect of the DDoS, the signing host had started falling
behind with signing gradually to the point it missed a signing cycle and
started throwing errors/refusing to sign the zone entirely, which then
led to the
We are out of the woods for now it seems, but we'll likely need to work
on followup in the coming days/weeks and future mitigation plans,
perhaps including some of the tuning you mentioned. Randy will likely
follow up separately, I think his mail server and DNS should be back up
and resolvable by now.
Thanks again, just wanted to drop a quick note of appreciation.
On 2024-06-12 15:52, libor.peltan via knot-dns-users wrote:
Hi Korry, Randy,
--
Korry Luke
ルーク, コリー
koluke(a)wide.ad.jp
Graduate School of Media and Governance
Keio University Shonan Fujisawa Campus
慶應義塾大学湘南藤沢キャンパス 政策・メディア研究科
> Manual: On
> Delete-Delay: 30d
> Unsafe-Operation: No-Check-Keyset
> ...
What is the original point in using unsafe operation within your zones?
In general, your key set is probably broken. As a result of the
unsafe-operation settings, Knot ignores the issue and tries to sign
the zone somehow, which however does not lead to good result in this
situation.
(Anyway, the delete-delay has no effect in manual policy.)
>
> # keymgr psg.com list
> 649b0d43d1493dd4ad30f8043ca4561c33c38b5a 53567 KSK
> RSASHA256/2048 publish=1078099200 active=1078099200
> 173597db4b4f2f072b568cb637710e891ac52246 25843 ZSK
> RSASHA256/2048 publish=1709251200 active=1709251200
> retire=1717977600 remove=1717977600
> 3194d896f2a64f10b103991e5018b72cd3f1cd28 59161 ZSK
> RSASHA256/2048 publish=1709251200 active=1709251200
> retire=1717977600 remove=1717977600
> 7b1bf414b34f605c68f9ddb7b52c32c6b53da8d3 5489 KSK
> RSASHA256/2048 publish=1718161132
> 902b8e02a5e75754bd69791735e76cb11c3e37af 22090 ZSK
> RSASHA256/2048 publish=1718161132
>
>
There are two ZSKs already retired by far, no problem with them. Than
there is a KSK in active state. Do you actually expect this a to be a
CSK, signing both the DNSKEY and the rest of the zone? Besides this,
there is one KSK and one ZSK in "published" state, which means that
they are not used for signing.
I'd recommend you to clean up you keyset. The most straightforward way
would be to make the existing published ZSK active (keymgr psg.com.
set 902b8e02a5e75754bd69791735e76cb11c3e37af active=+0). Than decide
what to do with the second published KSK (delete or what?). And
finally, maybe deleting the old keys might be good idea, depending on
their purpose.
Please let us know how you proceed with this issue.
Anyway, we are somewhat interested in your experience from under-DDoS
operation. How severe they are and how they are affecting your
authoritative DNS service? Maybe there would be good point in trying
out XDP if the circumstances allow it?
Thanks,
Libor
-- 
6:17 p.m.
Please also mention to Randy that it would be much better to separate his
signing and authoritative DNS servers. Running it all on the same server is
probably not a good idea, especially with the signer being exposed to the
public Internet.
Regards,
Anand
On Wed, 12 Jun 2024 at 17:42, Korry Luke <koluke(a)wide.ad.jp> wrote:
Hi Libor,
Thanks for the advice, that appears to have done the trick. Much
appreciated!
The original logic for the unsafe setting transition, as I understand,
was due to the effect of the DDoS, the signing host had started falling
behind with signing gradually to the point it missed a signing cycle and
started throwing errors/refusing to sign the zone entirely, which then
led to the
We are out of the woods for now it seems, but we'll likely need to work
on followup in the coming days/weeks and future mitigation plans,
perhaps including some of the tuning you mentioned. Randy will likely
follow up separately, I think his mail server and DNS should be back up
and resolvable by now.
Thanks again, just wanted to drop a quick note of appreciation.
On 2024-06-12 15:52, libor.peltan via knot-dns-users wrote:
Hi Korry, Randy,
--
Korry Luke
ルーク, コリー
koluke(a)wide.ad.jp
Graduate School of Media and Governance
Keio University Shonan Fujisawa Campus
慶應義塾大学湘南藤沢キャンパス 政策・メディア研究科
--
> Manual: On
> Delete-Delay: 30d
> Unsafe-Operation: No-Check-Keyset
> ...
What is the original point in using unsafe operation within your zones?
In general, your key set is probably broken. As a result of the
unsafe-operation settings, Knot ignores the issue and tries to sign
the zone somehow, which however does not lead to good result in this
situation.
(Anyway, the delete-delay has no effect in manual policy.)
>
> # keymgr psg.com list
> 649b0d43d1493dd4ad30f8043ca4561c33c38b5a 53567 KSK
> RSASHA256/2048 publish=1078099200 active=1078099200
> 173597db4b4f2f072b568cb637710e891ac52246 25843 ZSK
> RSASHA256/2048 publish=1709251200 active=1709251200
> retire=1717977600 remove=1717977600
> 3194d896f2a64f10b103991e5018b72cd3f1cd28 59161 ZSK
> RSASHA256/2048 publish=1709251200 active=1709251200
> retire=1717977600 remove=1717977600
> 7b1bf414b34f605c68f9ddb7b52c32c6b53da8d3 5489 KSK
> RSASHA256/2048 publish=1718161132
> 902b8e02a5e75754bd69791735e76cb11c3e37af 22090 ZSK
> RSASHA256/2048 publish=1718161132
>
>
There are two ZSKs already retired by far, no problem with them. Than
there is a KSK in active state. Do you actually expect this a to be a
CSK, signing both the DNSKEY and the rest of the zone? Besides this,
there is one KSK and one ZSK in "published" state, which means that
they are not used for signing.
I'd recommend you to clean up you keyset. The most straightforward way
would be to make the existing published ZSK active (keymgr psg.com.
set 902b8e02a5e75754bd69791735e76cb11c3e37af active=+0). Than decide
what to do with the second published KSK (delete or what?). And
finally, maybe deleting the old keys might be good idea, depending on
their purpose.
Please let us know how you proceed with this issue.
Anyway, we are somewhat interested in your experience from under-DDoS
operation. How severe they are and how they are affecting your
authoritative DNS service? Maybe there would be good point in trying
out XDP if the circumstances allow it?
Thanks,
Libor
--
614
days inactive
614
days old
3 comments
3 participants
participants (3)
-
Anand Buddhdev -
Korry Luke -
libor.peltan