12 Čen
2024
12 Čen
'24
17:40
so, school is out and the children are on the loose
2024-06-10T21:27:24.199750+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2620:171:c2::49@33322
2024-06-10T21:27:24.200561+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 167.99.160.10@14871
2024-06-10T21:27:24.200642+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 34.223.46.240@53392
2024-06-10T21:27:24.201218+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 167.99.160.10@2011
2024-06-10T21:27:24.201422+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 81.106.125.151@54192
2024-06-10T21:27:24.203263+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 34.223.46.240@53398
2024-06-10T21:27:24.203643+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 139.99.166.37@42942
2024-06-10T21:27:25.199585+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 3.228.173.229@34084
2024-06-10T21:27:25.199678+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 76.93.200.106@10371
2024-06-10T21:27:25.200951+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2a02:6b8:c04:262:0:433f:1:3@33586
2024-06-10T21:27:25.201029+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2600:3c09::f03c:93ff:fea9:4de0@54166
2024-06-10T21:27:25.201207+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 118.99.2.29@33170
2024-06-10T21:27:25.201385+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 213.187.92.252@40559
2024-06-10T21:27:26.200340+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2a02:6b8:c04:262:0:433f:1:3@33594
2024-06-10T21:27:26.200529+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 40.79.144.82@59683
2024-06-10T21:27:26.203837+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 103.85.93.93@60578
2024-06-10T21:27:26.205102+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 13.244.33.51@33812
2024-06-10T21:27:27.208589+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 18.139.204.179@46824
2024-06-10T21:27:27.210062+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 20.125.201.35@63627
2024-06-10T21:27:27.331742+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 172.217.37.144@64719
2024-06-10T21:27:27.332050+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 191.233.201.73@61718
2024-06-10T21:27:27.391797+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 81.106.125.151@50624
like tens of thousands. some children are like that.
so, we take this as an opportunity to learn a bit more about knot tuning
we shortened `tcp-idle-timeout: 2`
we set `tcp-max-clients: 20`
rate limiting seems unlikely to improve things as it is many sources, a
DDos
what else are we missing?
btw, it also whacked knot enough that it failed a resign cycle and we had
to add `unsafe-operation: no-check-keyset` to get back to signing.
clues appreciated. this can't be the only neighborhood with children.
randy
13 Čen
13 Čen
19:31
Hi!
What is your CPU (lscpu) and `knotc status workers`?
How do you install knot (our packages have increased limit on number of open files)?
Could you please provide us with the full list of terminated remote addresses? We (Knot
projects) have been implementing
some anti-DDoS solutions, so this could help us.
Daniel
On 6/10/24 23:48, Randy Bush wrote:
so, school is out and the children are on the loose
2024-06-10T21:27:24.199750+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2620:171:c2::49@33322
2024-06-10T21:27:24.200561+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 167.99.160.10@14871
2024-06-10T21:27:24.200642+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 34.223.46.240@53392
2024-06-10T21:27:24.201218+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 167.99.160.10@2011
2024-06-10T21:27:24.201422+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 81.106.125.151@54192
2024-06-10T21:27:24.203263+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 34.223.46.240@53398
2024-06-10T21:27:24.203643+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 139.99.166.37@42942
2024-06-10T21:27:25.199585+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 3.228.173.229@34084
2024-06-10T21:27:25.199678+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 76.93.200.106@10371
2024-06-10T21:27:25.200951+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2a02:6b8:c04:262:0:433f:1:3@33586
2024-06-10T21:27:25.201029+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2600:3c09::f03c:93ff:fea9:4de0@54166
2024-06-10T21:27:25.201207+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 118.99.2.29@33170
2024-06-10T21:27:25.201385+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 213.187.92.252@40559
2024-06-10T21:27:26.200340+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 2a02:6b8:c04:262:0:433f:1:3@33594
2024-06-10T21:27:26.200529+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 40.79.144.82@59683
2024-06-10T21:27:26.203837+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 103.85.93.93@60578
2024-06-10T21:27:26.205102+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 13.244.33.51@33812
2024-06-10T21:27:27.208589+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 18.139.204.179@46824
2024-06-10T21:27:27.210062+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 20.125.201.35@63627
2024-06-10T21:27:27.331742+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 172.217.37.144@64719
2024-06-10T21:27:27.332050+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 191.233.201.73@61718
2024-06-10T21:27:27.391797+00:00 rip knotd[1389]: notice: TCP, terminated inactive
client, address 81.106.125.151@50624
like tens of thousands. some children are like that.
so, we take this as an opportunity to learn a bit more about knot tuning
we shortened `tcp-idle-timeout: 2`
we set `tcp-max-clients: 20`
rate limiting seems unlikely to improve things as it is many sources, a
DDos
what else are we missing?
btw, it also whacked knot enough that it failed a resign cycle and we had
to add `unsafe-operation: no-check-keyset` to get back to signing.
clues appreciated. this can't be the only neighborhood with children.
randy
--
20:43
daniel:
What is your CPU (lscpu) and `knotc status workers`?
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 40 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
BIOS Vendor ID: QEMU
Model name: QEMU Virtual CPU version 2.5+
BIOS Model name: pc-i440fx-5.2 CPU @ 2.0GHz
BIOS CPU family: 1
CPU family: 6
Model: 6
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
Stepping: 3
BogoMIPS: 4389.68
Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca c
mov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_
good nopl xtopology cpuid tsc_known_freq pni cx16 x2ap
ic hypervisor lahf_lm cpuid_fault pti
Virtualization features:
Hypervisor vendor: KVM
Virtualization type: full
Caches (sum of all):
L1d: 128 KiB (4 instances)
L1i: 128 KiB (4 instances)
L2: 16 MiB (4 instances)
L3: 64 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-3
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: KVM: Mitigation: VMX unsupported
L1tf: Mitigation; PTE Inversion
Mds: Vulnerable: Clear CPU buffers attempted, no microcode;
SMT Host state unknown
Meltdown: Mitigation; PTI
Mmio stale data: Unknown: No mitigations
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Vulnerable
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointe
r sanitization
Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; P
BRSB-eIBRS Not affected; BHI Retpoline
Srbds: Not affected
Tsx async abort: Not affected
rip.psg.com:/var/lib/knot/signed# knotc status workers
UDP workers: 4, TCP workers: 10, XDP workers: 0, background workers: 4 (running: 0,
pending: 0)
How do you install knot (our packages have increased
limit on number
of open files)?
`apt install`
Could you please provide us with the full list of
terminated remote
addresses? We (Knot projects) have been implementing some anti-DDoS
solutions, so this could help us.
tcp sample as of a few days ago, https://archive.psg.com/attack.list.gz
randy
21:34
we are still chasing one anomaly
zones (LR, LB, PSG.COM, ....) have the one RRSIG, but
DNSSEC02 Error
The DNSKEY RRset is signed with an RRSIG with tag 3842 which cannot be validated by
the matching DNSKEY. Fetched from the nameservers with IP addresses "105.16.115.1;
105.16.170.1; 139.84.235.208; 158.38.0.181; 185.91.97.18; 2001:700:0:503::aa:5302;
2a01:3f0:0:306::53; 2a05:e380:2:4::2; 2a05:f480:3000:205f:5400:4ff:fea4:e565;
2c0f:feb0:2:1::1:8001; 2c0f:feb0:c:1::1:1; 77.72.229.254".
DNSSEC08 Error
The DNSKEY RRset is signed with an RRSIG with tag 3842 which cannot be validated by
the matching DNSKEY. Fetched from the nameservers with IP addresses "105.16.115.1;
105.16.170.1; 139.84.235.208; 158.38.0.181; 185.91.97.18; 2001:700:0:503::aa:5302;
2a01:3f0:0:306::53; 2a05:e380:2:4::2; 2a05:f480:3000:205f:5400:4ff:fea4:e565;
2c0f:feb0:2:1::1:8001; 2c0f:feb0:c:1::1:1; 77.72.229.254".
randy
14 Čen
14 Čen
14:19
I see two DNSKEY RRSIGs. If the zones are signed by Knot, there should be just one RRSIG
for DNSKEY.
Try `knotc zone-sign` to see if it removes the defective signatures.
Daniel
On 6/13/24 21:34, Randy Bush wrote:
we are still chasing one anomaly
zones (LR, LB, PSG.COM, ....) have the one RRSIG, but
DNSSEC02 Error
The DNSKEY RRset is signed with an RRSIG with tag 3842 which cannot be validated by
the matching DNSKEY. Fetched from the nameservers with IP addresses "105.16.115.1;
105.16.170.1; 139.84.235.208; 158.38.0.181; 185.91.97.18; 2001:700:0:503::aa:5302;
2a01:3f0:0:306::53; 2a05:e380:2:4::2; 2a05:f480:3000:205f:5400:4ff:fea4:e565;
2c0f:feb0:2:1::1:8001; 2c0f:feb0:c:1::1:1; 77.72.229.254".
DNSSEC08 Error
The DNSKEY RRset is signed with an RRSIG with tag 3842 which cannot be validated by
the matching DNSKEY. Fetched from the nameservers with IP addresses "105.16.115.1;
105.16.170.1; 139.84.235.208; 158.38.0.181; 185.91.97.18; 2001:700:0:503::aa:5302;
2a01:3f0:0:306::53; 2a05:e380:2:4::2; 2a05:f480:3000:205f:5400:4ff:fea4:e565;
2c0f:feb0:2:1::1:8001; 2c0f:feb0:c:1::1:1; 77.72.229.254".
randy
18:55
I see two DNSKEY RRSIGs. If the zones are signed by
Knot, there should
be just one RRSIG for DNSKEY. Try `knotc zone-sign` to see if it
removes the defective signatures.
so our attempts at diagnosis indicate that the primary, running knot,
and secondaries which run knot, return one RRSIG. secondaries running
bind return two. we're digging further.
randy
14:41
Thanks!
Four CPU cores aren't many, but they should still handle thousands of TCP connections
per second.
You could also reduce
https://www.knot-dns.cz/docs/3.3/singlehtml/index.html#tcp-max-clients (the default should
be 500k).
But it's a blind guess. It would be nice to see `perf top` during an attack.
Daniel
On 6/13/24 20:43, Randy Bush wrote:
daniel:
What is your CPU (lscpu) and `knotc status
workers`?
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 40 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
BIOS Vendor ID: QEMU
Model name: QEMU Virtual CPU version 2.5+
BIOS Model name: pc-i440fx-5.2 CPU @ 2.0GHz
BIOS CPU family: 1
CPU family: 6
Model: 6
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
Stepping: 3
BogoMIPS: 4389.68
Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca c
mov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_
good nopl xtopology cpuid tsc_known_freq pni cx16 x2ap
ic hypervisor lahf_lm cpuid_fault pti
Virtualization features:
Hypervisor vendor: KVM
Virtualization type: full
Caches (sum of all):
L1d: 128 KiB (4 instances)
L1i: 128 KiB (4 instances)
L2: 16 MiB (4 instances)
L3: 64 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-3
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: KVM: Mitigation: VMX unsupported
L1tf: Mitigation; PTE Inversion
Mds: Vulnerable: Clear CPU buffers attempted, no microcode;
SMT Host state unknown
Meltdown: Mitigation; PTI
Mmio stale data: Unknown: No mitigations
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Vulnerable
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointe
r sanitization
Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; P
BRSB-eIBRS Not affected; BHI Retpoline
Srbds: Not affected
Tsx async abort: Not affected
rip.psg.com:/var/lib/knot/signed# knotc status workers
UDP workers: 4, TCP workers: 10, XDP workers: 0, background workers: 4 (running: 0,
pending: 0)
How do you install knot (our packages have
increased limit on number
of open files)?
`apt install`
Could you please provide us with the full list of
terminated remote
addresses? We (Knot projects) have been implementing some anti-DDoS
solutions, so this could help us.
tcp sample as of a few days ago, https://archive.psg.com/attack.list.gz
randy
19:01
It would be nice to see `perf top` during an attack.
Samples: 61K of event 'cpu-clock:pppH', 4000 Hz, Event count (approx.): 6714836502
lost: 0/0 drop: 0/0
Overhead Shared Object Symbol
10.50% [kernel] [k] __softirqentry_text_start
7.51% [kernel] [k] _raw_spin_unlock_irqrestore
4.72% [kernel] [k] iowrite16
3.95% [kernel] [k] finish_task_switch.isra.0
2.08% [kernel] [k] syscall_enter_from_user_mode
1.04% libc.so.6 [.] epoll_wait
0.96% [kernel] [k] copy_user_generic_string
0.96% libc.so.6 [.] __sendmmsg
0.91% libc.so.6 [.] recvmmsg
0.83% [kernel] [k] __fget_light
0.70% libknot.so.13.0.0 [.] knot_rrset_to_wire_extra
0.64% libgmp.so.10.4.1 [.] __gmpn_cnd_add_n
0.59% libgmp.so.10.4.1 [.] __gmpn_cnd_sub_n
0.56% libhogweed.so.6.6 [.] _nettle_sec_tabselect
0.54% libgmp.so.10.4.1 [.] __gmpn_sqr_basecase
19:12
bumped memory
Samples: 29K of event 'cpu-clock:pppH', 4000 Hz, Event count (approx.): 4640795132
lost: 0/0 drop: 0/0
Overhead Command Shared Object
31.58% knotd [kernel]
30.80% swapper [kernel]
8.77% knotd knotd
7.22% knotd libknot.so.13.0.0
6.57% perf perf
5.58% knotd libc.so.6
2.47% perf libc.so.6
2.08% perf [kernel]
0.48% khugepaged [kernel]
0.40% systemd-journal [kernel]
0.40% systemd-journal libsystemd-shared-252.so
0.38% knotd libnettle.so.8.6
0.33% knotd ld-linux-x86-64.so.2
0.32% ksoftirqd/0 [kernel]
0.31% knotd [vdso]
0.24% systemd-journal libc.so.6
0.23% knotd libgnutls.so.30.34.3
0.22% perf-top-UI libslang.so.2.3.3
0.21% knotd liburcu.so.8.0.0
0.16% snmpd [kernel]
0.14% rs:main Q:Reg [kernel]
0.11% in:imuxsock [kernel]
0.07% rcu_preempt [kernel]
0.07% perf-top-UI libc.so.6
0.06% knotd libdnssec.so.9.0.0
19:15
Four CPU cores aren't many, but they should still
handle thousands of
TCP connections per second.
well above what we're seeing so far
You could also reduce
https://www.knot-dns.cz/docs/3.3/singlehtml/index.html#tcp-max-clients
(the default should be 500k).
yes, i have tried various values. i am trying to not inhibit transfers
to secondaries. can not find an allow-exception for them.
randy
20:12
we may be narrowing it down.
knot returns one RRSIG, bind two, see appended.
my guess is that, if this was generally true, we would have heard about
it before. so maybe it is something in how we're configured which
tickles bind secondaries the wrong way. still investigating.
randy
ryuu.rg.net:/Users/randy> dig +vc +dnssec +norec -t dnskey psg.com @rip.psg.com
; <<>> DiG 9.10.6 <<>> +vc +dnssec +norec -t dnskey psg.com
@rip.psg.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18969
;; flags: qr aa; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;psg.com. IN DNSKEY
;; ANSWER SECTION:
psg.com. 86400 IN DNSKEY 256 3 8
AwEAAZfG8Y++ZmGXwa1sgmHpruUSPljDwMR2pY5bUjjOaJNyUBeLlEAP
Fyya3MNAKryW26yTxFmwYmyt0UtXyc4L7Ib5/J/Ew+putYpjRfslwPlS
5TWblvnbiqGcY/ZMuGrtLeZkvK/o39vXM+Hy5y3xbG4Qu4ySiuW03xMM
pN50cr8+VcM2RDQn6/W6kESdiY8WaXyD1DT9eIgIyi5zTaOfhSB7u/g7
H+7LltCAiCZIcIF08CGbS1VEh0YUyw3Th1I6jiQmYeGG6OSGaci5SkjV
fGTDpHrJOjFlCnUVfg+cYc1YPEojbmo90qO/nG+VB5I+qDYtkU1IR8EB +qXNi7ZbBt8=
psg.com. 86400 IN DNSKEY 257 3 8
AwEAAaCgMhvfatdo1jeqr0AsHJY+QB/QVv2O+9W62Sfj+xKCbV5nGgvu
XqPq2A8tXKT1lG1YF0pe3/ABH2iYNZs7v/a6QAb1wEAYasNz6ZlvRca2
bDs6KXz/n2B/Oeb2JoWBJ6OqdNtzkDl6CYEOkQoDWRnbR9jlyINOQ0mN
xfTu2wbXMngSIz78yTadpieyuG/B/TsLQ1SlTUSf436G5NMdxzQ8r7j4
5nW7mEORzvvk5Z1mGtfX8v8taw4qFfoIlaf226N06lZ90jpnEHTOGSTA
T/ii5WVqjBZGFWFYWrNcHR51zHm4QAGKlZ5hzr6lrGZaXqgY7jE3GaOc 86mZhSlyYIs=
psg.com. 86400 IN RRSIG DNSKEY 8 2 86400 20240627155330
20240613142330 53567 psg.com. JYhwpuCx+3YcZuumCP2g/1iGCqmIKxR1h3FYP8GdwIjY2i8OZ/T91O5S
ml+jXmjfvhmb2nZ5+cV4i5KtUjUsS6otrpm4nxuNxUQwDZBxV1VEwFJc
frS7TaOC+BrsKndJJIVGQ1HftCHGWSIiE/JEeEgeMrRXVLdCKKzADC7e
oTYPOzf1piSO7rbHN4pGirIqTfBMci6xpc8BOlgc17DSB3aZJj5p3nEt
Ie/h2goOwh3hue0oh6nuarTnlJhyiKOSBCcSrCjTl1Gfzq9sKyflEA2N
NL0lJepqPkyf2kG+HkwGBKmrGlOeUDhNwR9qVwIvd/g/dtOscHnwTOWJ nuf7RQ==
;; Query time: 21 msec
;; SERVER: 2001:418:1::39#53(2001:418:1::39)
;; WHEN: Fri Jun 14 11:01:44 PDT 2024
;; MSG SIZE rcvd: 883
ryuu.rg.net:/Users/randy> dig +vc +dnssec +norec -t dnskey psg.com
@nlns.globnix.net
; <<>> DiG 9.10.6 <<>> +vc +dnssec +norec -t dnskey psg.com
@nlns.globnix.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9580
;; flags: qr aa; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;psg.com. IN DNSKEY
;; ANSWER SECTION:
psg.com. 86400 IN DNSKEY 256 3 8
AwEAAZfG8Y++ZmGXwa1sgmHpruUSPljDwMR2pY5bUjjOaJNyUBeLlEAP
Fyya3MNAKryW26yTxFmwYmyt0UtXyc4L7Ib5/J/Ew+putYpjRfslwPlS
5TWblvnbiqGcY/ZMuGrtLeZkvK/o39vXM+Hy5y3xbG4Qu4ySiuW03xMM
pN50cr8+VcM2RDQn6/W6kESdiY8WaXyD1DT9eIgIyi5zTaOfhSB7u/g7
H+7LltCAiCZIcIF08CGbS1VEh0YUyw3Th1I6jiQmYeGG6OSGaci5SkjV
fGTDpHrJOjFlCnUVfg+cYc1YPEojbmo90qO/nG+VB5I+qDYtkU1IR8EB +qXNi7ZbBt8=
psg.com. 86400 IN DNSKEY 257 3 8
AwEAAaCgMhvfatdo1jeqr0AsHJY+QB/QVv2O+9W62Sfj+xKCbV5nGgvu
XqPq2A8tXKT1lG1YF0pe3/ABH2iYNZs7v/a6QAb1wEAYasNz6ZlvRca2
bDs6KXz/n2B/Oeb2JoWBJ6OqdNtzkDl6CYEOkQoDWRnbR9jlyINOQ0mN
xfTu2wbXMngSIz78yTadpieyuG/B/TsLQ1SlTUSf436G5NMdxzQ8r7j4
5nW7mEORzvvk5Z1mGtfX8v8taw4qFfoIlaf226N06lZ90jpnEHTOGSTA
T/ii5WVqjBZGFWFYWrNcHR51zHm4QAGKlZ5hzr6lrGZaXqgY7jE3GaOc 86mZhSlyYIs=
psg.com. 86400 IN RRSIG DNSKEY 8 2 86400 20240626012025
20240611235025 53567 psg.com. G37kmJujQDabkfi9uQkgbaYfSm3f7D8Z7ulaH+a8MOaE23s1ZX0MMUkF
gaZ6ESgJechUXt7mWRnuLQtp+G5GhnQz80NO1ZUba3EPU4ITAd2MRykn
p3gM1dy82eGojjHDhLNIdE1FPExhmbluQx1WpCJPPCRc+oy0eAGfoLtu
cPFhBH1s31EVvN4wXF1x8LJ3GQz7kn7BehMDFHEA4lAX9L5zRsLmYX6J
5wWH9HZ2pCLkqzYR78/9iqmmmiUlEjfW0j0egjYCk1Fxm2GSpEMRy0q2
s+cChVRpg/WvHH2ORjGf9MyAFKyu7k71F0R/vncbU5mkdymR23UEvILt 1xuAYQ==
psg.com. 86400 IN RRSIG DNSKEY 8 2 86400 20240627155330
20240613142330 53567 psg.com. JYhwpuCx+3YcZuumCP2g/1iGCqmIKxR1h3FYP8GdwIjY2i8OZ/T91O5S
ml+jXmjfvhmb2nZ5+cV4i5KtUjUsS6otrpm4nxuNxUQwDZBxV1VEwFJc
frS7TaOC+BrsKndJJIVGQ1HftCHGWSIiE/JEeEgeMrRXVLdCKKzADC7e
oTYPOzf1piSO7rbHN4pGirIqTfBMci6xpc8BOlgc17DSB3aZJj5p3nEt
Ie/h2goOwh3hue0oh6nuarTnlJhyiKOSBCcSrCjTl1Gfzq9sKyflEA2N
NL0lJepqPkyf2kG+HkwGBKmrGlOeUDhNwR9qVwIvd/g/dtOscHnwTOWJ nuf7RQ==
;; Query time: 250 msec
;; SERVER: 2a02:898:31::53:0#53(2a02:898:31::53:0)
;; WHEN: Fri Jun 14 11:01:27 PDT 2024
;; MSG SIZE rcvd: 1178
22:48
Hi Randy,
Perhaps during the DDoS, the BIND secondary received a corrupt IXFR that
added a new RRSIG, but didn't delete the old one? If that's the case, the
old RRSIG will persist until you force AXFR; it's the only way to overwrite
the zone fully at the secondary. You can set "provide-ixfr: no" for this
zone, and reload the configuration and then re-sign the zone with "knotc
zone-sign <zone>". Once the secondary is corrected, you can remove the
"provide-ixfr" option to go back to the default of providing IXFR.
You should consider separating the signing and authoritative functions.
Your signer should only sign the zones, and provide XFR to permitted
secondaries. It's not a good idea to expose a signer directly to the
Internet.
Regards,
Anand
On Fri, 14 Jun 2024 at 20:13, Randy Bush <randy(a)psg.com> wrote:
we may be narrowing it down.
knot returns one RRSIG, bind two, see appended.
my guess is that, if this was generally true, we would have heard about
it before. so maybe it is something in how we're configured which
tickles bind secondaries the wrong way. still investigating.
randy
ryuu.rg.net:/Users/randy> dig +vc +dnssec +norec -t dnskey psg.com @
rip.psg.com
; <<>> DiG 9.10.6 <<>> +vc +dnssec +norec -t dnskey psg.com
@
rip.psg.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18969
;; flags: qr aa; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;psg.com. IN DNSKEY
;; ANSWER SECTION:
psg.com. 86400 IN DNSKEY 256 3 8
AwEAAZfG8Y++ZmGXwa1sgmHpruUSPljDwMR2pY5bUjjOaJNyUBeLlEAP
Fyya3MNAKryW26yTxFmwYmyt0UtXyc4L7Ib5/J/Ew+putYpjRfslwPlS
5TWblvnbiqGcY/ZMuGrtLeZkvK/o39vXM+Hy5y3xbG4Qu4ySiuW03xMM
pN50cr8+VcM2RDQn6/W6kESdiY8WaXyD1DT9eIgIyi5zTaOfhSB7u/g7
H+7LltCAiCZIcIF08CGbS1VEh0YUyw3Th1I6jiQmYeGG6OSGaci5SkjV
fGTDpHrJOjFlCnUVfg+cYc1YPEojbmo90qO/nG+VB5I+qDYtkU1IR8EB +qXNi7ZbBt8=
psg.com. 86400 IN DNSKEY 257 3 8
AwEAAaCgMhvfatdo1jeqr0AsHJY+QB/QVv2O+9W62Sfj+xKCbV5nGgvu
XqPq2A8tXKT1lG1YF0pe3/ABH2iYNZs7v/a6QAb1wEAYasNz6ZlvRca2
bDs6KXz/n2B/Oeb2JoWBJ6OqdNtzkDl6CYEOkQoDWRnbR9jlyINOQ0mN
xfTu2wbXMngSIz78yTadpieyuG/B/TsLQ1SlTUSf436G5NMdxzQ8r7j4
5nW7mEORzvvk5Z1mGtfX8v8taw4qFfoIlaf226N06lZ90jpnEHTOGSTA
T/ii5WVqjBZGFWFYWrNcHR51zHm4QAGKlZ5hzr6lrGZaXqgY7jE3GaOc 86mZhSlyYIs=
psg.com. 86400 IN RRSIG DNSKEY 8 2 86400
20240627155330 20240613142330 53567 psg.com.
JYhwpuCx+3YcZuumCP2g/1iGCqmIKxR1h3FYP8GdwIjY2i8OZ/T91O5S
ml+jXmjfvhmb2nZ5+cV4i5KtUjUsS6otrpm4nxuNxUQwDZBxV1VEwFJc
frS7TaOC+BrsKndJJIVGQ1HftCHGWSIiE/JEeEgeMrRXVLdCKKzADC7e
oTYPOzf1piSO7rbHN4pGirIqTfBMci6xpc8BOlgc17DSB3aZJj5p3nEt
Ie/h2goOwh3hue0oh6nuarTnlJhyiKOSBCcSrCjTl1Gfzq9sKyflEA2N
NL0lJepqPkyf2kG+HkwGBKmrGlOeUDhNwR9qVwIvd/g/dtOscHnwTOWJ nuf7RQ==
;; Query time: 21 msec
;; SERVER: 2001:418:1::39#53(2001:418:1::39)
;; WHEN: Fri Jun 14 11:01:44 PDT 2024
;; MSG SIZE rcvd: 883
ryuu.rg.net:/Users/randy> dig +vc +dnssec +norec -t dnskey psg.com @
nlns.globnix.net
; <<>> DiG 9.10.6 <<>> +vc +dnssec +norec -t dnskey psg.com
@
nlns.globnix.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9580
;; flags: qr aa; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;psg.com. IN DNSKEY
;; ANSWER SECTION:
psg.com. 86400 IN DNSKEY 256 3 8
AwEAAZfG8Y++ZmGXwa1sgmHpruUSPljDwMR2pY5bUjjOaJNyUBeLlEAP
Fyya3MNAKryW26yTxFmwYmyt0UtXyc4L7Ib5/J/Ew+putYpjRfslwPlS
5TWblvnbiqGcY/ZMuGrtLeZkvK/o39vXM+Hy5y3xbG4Qu4ySiuW03xMM
pN50cr8+VcM2RDQn6/W6kESdiY8WaXyD1DT9eIgIyi5zTaOfhSB7u/g7
H+7LltCAiCZIcIF08CGbS1VEh0YUyw3Th1I6jiQmYeGG6OSGaci5SkjV
fGTDpHrJOjFlCnUVfg+cYc1YPEojbmo90qO/nG+VB5I+qDYtkU1IR8EB +qXNi7ZbBt8=
psg.com. 86400 IN DNSKEY 257 3 8
AwEAAaCgMhvfatdo1jeqr0AsHJY+QB/QVv2O+9W62Sfj+xKCbV5nGgvu
XqPq2A8tXKT1lG1YF0pe3/ABH2iYNZs7v/a6QAb1wEAYasNz6ZlvRca2
bDs6KXz/n2B/Oeb2JoWBJ6OqdNtzkDl6CYEOkQoDWRnbR9jlyINOQ0mN
xfTu2wbXMngSIz78yTadpieyuG/B/TsLQ1SlTUSf436G5NMdxzQ8r7j4
5nW7mEORzvvk5Z1mGtfX8v8taw4qFfoIlaf226N06lZ90jpnEHTOGSTA
T/ii5WVqjBZGFWFYWrNcHR51zHm4QAGKlZ5hzr6lrGZaXqgY7jE3GaOc 86mZhSlyYIs=
psg.com. 86400 IN RRSIG DNSKEY 8 2 86400
20240626012025 20240611235025 53567 psg.com.
G37kmJujQDabkfi9uQkgbaYfSm3f7D8Z7ulaH+a8MOaE23s1ZX0MMUkF
gaZ6ESgJechUXt7mWRnuLQtp+G5GhnQz80NO1ZUba3EPU4ITAd2MRykn
p3gM1dy82eGojjHDhLNIdE1FPExhmbluQx1WpCJPPCRc+oy0eAGfoLtu
cPFhBH1s31EVvN4wXF1x8LJ3GQz7kn7BehMDFHEA4lAX9L5zRsLmYX6J
5wWH9HZ2pCLkqzYR78/9iqmmmiUlEjfW0j0egjYCk1Fxm2GSpEMRy0q2
s+cChVRpg/WvHH2ORjGf9MyAFKyu7k71F0R/vncbU5mkdymR23UEvILt 1xuAYQ==
psg.com. 86400 IN RRSIG DNSKEY 8 2 86400
20240627155330 20240613142330 53567 psg.com.
JYhwpuCx+3YcZuumCP2g/1iGCqmIKxR1h3FYP8GdwIjY2i8OZ/T91O5S
ml+jXmjfvhmb2nZ5+cV4i5KtUjUsS6otrpm4nxuNxUQwDZBxV1VEwFJc
frS7TaOC+BrsKndJJIVGQ1HftCHGWSIiE/JEeEgeMrRXVLdCKKzADC7e
oTYPOzf1piSO7rbHN4pGirIqTfBMci6xpc8BOlgc17DSB3aZJj5p3nEt
Ie/h2goOwh3hue0oh6nuarTnlJhyiKOSBCcSrCjTl1Gfzq9sKyflEA2N
NL0lJepqPkyf2kG+HkwGBKmrGlOeUDhNwR9qVwIvd/g/dtOscHnwTOWJ nuf7RQ==
;; Query time: 250 msec
;; SERVER: 2a02:898:31::53:0#53(2a02:898:31::53:0)
;; WHEN: Fri Jun 14 11:01:27 PDT 2024
;; MSG SIZE rcvd: 1178
--
15 Čen
15 Čen
0:17
hi anand,
Perhaps during the DDoS, the BIND secondary received a
corrupt IXFR that
added a new RRSIG, but didn't delete the old one? If that's the case, the
old RRSIG will persist until you force AXFR; it's the only way to overwrite
the zone fully at the secondary. You can set "provide-ixfr: no" for this
zone, and reload the configuration and then re-sign the zone with "knotc
zone-sign <zone>". Once the secondary is corrected, you can remove the
"provide-ixfr" option to go back to the default of providing IXFR.
bingo!!!
thank you
randy
0:28
Ok, that (finally!) makes sense, including explaining why forcing a
retransfer from the secondary's side with `rndc retransfer` fixes the
problem, since retransfer almost certainly does AXFR. Thanks Anand!
Remains the question of whether the cause was Knot generating a bad
IXFR or BIND9 failing to purge data from a partial IXFR.
7:21
It was the perfect storm…
Regards,
Saad
Sent from my iPhone
On 15 Jun 2024, at 8:09 AM, Rob Austein
<sra(a)hactrn.net> wrote:
Ok, that (finally!) makes sense, including explaining why forcing a
retransfer from the secondary's side with `rndc retransfer` fixes the
problem, since retransfer almost certainly does AXFR. Thanks Anand!
Remains the question of whether the cause was Knot generating a bad
IXFR or BIND9 failing to purge data from a partial IXFR.
--
10:25
Jun 15, 2024 07:09:12 Rob Austein <sra(a)hactrn.net>et>:
Ok, that (finally!) makes sense, including explaining
why forcing a
retransfer from the secondary's side with `rndc retransfer` fixes the
problem, since retransfer almost certainly does AXFR. Thanks Anand!
Remains the question of whether the cause was Knot generating a bad
IXFR or BIND9 failing to purge data from a partial IXFR.
--
Looking at the journal with: kjournal $zone
might help to answer that question.
Regards,
Tamás
16:17
I suspect that the RRSIG issue was a consequence of `Unsafe-Operation: No-Check-Keyset`.
I would definitely recommend investigating the journal history if you still have it.
Daniel
On 6/15/24 00:28, Rob Austein wrote:
Ok, that (finally!) makes sense, including explaining
why forcing a
retransfer from the secondary's side with `rndc retransfer` fixes the
problem, since retransfer almost certainly does AXFR. Thanks Anand!
Remains the question of whether the cause was Knot generating a bad
IXFR or BIND9 failing to purge data from a partial IXFR.
--
106
days inactive
109
days old
16 comments
6 participants
participants (6)
-
Anand Buddhdev -
Csillag Tamás -
Daniel Salzman -
Randy Bush -
Rob Austein -
Saad Itani