Re: [knot-dns-users] Missing private keys in backup

Hi Libor, sure, here is the output: root@signer-0:/tmp/backup/keys/keys# strace -p 9781 strace: Process 9781 attached restart_syscall(<... resuming interrupted restart_syscall ...>) = 1 accept4(16, NULL, NULL, SOCK_NONBLOCK) = 23 recvmsg(23, {msg_namelen=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=23, events=POLLIN}], 1, 5000) = 1 ([{fd=23, revents=POLLIN}]) recvmsg(23, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\20\0\vzone-backup\21\0\0\32\0\v/tmp/backup"..., iov_len=262144}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 37 write(6, "2020-12-09T13:57:12+0000 info: c"..., 71) = 71 stat("/var/lib/knot/catalog/data.mdb", 0x7ffd0fb9f090) = -1 ENOENT (No such file or directory) stat("/var/lib/knot/journal/data.mdb", {st_mode=S_IFREG|0660, st_size=9936896, ...}) = 0 stat("/var/lib/knot/keys/data.mdb", {st_mode=S_IFREG|0660, st_size=45056, ...}) = 0 mkdir("/tmp/backup", 0770) = 0 openat(AT_FDCWD, "/tmp/backup/knot.backup.lockfile", O_RDONLY|O_CREAT|O_EXCL, 0600) = 24 mkdir("/tmp/backup", 0750) = -1 EEXIST (File exists) stat("/var/lib/knot/catalog/data.mdb", 0x7ffd0fb9efa0) = -1 ENOENT (No such file or directory) futex(0x7ffd0fb9f8e0, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7ffd0fb9f890, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7ffd0fb9f8e4, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7ffd0fb9f890, FUTEX_WAKE_PRIVATE, 1) = 1 sendmsg(23, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3", iov_len=1}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 1 recvmsg(23, {msg_namelen=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=23, events=POLLIN}], 1, 5000) = 1 ([{fd=23, revents=POLLIN}]) recvmsg(23, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0", iov_len=262144}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 1 close(23) = 0 poll([{fd=16, events=POLLIN}], 1, -1 Thomas On 09.12.20 13:34, libor.peltan wrote: > Hi Thomas, > > would you try attaching `strace` to the running Knot and try the backup > again, in order to see if some syscall(s) return error codes? > > Thanks much! > > Libor > > Dne 09. 12. 20 v 12:25 Thomas E. napsal(a): >> Hi Libor, >> >> >> # knotc -V >> knotc (Knot DNS), version 3.0.2 >> >> >> Thomas >> >> On 09.12.20 12:16, libor.peltan wrote: >>> Hi Thomas, >>> >>> aren't you using Knot version 3.0.0 (thus lower than 3.0.1) ? >>> >>> Libor >>> >>> Dne 09. 12. 20 v 11:39 Thomas E. napsal(a): >>>> Hi Libor, >>>> >>>> I deleted the backup directory a few times, but I receive the same >>>> error >>>> every time: >>>> >>>> 2020-12-09T10:31:52+0000 info: control, received command 'zone-backup' >>>> 2020-12-09T10:31:52+0000 warning: [xxx.] zone backup failed (not >>>> exists) >>>> 2020-12-09T10:31:52+0000 error: [xxx.] zone event 'backup/restore' >>>> failed (not exists) >>>> 2020-12-09T10:31:52+0000 warning: [yyy.] zone backup failed (not >>>> exists) >>>> 2020-12-09T10:31:52+0000 error: [yyy.] zone event 'backup/restore' >>>> failed (not exists) >>>> >>>> >>>> >>>> One of the keys is the ZSK of zone xxx and the oterh the KSKS of zone >>>> yyy. >>>> >>>> >>>> root@signer-0:/#  keymgr xxx list human | grep >>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da >>>> >>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da ksk=no  zsk=yes tag=39969 >>>> algorithm=7  size=1024 public-only=no  pre-active=0 >>>> publish=-1M3D17h49m4s ready=0 active=-1M3D17h49m4s retire-active=0 >>>> retire=0 post-active=0 revoke=0 remove=0 >>>> >>>> >>>> root@signer-0:/#  keymgr yyy list human | grep >>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d >>>> >>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d ksk=yes zsk=no  tag=37419 >>>> algorithm=7  size=2048 public-only=no  pre-active=0 >>>> publish=-1M3D18h11m7s ready=-1M3D18h11m7s active=-26D12h49m4s >>>> retire-active=0 retire=0 post-active=0 revoke=0 remove=0 >>>> >>>> Best regards, >>>> Thomas >>>> >>>> On 09.12.20 11:15, libor.peltan wrote: >>>>> Hi Thomas, >>>>> >>>>> could you please try if this issue is reproducible: if whenever you >>>>> attempt the backup (to a fresh empty target directory), it fails with >>>>> "not exists"? >>>>> >>>>> Could you please check if the keys that happen to make it to the >>>>> backup >>>>> belong to the same zone, or that it's one from each zone? (You >>>>> light use >>>>> `keymgr list` to check which key ID belongs to which zone.) >>>>> >>>>> Thanks, >>>>> >>>>> Libor >>>>> >>>>> Dne 08. 12. 20 v 21:36 Thomas napsal(a): >>>>>> Hi Libor, >>>>>> >>>>>> sorry, I was really too unspecific. >>>>>> >>>>>> I'm hosting 2 zones. These 4 keys are on the production machine: >>>>>> >>>>>> root@signer-0:/var/lib/knot/keys/keys# ls -alh >>>>>> >>>>>> >>>>>> -rw-r----- 1 knot knot 1,7K Nov  5 16:22 >>>>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d.pem >>>>>> -rw-r----- 1 knot knot  916 Nov  5 16:44 >>>>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da.pem >>>>>> -rw-r----- 1 knot knot  916 Nov  5 16:22 >>>>>> 6ebb8eb3ec2ddaf150119b4bc11b47dcec91621a.pem >>>>>> -rw-r----- 1 knot knot 1,7K Nov  5 16:44 >>>>>> d7e47e2909f4d5947d8fb8684cb79ed06feb4b0a.pem >>>>>> >>>>>> >>>>>> Performing a backup with the following command: >>>>>> >>>>>> # knotc zone-backup +backupdir /tmp/backup >>>>>> >>>>>> Backup directory after performing the backup shows: >>>>>> >>>>>> root@signer-0:/tmp/backup/keys/keys# ls -ahl >>>>>> >>>>>> -rw-r----- 1 knot knot 1,7K Dez  8 20:21 >>>>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d.pem >>>>>> -rw-r----- 1 knot knot  916 Dez  8 20:21 >>>>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da.pem >>>>>> >>>>>> 2 keys are missing. >>>>>> >>>>>> Hhmm ok, there is an error in the log: >>>>>> >>>>>> 2020-12-08T20:26:43+0000 info: control, received command >>>>>> 'zone-backup' >>>>>> 2020-12-08T20:26:43+0000 warning: [xxx.] zone backup failed (not >>>>>> exists) >>>>>> 2020-12-08T20:26:43+0000 error: [xxx.] zone event 'backup/restore' >>>>>> failed (not exists) >>>>>> 2020-12-08T20:26:43+0000 warning: [yyy.] zone backup failed (not >>>>>> exists) >>>>>> 2020-12-08T20:26:43+0000 error: [yyy.] zone event 'backup/restore' >>>>>> failed (not exists) >>>>>> >>>>>> I'm using the latest knot version. >>>>>> >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Thomas >>>>>> >>>>>> >>>>>> >>>>>> Am 08.12.20 um 16:56 schrieb libor.peltan: >>>>>>> Hi Thomas, >>>>>>> >>>>>>> could you be more specific about "half of private keys were in the >>>>>>> backup" ? How many were, how many weren't, and was there some >>>>>>> obvious >>>>>>> difference between them? >>>>>>> >>>>>>> Could you share the log snippets covering the backup and the restore >>>>>>> procedures? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Libor >>>>>>> >>>>>>> Dne 08. 12. 20 v 16:48 Thomas E. napsal(a): >>>>>>>> Hi (again), >>>>>>>> >>>>>>>> I was trying to backup and restore a server with the new knotc >>>>>>>> zone-backup/restore command. >>>>>>>> >>>>>>>> I recognized that only half of the private keys were in the backup, >>>>>>>> which leads to an error: >>>>>>>> >>>>>>>> 2020-12-08T14:44:00+0100 error: [xxx.] DNSSEC, failed to load >>>>>>>> private >>>>>>>> keys (not exists) >>>>>>>> 2020-12-08T14:44:00+0100 error: [xxx.] DNSSEC, failed to load keys >>>>>>>> (not >>>>>>>> exists) >>>>>>>> >>>>>>>> Shouldn't the backup contain all private keys? >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Thomas