Re: [knot-dns-users] Missing private keys in backup

Hi Thomas, are you sure this strace has been collected while the issue was reproduced and half of the keys copied again? Maybe you just have deattached strace too early(?). Libor Dne 09. 12. 20 v 15:00 Thomas E. napsal(a): > Hi Libor, > > sure, here is the output: > > > root@signer-0:/tmp/backup/keys/keys# strace -p 9781 > strace: Process 9781 attached > restart_syscall(<... resuming interrupted restart_syscall ...>) = 1 > accept4(16, NULL, NULL, SOCK_NONBLOCK)  = 23 > recvmsg(23, {msg_namelen=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN > (Resource temporarily unavailable) > poll([{fd=23, events=POLLIN}], 1, 5000) = 1 ([{fd=23, revents=POLLIN}]) > recvmsg(23, {msg_name=NULL, msg_namelen=0, > msg_iov=[{iov_base="\1\20\0\vzone-backup\21\0\0\32\0\v/tmp/backup"..., > iov_len=262144}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, > MSG_DONTWAIT|MSG_NOSIGNAL) = 37 > write(6, "2020-12-09T13:57:12+0000 info: c"..., 71) = 71 > stat("/var/lib/knot/catalog/data.mdb", 0x7ffd0fb9f090) = -1 ENOENT (No > such file or directory) > stat("/var/lib/knot/journal/data.mdb", {st_mode=S_IFREG|0660, > st_size=9936896, ...}) = 0 > stat("/var/lib/knot/keys/data.mdb", {st_mode=S_IFREG|0660, > st_size=45056, ...}) = 0 > mkdir("/tmp/backup", 0770)              = 0 > openat(AT_FDCWD, "/tmp/backup/knot.backup.lockfile", > O_RDONLY|O_CREAT|O_EXCL, 0600) = 24 > mkdir("/tmp/backup", 0750)              = -1 EEXIST (File exists) > stat("/var/lib/knot/catalog/data.mdb", 0x7ffd0fb9efa0) = -1 ENOENT (No > such file or directory) > futex(0x7ffd0fb9f8e0, FUTEX_WAKE_PRIVATE, 1) = 1 > futex(0x7ffd0fb9f890, FUTEX_WAKE_PRIVATE, 1) = 1 > futex(0x7ffd0fb9f8e4, FUTEX_WAKE_PRIVATE, 1) = 1 > futex(0x7ffd0fb9f890, FUTEX_WAKE_PRIVATE, 1) = 1 > sendmsg(23, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3", > iov_len=1}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, > MSG_NOSIGNAL) = 1 > recvmsg(23, {msg_namelen=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN > (Resource temporarily unavailable) > poll([{fd=23, events=POLLIN}], 1, 5000) = 1 ([{fd=23, revents=POLLIN}]) > recvmsg(23, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0", > iov_len=262144}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, > MSG_DONTWAIT|MSG_NOSIGNAL) = 1 > close(23)                               = 0 > poll([{fd=16, events=POLLIN}], 1, -1 > > > Thomas > > > > On 09.12.20 13:34, libor.peltan wrote: >> Hi Thomas, >> >> would you try attaching `strace` to the running Knot and try the backup >> again, in order to see if some syscall(s) return error codes? >> >> Thanks much! >> >> Libor >> >> Dne 09. 12. 20 v 12:25 Thomas E. napsal(a): >>> Hi Libor, >>> >>> >>> # knotc -V >>> knotc (Knot DNS), version 3.0.2 >>> >>> >>> Thomas >>> >>> On 09.12.20 12:16, libor.peltan wrote: >>>> Hi Thomas, >>>> >>>> aren't you using Knot version 3.0.0 (thus lower than 3.0.1) ? >>>> >>>> Libor >>>> >>>> Dne 09. 12. 20 v 11:39 Thomas E. napsal(a): >>>>> Hi Libor, >>>>> >>>>> I deleted the backup directory a few times, but I receive the same >>>>> error >>>>> every time: >>>>> >>>>> 2020-12-09T10:31:52+0000 info: control, received command >>>>> 'zone-backup' >>>>> 2020-12-09T10:31:52+0000 warning: [xxx.] zone backup failed (not >>>>> exists) >>>>> 2020-12-09T10:31:52+0000 error: [xxx.] zone event 'backup/restore' >>>>> failed (not exists) >>>>> 2020-12-09T10:31:52+0000 warning: [yyy.] zone backup failed (not >>>>> exists) >>>>> 2020-12-09T10:31:52+0000 error: [yyy.] zone event 'backup/restore' >>>>> failed (not exists) >>>>> >>>>> >>>>> >>>>> One of the keys is the ZSK of zone xxx and the oterh the KSKS of zone >>>>> yyy. >>>>> >>>>> >>>>> root@signer-0:/#  keymgr xxx list human | grep >>>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da >>>>> >>>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da ksk=no  zsk=yes tag=39969 >>>>> algorithm=7  size=1024 public-only=no  pre-active=0 >>>>> publish=-1M3D17h49m4s ready=0 active=-1M3D17h49m4s retire-active=0 >>>>> retire=0 post-active=0 revoke=0 remove=0 >>>>> >>>>> >>>>> root@signer-0:/#  keymgr yyy list human | grep >>>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d >>>>> >>>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d ksk=yes zsk=no  tag=37419 >>>>> algorithm=7  size=2048 public-only=no  pre-active=0 >>>>> publish=-1M3D18h11m7s ready=-1M3D18h11m7s active=-26D12h49m4s >>>>> retire-active=0 retire=0 post-active=0 revoke=0 remove=0 >>>>> >>>>> Best regards, >>>>> Thomas >>>>> >>>>> On 09.12.20 11:15, libor.peltan wrote: >>>>>> Hi Thomas, >>>>>> >>>>>> could you please try if this issue is reproducible: if whenever you >>>>>> attempt the backup (to a fresh empty target directory), it fails >>>>>> with >>>>>> "not exists"? >>>>>> >>>>>> Could you please check if the keys that happen to make it to the >>>>>> backup >>>>>> belong to the same zone, or that it's one from each zone? (You >>>>>> light use >>>>>> `keymgr list` to check which key ID belongs to which zone.) >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Libor >>>>>> >>>>>> Dne 08. 12. 20 v 21:36 Thomas napsal(a): >>>>>>> Hi Libor, >>>>>>> >>>>>>> sorry, I was really too unspecific. >>>>>>> >>>>>>> I'm hosting 2 zones. These 4 keys are on the production machine: >>>>>>> >>>>>>> root@signer-0:/var/lib/knot/keys/keys# ls -alh >>>>>>> >>>>>>> >>>>>>> -rw-r----- 1 knot knot 1,7K Nov  5 16:22 >>>>>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d.pem >>>>>>> -rw-r----- 1 knot knot  916 Nov  5 16:44 >>>>>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da.pem >>>>>>> -rw-r----- 1 knot knot  916 Nov  5 16:22 >>>>>>> 6ebb8eb3ec2ddaf150119b4bc11b47dcec91621a.pem >>>>>>> -rw-r----- 1 knot knot 1,7K Nov  5 16:44 >>>>>>> d7e47e2909f4d5947d8fb8684cb79ed06feb4b0a.pem >>>>>>> >>>>>>> >>>>>>> Performing a backup with the following command: >>>>>>> >>>>>>> # knotc zone-backup +backupdir /tmp/backup >>>>>>> >>>>>>> Backup directory after performing the backup shows: >>>>>>> >>>>>>> root@signer-0:/tmp/backup/keys/keys# ls -ahl >>>>>>> >>>>>>> -rw-r----- 1 knot knot 1,7K Dez  8 20:21 >>>>>>> 087cc573318e070befff1d9cbcf07e3b5cf5444d.pem >>>>>>> -rw-r----- 1 knot knot  916 Dez  8 20:21 >>>>>>> 1fb3900b2e5ac72d30f927016ea4546ca561a5da.pem >>>>>>> >>>>>>> 2 keys are missing. >>>>>>> >>>>>>> Hhmm ok, there is an error in the log: >>>>>>> >>>>>>> 2020-12-08T20:26:43+0000 info: control, received command >>>>>>> 'zone-backup' >>>>>>> 2020-12-08T20:26:43+0000 warning: [xxx.] zone backup failed (not >>>>>>> exists) >>>>>>> 2020-12-08T20:26:43+0000 error: [xxx.] zone event 'backup/restore' >>>>>>> failed (not exists) >>>>>>> 2020-12-08T20:26:43+0000 warning: [yyy.] zone backup failed (not >>>>>>> exists) >>>>>>> 2020-12-08T20:26:43+0000 error: [yyy.] zone event 'backup/restore' >>>>>>> failed (not exists) >>>>>>> >>>>>>> I'm using the latest knot version. >>>>>>> >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Thomas >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am 08.12.20 um 16:56 schrieb libor.peltan: >>>>>>>> Hi Thomas, >>>>>>>> >>>>>>>> could you be more specific about "half of private keys were in the >>>>>>>> backup" ? How many were, how many weren't, and was there some >>>>>>>> obvious >>>>>>>> difference between them? >>>>>>>> >>>>>>>> Could you share the log snippets covering the backup and the >>>>>>>> restore >>>>>>>> procedures? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Libor >>>>>>>> >>>>>>>> Dne 08. 12. 20 v 16:48 Thomas E. napsal(a): >>>>>>>>> Hi (again), >>>>>>>>> >>>>>>>>> I was trying to backup and restore a server with the new knotc >>>>>>>>> zone-backup/restore command. >>>>>>>>> >>>>>>>>> I recognized that only half of the private keys were in the >>>>>>>>> backup, >>>>>>>>> which leads to an error: >>>>>>>>> >>>>>>>>> 2020-12-08T14:44:00+0100 error: [xxx.] DNSSEC, failed to load >>>>>>>>> private >>>>>>>>> keys (not exists) >>>>>>>>> 2020-12-08T14:44:00+0100 error: [xxx.] DNSSEC, failed to load >>>>>>>>> keys >>>>>>>>> (not >>>>>>>>> exists) >>>>>>>>> >>>>>>>>> Shouldn't the backup contain all private keys? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Thomas