5 Dub
2024
5 Dub
'24
13:22
Hi,
for some time now I have a problem running kresd on my raspberry pi.
I am running pihole and use kresd as resolver behind pihole. Everything
works fine until some day where kresd "decides" to crash. It is always
the same error message (please see below).
I then have to manually stop the garbage collector, remove all cache
files and restart kresd (which automatically starts the garbage collector).
My pi is pxe boot and the root partition is on an nfs mounted volume.
The volume has several terra byte of space left.
After a restart it will run flawless for tow or three weeks just to
crash with the same error again.
Any idea how this happens?
/Ulrich
root@pi-hole1:~# systemctl status kresd
● kresd.service - Knot Resolver daemon
Loaded: loaded (/lib/systemd/system/kresd.service; enabled; vendor
preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-04-05 02:31:47
CEST; 6h ago
Docs: man:kresd.systemd(7)
man:kresd(8)
Process: 5106 ExecStart=/usr/sbin/kresd -c
/usr/lib/arm-linux-gnueabihf/knot-resolver/distro-preconfig.lua -c
/etc/knot-resolver/kresd.conf -n (code=exited, status=1/FAILURE)
Process: 5110 ExecStopPost=/usr/bin/env rm -f
/run/knot-resolver/control/ (code=exited, status=1/FAILURE)
Main PID: 5106 (code=exited, status=1/FAILURE)
CPU: 648ms
Apr 05 02:31:45 pi-hole1 kresd[5106]: [C]: at 0x0001b2d8
Apr 05 02:31:45 pi-hole1 kresd[5106]: [C]: in function 'pcall'
Apr 05 02:31:45 pi-hole1 kresd[5106]:
...b/arm-linux-gnueabihf/knot-resolver/distro-preconfig.lua:9: in main chunk
Apr 05 02:31:45 pi-hole1 kresd[5106]: ERROR: net.listen() failed to bind
Apr 05 02:31:46 pi-hole1 kresd[5106]: [system] error while loading
config: /usr/lib/arm-linux-gnueabihf/knot-resolver/sandbox.lua:402:
can't open cache path '/var/cache/knot-resolver'; working directory
'/var/lib/knot-resolver'; No space left on device (workdir
'/var/lib/knot-resolver')
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Main process exited,
code=exited, status=1/FAILURE
Apr 05 02:31:47 pi-hole1 env[5110]: rm: cannot remove
'/run/knot-resolver/control/': Is a directory
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Control process
exited, code=exited, status=1/FAILURE
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Failed with result
'exit-code'.
Apr 05 02:31:47 pi-hole1 systemd[1]: Failed to start Knot Resolver daemon.
root@pi-hole1:~# ps aux Op | grep kres
knot-re+ 569 0.1 0.1 114480 4140 ? Ss Apr02 5:15
/usr/sbin/kres-cache-gc -c /var/cache/knot-resolver -d 1000
root 23835 0.0 0.0 10292 504 pts/0 S+ 09:00 0:00 grep kres
root@pi-hole1:~# systemctl status | grep kres
│ └─23861 grep kres
├─system-kresd.slice
│ └─kres-cache-gc.service
│ └─569 /usr/sbin/kres-cache-gc -c
/var/cache/knot-resolver -d 1000
root@pi-hole1:~# systemctl stop kres-cache-gc
5 Dub
5 Dub
14:31
Hi, Ulrich,
I am not very familiar with setups where the whole root is nfs-mounted,
but I have a suspicion it could be the cause if the system also treats
/var/cache as nfs-mounted (this can be checked using the 'mount' command
without arguments). This could cause problems, as the cache is mapped to
Resolver's virtual memory and the Resolver writes into it frequently,
which I suspect could fill up some kind of an nfs journal or something
like that, causing the "No space left on device" error.
What I would recommend is to get the cache (and ideally the workdir as
well) into a tmpfs-mounted space (this is generally /tmp, /dev/shm, or
anything you specify using /etc/fstab) so that it is locally in the
RAM/swap. To move the cache, use the cache.storage property in the
config [1].
[1]:
https://knot-resolver.readthedocs.io/en/stable/daemon-bindings-cache.html#e…
Best regards,
Oto Šťáva | Knot Resolver team leader | CZ.NIC z.s.p.o.
On 4/5/24 1:22 PM, Ulrich Wisser via knot-resolver-users wrote:
Hi,
for some time now I have a problem running kresd on my raspberry pi.
I am running pihole and use kresd as resolver behind pihole.
Everything works fine until some day where kresd "decides" to crash.
It is always the same error message (please see below).
I then have to manually stop the garbage collector, remove all cache
files and restart kresd (which automatically starts the garbage
collector).
My pi is pxe boot and the root partition is on an nfs mounted volume.
The volume has several terra byte of space left.
After a restart it will run flawless for tow or three weeks just to
crash with the same error again.
Any idea how this happens?
/Ulrich
root@pi-hole1:~# systemctl status kresd
● kresd.service - Knot Resolver daemon
Loaded: loaded (/lib/systemd/system/kresd.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-04-05 02:31:47
CEST; 6h ago
Docs: man:kresd.systemd(7)
man:kresd(8)
Process: 5106 ExecStart=/usr/sbin/kresd -c
/usr/lib/arm-linux-gnueabihf/knot-resolver/distro-preconfig.lua -c
/etc/knot-resolver/kresd.conf -n (code=exited, status=1/FAILURE)
Process: 5110 ExecStopPost=/usr/bin/env rm -f
/run/knot-resolver/control/ (code=exited, status=1/FAILURE)
Main PID: 5106 (code=exited, status=1/FAILURE)
CPU: 648ms
Apr 05 02:31:45 pi-hole1 kresd[5106]: [C]: at 0x0001b2d8
Apr 05 02:31:45 pi-hole1 kresd[5106]: [C]: in function 'pcall'
Apr 05 02:31:45 pi-hole1 kresd[5106]:
...b/arm-linux-gnueabihf/knot-resolver/distro-preconfig.lua:9: in main
chunk
Apr 05 02:31:45 pi-hole1 kresd[5106]: ERROR: net.listen() failed to bind
Apr 05 02:31:46 pi-hole1 kresd[5106]: [system] error while loading
config: /usr/lib/arm-linux-gnueabihf/knot-resolver/sandbox.lua:402:
can't open cache path '/var/cache/knot-resolver'; working directory
'/var/lib/knot-resolver'; No space left on device (workdir
'/var/lib/knot-resolver')
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Main process
exited, code=exited, status=1/FAILURE
Apr 05 02:31:47 pi-hole1 env[5110]: rm: cannot remove
'/run/knot-resolver/control/': Is a directory
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Control process
exited, code=exited, status=1/FAILURE
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Failed with result
'exit-code'.
Apr 05 02:31:47 pi-hole1 systemd[1]: Failed to start Knot Resolver
daemon.
root@pi-hole1:~# ps aux Op | grep kres
knot-re+ 569 0.1 0.1 114480 4140 ? Ss Apr02 5:15
/usr/sbin/kres-cache-gc -c /var/cache/knot-resolver -d 1000
root 23835 0.0 0.0 10292 504 pts/0 S+ 09:00 0:00 grep
kres
root@pi-hole1:~# systemctl status | grep kres
│ └─23861 grep kres
├─system-kresd.slice
│ └─kres-cache-gc.service
│ └─569 /usr/sbin/kres-cache-gc -c
/var/cache/knot-resolver -d 1000
root@pi-hole1:~# systemctl stop kres-cache-gc
--
16:41
Yes, my cache directory is nfs mounted.
It seems unlikely that nfs is the source, as the error prevails even if
I restart kresd (and gc). Only removal of the cache files solves the
problem.
I have changed my config to use a different directory, one on tmpfs. Now
I'll wait and see.
Thanks for the fast help!
/Ulrich
Am 05.04.24 um 14:31 schrieb Oto Šťáva via knot-resolver-users:
Hi, Ulrich,
I am not very familiar with setups where the whole root is
nfs-mounted, but I have a suspicion it could be the cause if the
system also treats /var/cache as nfs-mounted (this can be checked
using the 'mount' command without arguments). This could cause
problems, as the cache is mapped to Resolver's virtual memory and the
Resolver writes into it frequently, which I suspect could fill up some
kind of an nfs journal or something like that, causing the "No space
left on device" error.
What I would recommend is to get the cache (and ideally the workdir as
well) into a tmpfs-mounted space (this is generally /tmp, /dev/shm, or
anything you specify using /etc/fstab) so that it is locally in the
RAM/swap. To move the cache, use the cache.storage property in the
config [1].
[1]:
https://knot-resolver.readthedocs.io/en/stable/daemon-bindings-cache.html#e…
Best regards,
Oto Šťáva | Knot Resolver team leader | CZ.NIC z.s.p.o.
On 4/5/24 1:22 PM, Ulrich Wisser via knot-resolver-users wrote:
Hi,
for some time now I have a problem running kresd on my raspberry pi.
I am running pihole and use kresd as resolver behind pihole.
Everything works fine until some day where kresd "decides" to crash.
It is always the same error message (please see below).
I then have to manually stop the garbage collector, remove all cache
files and restart kresd (which automatically starts the garbage
collector).
My pi is pxe boot and the root partition is on an nfs mounted volume.
The volume has several terra byte of space left.
After a restart it will run flawless for tow or three weeks just to
crash with the same error again.
Any idea how this happens?
/Ulrich
root@pi-hole1:~# systemctl status kresd
● kresd.service - Knot Resolver daemon
Loaded: loaded (/lib/systemd/system/kresd.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-04-05 02:31:47
CEST; 6h ago
Docs: man:kresd.systemd(7)
man:kresd(8)
Process: 5106 ExecStart=/usr/sbin/kresd -c
/usr/lib/arm-linux-gnueabihf/knot-resolver/distro-preconfig.lua -c
/etc/knot-resolver/kresd.conf -n (code=exited, status=1/FAILURE)
Process: 5110 ExecStopPost=/usr/bin/env rm -f
/run/knot-resolver/control/ (code=exited, status=1/FAILURE)
Main PID: 5106 (code=exited, status=1/FAILURE)
CPU: 648ms
Apr 05 02:31:45 pi-hole1 kresd[5106]: [C]: at 0x0001b2d8
Apr 05 02:31:45 pi-hole1 kresd[5106]: [C]: in function 'pcall'
Apr 05 02:31:45 pi-hole1 kresd[5106]:
...b/arm-linux-gnueabihf/knot-resolver/distro-preconfig.lua:9: in
main chunk
Apr 05 02:31:45 pi-hole1 kresd[5106]: ERROR: net.listen() failed to bind
Apr 05 02:31:46 pi-hole1 kresd[5106]: [system] error while loading
config: /usr/lib/arm-linux-gnueabihf/knot-resolver/sandbox.lua:402:
can't open cache path '/var/cache/knot-resolver'; working directory
'/var/lib/knot-resolver'; No space left on device (workdir
'/var/lib/knot-resolver')
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Main process
exited, code=exited, status=1/FAILURE
Apr 05 02:31:47 pi-hole1 env[5110]: rm: cannot remove
'/run/knot-resolver/control/': Is a directory
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Control process
exited, code=exited, status=1/FAILURE
Apr 05 02:31:47 pi-hole1 systemd[1]: kresd.service: Failed with
result 'exit-code'.
Apr 05 02:31:47 pi-hole1 systemd[1]: Failed to start Knot Resolver
daemon.
root@pi-hole1:~# ps aux Op | grep kres
knot-re+ 569 0.1 0.1 114480 4140 ? Ss Apr02 5:15
/usr/sbin/kres-cache-gc -c /var/cache/knot-resolver -d 1000
root 23835 0.0 0.0 10292 504 pts/0 S+ 09:00 0:00 grep
kres
root@pi-hole1:~# systemctl status | grep kres
│ └─23861 grep kres
├─system-kresd.slice
│ └─kres-cache-gc.service
│ └─569 /usr/sbin/kres-cache-gc -c
/var/cache/knot-resolver -d 1000
root@pi-hole1:~# systemctl stop kres-cache-gc
--
--
46
days inactive
46
days old
knot-resolver-users@lists.nic.cz
2 comments
2 participants
participants (2)
-
Oto Šťáva -
Ulrich Wisser