5 Kvě
2023
5 Kvě
'23
11:01
Hi,
Yesterday we got hit by the per-zone journal becoming full [1]
As a result we're looking into how we can monitor the status to warn us
if we're near the journal limits, but I can't find a way to report the
currant journal usage (global and per-zone).
Any ideas?
[1] https://gitlab.nic.cz/knot/knot-dns/-/issues/842
5 Kvě
5 Kvě
11:11
Hi Einar,
thank you for the good question :)
You can monitor both global and per-zone journal usage with
kjournalprint utility (parameter -d)
https://www.knot-dns.cz/docs/3.2/singlehtml/index.html#document-man_kjourna…
The drawback is that the output form kjournalprint is intended for human
reading, not really for script parsing.
Anyway, with proper journal configuration, one should prevent any issues
with journal becoming full. It's possible to set per-zone "soft" limits
in a way that the effect of global "hard" limit never takes place.
Hitting the "soft" limit only leads to some internal cleanup, not
affecting DNS service in any way. Read more
https://www.knot-dns.cz/docs/3.2/singlehtml/index.html#journal-behaviour
Thanks,
Libor
Dne 05. 05. 23 v 11:01 Einar Bjarni Halldórsson napsal(a):
Hi,
Yesterday we got hit by the per-zone journal becoming full [1]
As a result we're looking into how we can monitor the status to warn
us if we're near the journal limits, but I can't find a way to report
the currant journal usage (global and per-zone).
Any ideas?
[1] https://gitlab.nic.cz/knot/knot-dns/-/issues/842
--
11:16
On 05/05/2023 11:01, Einar Bjarni Halldórsson wrote:
Hello Einar,
Yesterday we got hit by the per-zone journal becoming
full [1]
As a result we're looking into how we can monitor the status to warn us
if we're near the journal limits, but I can't find a way to report the
currant journal usage (global and per-zone).
kjournalprint -dz will report journal usage, both per zone and total.
The Knot DNS journal provides many advantages over plain files, but this
is one of its major downsides, and needs careful monitoring. We have run
into this at RIPE NCC as well. Older versions of Knot DNS didn't manage
journal usage well, and it led to lots of fragmentation.
I don't know which version of Knot you're running, but newer versions
have improvements in the journal code to avoid some of those problems.
If possible, you should upgrade to the latest version.
Regards,
Anand
11:24
On 5/5/23 09:11, libor.peltan wrote:
Anyway, with proper journal configuration, one should
prevent any issues
with journal becoming full. It's possible to set per-zone "soft" limits
in a way that the effect of global "hard" limit never takes place.
Hitting the "soft" limit only leads to some internal cleanup, not
affecting DNS service in any way. Read more
https://www.knot-dns.cz/docs/3.2/singlehtml/index.html#journal-behaviour
On 5/5/23 09:16, Anand Buddhdev wrote:
On 05/05/2023 11:01, Einar Bjarni Halldórsson wrote:
kjournalprint -dz will report journal usage, both per zone and total.
The Knot DNS journal provides many advantages over plain files, but this
is one of its major downsides, and needs careful monitoring. We have run
into this at RIPE NCC as well. Older versions of Knot DNS didn't manage
journal usage well, and it led to lots of fragmentation.
I don't know which version of Knot you're running, but newer versions
have improvements in the journal code to avoid some of those problems.
If possible, you should upgrade to the latest version.
Thank you very much Libor and Anand!
This was exactly the information I was looking for.
We're running 3.2.6 and this is the first problem we've hit since going
into production more than a year ago. One gotcha for more than a year of
zero issue running is pretty good :)
.einar
12:50
On 5/5/23 09:11, libor.peltan wrote:
Anyway, with proper journal configuration, one should prevent any issues
with journal becoming full. It's possible to set per-zone "soft" limits
in a way that the effect of global "hard" limit never takes place.
Hitting the "soft" limit only leads to some internal cleanup, not
affecting DNS service in any way. Read more
https://www.knot-dns.cz/docs/3.2/singlehtml/index.html#journal-behaviour
We have `journal-max-depth` set to 10, everything else set to defaults.
We're not hitting the global hard limit, `journal-db-max-size` is
default 20 GiB and we're nowhere near that.
`journal-max-usage` was at default 100 MiB, raising that to 1 GiB fixed
the problem. We have `zonefile-sync` turned off. According to the docs,
knot will attempt some housekeeping to reduce used space in the journal,
but failing that it should merge changes into the special changeset.
I guess we hit an error when knot tried to merge changesets.
.einar
381
days inactive
381
days old
4 comments
3 participants
participants (3)
-
Anand Buddhdev -
Einar Bjarni Halldórsson -
libor.peltan