31 Srp
2022
31 Srp
'22
10:47
Good morning,
In Knot 3.2.0 the rrsig-refresh default changed, excerpt changlog:
knotd: default value for 'policy.rrsig-refresh' is propagation delay +
zone maximum TTL
I'd like to understand the rationale behind this change and whether or
not we should tune this parameter in our deployment.
We currently have monitoring in place to ensure that we always serve
valid signatures. In my understanding with the old defaults < 3.2.0 of
rrsig-refresh of 7d and rrsig-lifetime of 14d, we always ended up with
signatures that were at least valid for 7 days. As I understand, with
the new defaults, signatures might be refreshed way closer to their
expiry date. This makes me a bit uneasy, as if there are issues with
signing this gives us hardly any time to react and fix potential issues
before the current signatures expire.
I assume setting rrsig-refresh explicitly to 7d would restore the old
behavior, but I'm wondering if this is somehow bad practice and if we
are overly paranoid with our monitoring.
How do other people handle this? Are there any downsides of setting a
higher value of rrsig-refresh that we are not aware of?
Regards
André
31 Srp
31 Srp
13:14
Hi André,
thanks much for your concern in operational aspects of Knot DNS.
The point of recent change in rrsig-refresh default is to actually align
it with other closely related options, to make it suitable in wider
range of configurations. It's obvious that 7d may be often too high or
too low.
What rrsig-refresh actually serves for, is to refresh RRSIGs soon
enough, so they they don't expire due to delays in:
1) propagation among authoritative servers, that means synchronization
of secondaries with primaries, including e.g. the lengthy process of
signing itself (in case of huge zone)
2) propagation to resolvers' caches
When I thought about this, I actually saw that (1) is exactly
propagation-delay and (2) is exactly the RRSIG's TTL. Setting
rrsig-refresh default to the sum of both values was a logical conclusion.
I'd say that the setting of propagation-delay is still in your hands, as
well as setting non-default rrsig-refresh. The only disadvantage of too
high rrsig-refresh is that zone signing takes place more often and
creates larger change-sets to be propagated to secondaries. In other
words, utilizing more of all resources (CPU, memory, disk, network).
Anyway, we already received concerns about this change. Another argument
was that in their case, the rrsig-refresh ought to also cover possible
outages of otherwise regular signing process (signer server down). It is
a philosophical question if propagation-delay should cover those as well.
This all makes me think if the one-hour default of propagation-delay is
maybe not optimal...?
Please let me know your ideas/opinions in more detail. Any real
operational experience is very very valuable for us!
Thank you,
Libor
Dne 31. 08. 22 v 10:47 André Keller napsal(a):
Good morning,
In Knot 3.2.0 the rrsig-refresh default changed, excerpt changlog:
knotd: default value for 'policy.rrsig-refresh' is propagation delay +
zone maximum TTL
I'd like to understand the rationale behind this change and whether or
not we should tune this parameter in our deployment.
We currently have monitoring in place to ensure that we always serve
valid signatures. In my understanding with the old defaults < 3.2.0 of
rrsig-refresh of 7d and rrsig-lifetime of 14d, we always ended up with
signatures that were at least valid for 7 days. As I understand, with
the new defaults, signatures might be refreshed way closer to their
expiry date. This makes me a bit uneasy, as if there are issues with
signing this gives us hardly any time to react and fix potential
issues before the current signatures expire.
I assume setting rrsig-refresh explicitly to 7d would restore the old
behavior, but I'm wondering if this is somehow bad practice and if we
are overly paranoid with our monitoring.
How do other people handle this? Are there any downsides of setting a
higher value of rrsig-refresh that we are not aware of?
Regards
André
--
13:53
Hi Libor,
On 31.08.22 13:14, libor.peltan wrote:
What rrsig-refresh actually serves for, is to refresh
RRSIGs soon
enough, so they they don't expire due to delays in:
1) propagation among authoritative servers, that means synchronization
of secondaries with primaries, including e.g. the lengthy process of
signing itself (in case of huge zone)
2) propagation to resolvers' caches
When I thought about this, I actually saw that (1) is exactly
propagation-delay and (2) is exactly the RRSIG's TTL. Setting
rrsig-refresh default to the sum of both values was a logical conclusion.
I see, I guess from a knot standpoint this makes sense, but I feel this
does not take into account any potential delays that are caused by
operational issues.
To paint a very simplified picture of our own architecture:
* We use Puppet Configuration management create/maintain the knot
configuration on all involved servers
* We have a hidden primary, that holds the zonedata and does the signing
* We have public secondaries that sync these zones via the normal
TSIG/AXFR/IXFR protocol
* Zonedata update on the hidden primary is done via a CI pipeline
towards the hidden primary
So for the actual "public" facing service, only the secondaries are
relevant as long as we do not need to change zonedata. That means the
hidden primary also has no redundancy built in. If it breaks, we will
simply redeploy it with puppet, rerun the pipeline and we are up and
running again. However this would take time depending on when the outage
is. So having signatures refresh early before they expire give us some
headroom there were the secondaries can serve the current zonedata
without being dependent on the primary.
Another issue I can think of, could be temporary network issues between
the primary and the secondaries.
I'd say that the setting of propagation-delay is
still in your hands,
as well as setting non-default rrsig-refresh. The only disadvantage of
too high rrsig-refresh is that zone signing takes place more often and
creates larger change-sets to be propagated to secondaries. In other
words, utilizing more of all resources (CPU, memory, disk, network).
For our deployment this is not really a concern. We do not have huge
zones, we just have many of them. Also, they are mostly static. So
signing performance was never an issue until now.
I would probably prefer to set a higher rrsig-refresh as compared to
increase propagation-delay, it seems clearer to me what it does.
Propagation delay for me is the time it takes during normal operations
for all primaries and secondaries to be in sync, plus some margin for
taking into account caching on resolvers. On top of that I'd like to
have some sort of safety margin against operational issues, so setting
rrsig-refresh is probably the way we go about in the future.
This all makes me think if the one-hour default of
propagation-delay
is maybe not optimal...?
Please let me know your ideas/opinions in more detail. Any real
operational experience is very very valuable for us!
As already said, at least to me propagation-delay is not what I would
associate with operational issues, I would expect all my primaries and
secondary to be in sync during normal operation well within the default
of one hour.
I guess choosing default values is always hard and I do not have an
issue with making our configuration more explicit to cover our specific
use case. I just wish this, at least for us, quite significant change in
behavior would have been made a bit more obvious in the changelog. It
caught us by suprise :)
Regards
André
14:15
Hi André,
I understand your concerns but still you can explicitly set rrsig-refresh.
Based on our experience DNS deployments are very diverse. So what is the right default
value?
Regards,
Daniel
On 8/31/22 13:53, André Keller wrote:
Hi Libor,
On 31.08.22 13:14, libor.peltan wrote:
What rrsig-refresh actually serves for, is to
refresh RRSIGs soon enough, so they they don't expire due to delays in:
1) propagation among authoritative servers, that means synchronization of secondaries
with primaries, including e.g. the lengthy process of signing itself (in case of huge
zone)
2) propagation to resolvers' caches
When I thought about this, I actually saw that (1) is exactly propagation-delay and (2)
is exactly the RRSIG's TTL. Setting rrsig-refresh default to the sum of both values
was a logical conclusion.
I see, I guess from a knot standpoint this makes sense, but I feel this does not take
into account any potential delays that are caused by operational issues.
To paint a very simplified picture of our own architecture:
* We use Puppet Configuration management create/maintain the knot configuration on all
involved servers
* We have a hidden primary, that holds the zonedata and does the signing
* We have public secondaries that sync these zones via the normal TSIG/AXFR/IXFR
protocol
* Zonedata update on the hidden primary is done via a CI pipeline towards the hidden
primary
So for the actual "public" facing service, only the secondaries are relevant as
long as we do not need to change zonedata. That means the hidden primary also has no
redundancy built in. If it breaks,
we will simply redeploy it with puppet, rerun the pipeline and we are up and running
again. However this would take time depending on when the outage is. So having signatures
refresh early before they
expire give us some headroom there were the secondaries can serve the current zonedata
without being dependent on the primary.
Another issue I can think of, could be temporary network issues between the primary and
the secondaries.
I'd say that the setting of propagation-delay
is still in your hands, as well as setting non-default rrsig-refresh. The only
disadvantage of too high rrsig-refresh is that zone signing takes place
more often and creates larger change-sets to be propagated to secondaries. In other
words, utilizing more of all resources (CPU, memory, disk, network).
For our deployment this is not really a concern. We do not have huge zones, we just have
many of them. Also, they are mostly static. So signing performance was never an issue
until now.
I would probably prefer to set a higher rrsig-refresh as compared to increase
propagation-delay, it seems clearer to me what it does. Propagation delay for me is the
time it takes during normal
operations for all primaries and secondaries to be in sync, plus some margin for taking
into account caching on resolvers. On top of that I'd like to have some sort of safety
margin against
operational issues, so setting rrsig-refresh is probably the way we go about in the
future.
This all makes me think if the one-hour default
of propagation-delay is maybe not optimal...?
Please let me know your ideas/opinions in more detail. Any real operational experience is
very very valuable for us!
As already said, at least to me propagation-delay is not what I would associate with
operational issues, I would expect all my primaries and secondary to be in sync during
normal operation well within
the default of one hour.
I guess choosing default values is always hard and I do not have an issue with making our
configuration more explicit to cover our specific use case. I just wish this, at least for
us, quite
significant change in behavior would have been made a bit more obvious in the changelog.
It caught us by suprise :)
Regards
André
--
14:30
Hi Daniel,
On 31.08.22 14:15, Daniel Salzman wrote:
I understand your concerns but still you can
explicitly set
rrsig-refresh.
Absolutely, as I said I'm happy to adjust the configuration to fit our
specific deployment. I'm just unsure if rrsig-refresh or
propagation-delay is the correct option to tune. Seems I'm interpreting
the options differently than Libor does.
Based on our experience DNS deployments are very
diverse. So what is
the right default value?
I do not think that the new default value is necessarily bad, it is just
that it was a significant change that we did not recognize as such. For
our specific use case, the old default fit perfectly, but I'm definitely
aware that this might not be the case for everyone and I do not propose
to go back to the old default.
Maybe would make sense to include some considerations for this value in
the docs to give people a bit more guidance for tuning this aspect of
the DNSSEC policy.
Regards
André
15:52
Hi,
let me elaborate more on propagation-delay, as it seems that almost
nobody fully understands it, including possibly me :)
This option has been invented with first (ZSK) key roll-overs, and it
roughly means "anything that can happen between a change in key set, and
the resulting publication of newly signed zone in all public-facing
secondaries".
I oppose to the idea that this is limited to normal, fully operational
state of all the involved services.
Imagine a case when a server breaks down just after new key had been
generated, preventing any further propagation. You need to take some
action, repair your services and just after that, it takes the normal
amount of minutes to propagate the zone to public secondaries. The
signer proceeds with next key roll-over step in (propagation-delay +
DNSKEY TTL) after the key generation. If all the previous is not covered
by propagation-delay, it might happen (not probably, but possibly) that
the public secondaries receive the two updates (new key published, and
new key activated+old retired) in too close succession, leading to
possible temporary bogus at some validating resolvers.
Do you understand this scenario and agree with my thoughts?
This all suggest that we shall focus more on propagation-delay setting,
and even its default. However, if my thoughts are correct, proper
setting of propagation-delay implies that the calculated rrsig-refresh
is automatically correct. A question might follow, why rrsig-refresh
option even exists? One explanation might be, that it has been invented
first. Automatic key roll-overs are way younger than automatic RRSIG
refresh. While it's possible to calculate rrsig-refresh from
propagation-delay, it's not possible the other way.
Let me know what you think.
Libor
Dne 31. 08. 22 v 14:30 André Keller napsal(a):
Hi Daniel,
On 31.08.22 14:15, Daniel Salzman wrote:
I understand your concerns but still you can
explicitly set
rrsig-refresh.
Absolutely, as I said I'm happy to adjust the configuration to fit our
specific deployment. I'm just unsure if rrsig-refresh or
propagation-delay is the correct option to tune. Seems I'm
interpreting the options differently than Libor does.
Based on our experience DNS deployments are very
diverse. So what is
the right default value?
I do not think that the new default value is necessarily bad, it is
just that it was a significant change that we did not recognize as
such. For our specific use case, the old default fit perfectly, but
I'm definitely aware that this might not be the case for everyone and
I do not propose to go back to the old default.
Maybe would make sense to include some considerations for this value
in the docs to give people a bit more guidance for tuning this aspect
of the DNSSEC policy.
Regards
André
--
1 Zář
1 Zář
9:23
Hi Libor,
On 31.08.22 15:52, libor.peltan wrote:
This option has been invented with first (ZSK) key
roll-overs, and it
roughly means "anything that can happen between a change in key set,
and the resulting publication of newly signed zone in all
public-facing secondaries".
I oppose to the idea that this is limited to normal, fully operational
state of all the involved services.
Imagine a case when a server breaks down just after new key had been
generated, preventing any further propagation. You need to take some
action, repair your services and just after that, it takes the normal
amount of minutes to propagate the zone to public secondaries. The
signer proceeds with next key roll-over step in (propagation-delay +
DNSKEY TTL) after the key generation. If all the previous is not
covered by propagation-delay, it might happen (not probably, but
possibly) that the public secondaries receive the two updates (new key
published, and new key activated+old retired) in too close succession,
leading to possible temporary bogus at some validating resolvers.
Do you understand this scenario and agree with my thoughts?
If you describe it like that, that makes absolutely sense and I would agree.
This all suggest that we shall focus more on
propagation-delay
setting, and even its default. However, if my thoughts are correct,
proper setting of propagation-delay implies that the calculated
rrsig-refresh is automatically correct. A question might follow, why
rrsig-refresh option even exists? One explanation might be, that it
has been invented first. Automatic key roll-overs are way younger than
automatic RRSIG refresh. While it's possible to calculate
rrsig-refresh from propagation-delay, it's not possible the other way.
Agreed. As to the default of this parameter, I'm not in a position to
make any suggestions though. I only know our setups and constraints, not
how over people are using knot. I would imagine this would be quite
different for someone service many more zones than we do, or single very
large zones.
It would be nice that documentation would give some guidance regarding
this setting.
Regards
André
613
days inactive
614
days old
6 comments
3 participants
participants (3)
-
André Keller -
Daniel Salzman -
libor.peltan