9 Bře
2013
9 Bře
'13
3:07
Hello Knot developers,
I configured an instance of Knot 1.2.0rc3 with 5174 slave zones, and
started it. I wanted to see how long it would take to transfer all the
zones in. It has been running for about an hour, and it still hasn't
managed to load all the zones from its master. I'll pick just one zone
as an example to show how long it took:
# grep apnic.net /var/log/knot/knot.log
2013-03-09T00:50:19.361489-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 37s.
2013-03-09T01:06:31.330875-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 35s.
2013-03-09T01:16:19.741644-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Started.
2013-03-09T01:16:35.282203-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 52s.
2013-03-09T01:52:16.477212-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 37s.
2013-03-09T01:56:02.806828-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 13s.
2013-03-09T01:57:00.901518-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Started.
2013-03-09T01:57:01.096493-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Finished.
From the time Knot decided it needed to bootstrap
apnic.net, to the time
it actually transferred the zone in, it took 67 minutes! All
that while,
it was responsing with SERVFAIL for the zone.
While the zone "apnic.net" has now been loaded, Knot is still busy
loading many of the other zones. I can't tell how far it is. Perhaps
there could be a command for knotc, called "zonestatus" which would
print out a list of all configured zone, and their statuses.
I understand that transferring in lots of zones takes time. However, if
I start an instance of BIND with the same slave zones, it can transfer
them all in within about 15-20 minutes. Knot appears to be much slower
in comparison.
Do you have any suggestions for speeding things up?
Regards,
Anand
9 Bře
9 Bře
9:10
Thanks for valuable feedback. That's exactly what we were looking for. We will be
fixing those issue before final release and adding regression test for those as well. I am
quite sure we already did successful work on speeding the loading of hundred of thousands
of zones and not just tens of thousands.
Ondřej Surý
On 9. 3. 2013, at 3:07, Anand Buddhdev <anandb(a)ripe.net> wrote:
Hello Knot developers,
I configured an instance of Knot 1.2.0rc3 with 5174 slave zones, and
started it. I wanted to see how long it would take to transfer all the
zones in. It has been running for about an hour, and it still hasn't
managed to load all the zones from its master. I'll pick just one zone
as an example to show how long it took:
# grep apnic.net /var/log/knot/knot.log
2013-03-09T00:50:19.361489-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 37s.
2013-03-09T01:06:31.330875-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 35s.
2013-03-09T01:16:19.741644-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Started.
2013-03-09T01:16:35.282203-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 52s.
2013-03-09T01:52:16.477212-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 37s.
2013-03-09T01:56:02.806828-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 13s.
2013-03-09T01:57:00.901518-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Started.
2013-03-09T01:57:01.096493-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Finished.
From the time Knot decided it needed to bootstrap apnic.net, to the time
it actually transferred the zone in, it took 67 minutes! All that while,
it was responsing with SERVFAIL for the zone.
While the zone "apnic.net" has now been loaded, Knot is still busy
loading many of the other zones. I can't tell how far it is. Perhaps
there could be a command for knotc, called "zonestatus" which would
print out a list of all configured zone, and their statuses.
I understand that transferring in lots of zones takes time. However, if
I start an instance of BIND with the same slave zones, it can transfer
them all in within about 15-20 minutes. Knot appears to be much slower
in comparison.
Do you have any suggestions for speeding things up?
Regards,
Anand
_______________________________________________
knot-dns-users mailing list
knot-dns-users(a)lists.nic.cz
https://lists.nic.cz/cgi-bin/mailman/listinfo/knot-dns-users
11 Bře
11 Bře
15:27
Hi again Anand,
On 9 March 2013 03:07, Anand Buddhdev <anandb(a)ripe.net> wrote:
Hello Knot developers,
I configured an instance of Knot 1.2.0rc3 with 5174 slave zones, and
started it. I wanted to see how long it would take to transfer all the
zones in. It has been running for about an hour, and it still hasn't
managed to load all the zones from its master. I'll pick just one zone
as an example to show how long it took:
# grep apnic.net /var/log/knot/knot.log
2013-03-09T00:50:19.361489-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 37s.
2013-03-09T01:06:31.330875-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 35s.
2013-03-09T01:16:19.741644-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Started.
2013-03-09T01:16:35.282203-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 52s.
2013-03-09T01:52:16.477212-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 37s.
2013-03-09T01:56:02.806828-00:00 Will attempt to bootstrap zone
apnic.net. from AXFR master in 13s.
2013-03-09T01:57:00.901518-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Started.
2013-03-09T01:57:01.096493-00:00 Incoming AXFR transfer of 'apnic.net.'
with '193.0.0.198@53' key 'ripencc-20110222.': Finished.
From the time Knot decided it needed to bootstrap apnic.net, to the time
it actually transferred the zone in, it took 67 minutes! All that while,
it was responsing with SERVFAIL for the zone.
Well, you probably hit the weak spot of current implementation.
We regularly test bootstrap speed of about 5k small zones and it
finishes in about 1 minute or so,
but the problem is that this is done over a 1GbE. The thing is we do
not handle congestion very efficiently when the there are a large
number of larger zones or the line is slower.
As of current implementation, bootstrap requests are scheduled with
jittered timer and some stepping,
but over non-ideal lines it may happen that the transfer rate is
slower, packets may be lost, connections may be interrupted and so on.
We are working on a new implementation with a fixed queue, that would
handle this situation efficiently (it will be self-throttling) but it
probably won't get into 1.2.0.
For what it's worth, the problem is most evident on a bootstrap, when
you have most of the zones and
reasonable refresh timers, it will get up to speed again. Sorry for that.
While the zone "apnic.net" has now been
loaded, Knot is still busy
loading many of the other zones. I can't tell how far it is. Perhaps
there could be a command for knotc, called "zonestatus" which would
print out a list of all configured zone, and their statuses.
This is a good idea, I'll add this.
I understand that transferring in lots of zones takes
time. However, if
I start an instance of BIND with the same slave zones, it can transfer
them all in within about 15-20 minutes. Knot appears to be much slower
in comparison.
Do you have any suggestions for speeding things up?
As I said earlier, please bear with the initial bootstrap and then
things will get faster I hope.
I'll let you know as soon as the new scheduler makes it to the release.
Kind regards,
Marek
Regards,
Anand
_______________________________________________
knot-dns-users mailing list
knot-dns-users(a)lists.nic.cz
https://lists.nic.cz/cgi-bin/mailman/listinfo/knot-dns-users
17:43
On 11/03/2013 15:27, Marek Vavruša wrote:
Hi Marek,
Well, you probably hit the weak spot of current
implementation.
We regularly test bootstrap speed of about 5k small zones and it
finishes in about 1 minute or so,
Oh, interesting. How many parallel XFRs does Knot try? Our master is
configured to allow a maximum of 500 XFRs in parallel, but of course
there are other clients as well, so Knot would get a share of that. And
then the master will refuse additional connections.
but the problem is that this is done over a 1GbE. The
thing is we do
not handle congestion very efficiently when the there are a large
number of larger zones or the line is slower.
In our case, we have a mixture of zones. Some are small, while others
are quite large. Additionally, not all the zones can be loaded. For many
zones, the master replies with SERVFAIL, because the upstream master of
that zone has not provided a zone transfer, and the zone has gone stale
on our intermediate distribution master.
The connection between our Knot instance and the master is a 1 GbE
connection, but as I explained, the master cannot cope with a thundering
herd of incoming AXFR requests from Knot.
As of current implementation, bootstrap requests are
scheduled with
jittered timer and some stepping,
but over non-ideal lines it may happen that the transfer rate is
slower, packets may be lost, connections may be interrupted and so on.
We are working on a new implementation with a fixed queue, that would
handle this situation efficiently (it will be self-throttling) but it
probably won't get into 1.2.0.
For what it's worth, the problem is most evident on a bootstrap, when
you have most of the zones and
reasonable refresh timers, it will get up to speed again. Sorry for that.
Okay fair enough. We don't expect to have to bootstrap a server too
often, but when we do have to, it's not ideal to have to wait so long
for it to be ready, so better queuing of the AXFR requests would be good.
Any idea which release you expect to put this code in?
Regards,
Anand
12 Bře
12 Bře
11:26
On 11 March 2013 17:43, Anand Buddhdev <anandb(a)ripe.net> wrote:
I see, that would be the case.
I'll try my best to put it into 1.3.
Kind regards,
Marek
On 11/03/2013 15:27, Marek Vavruša wrote:
Hi Marek,
There is no finite upper bound, at any time there can only be 3
transfers processed but
others may be pending and waiting for data for example. When the
transfer is pending for a long
time without data, it get's discarded (I think it's about 5 minutes
between packets).
The congestion is "solved" really primitively using jittered timers,
but that may or may not work
and gives no guarantee, that's why I wan't to rework it.
Well, you probably hit the weak spot of current
implementation.
We regularly test bootstrap speed of about 5k small zones and it
finishes in about 1 minute or so,
Oh, interesting. How many parallel XFRs does Knot try? Our master is
configured to allow a maximum of 500 XFRs in parallel, but of course
there are other clients as well, so Knot would get a share of that. And
then the master will refuse additional connections. but the
problem is that this is done over a 1GbE. The thing is we do
not handle congestion very efficiently when the there are a large
number of larger zones or the line is slower.
In our case, we have a mixture of zones. Some are small, while others
are quite large. Additionally, not all the zones can be loaded. For many
zones, the master replies with SERVFAIL, because the upstream master of
that zone has not provided a zone transfer, and the zone has gone stale
on our intermediate distribution master.
The connection between our Knot instance and the master is a 1 GbE
connection, but as I explained, the master cannot cope with a thundering
herd of incoming AXFR requests from Knot. As of current implementation, bootstrap requests
are scheduled with
jittered timer and some stepping,
but over non-ideal lines it may happen that the transfer rate is
slower, packets may be lost, connections may be interrupted and so on.
We are working on a new implementation with a fixed queue, that would
handle this situation efficiently (it will be self-throttling) but it
probably won't get into 1.2.0.
For what it's worth, the problem is most evident on a bootstrap, when
you have most of the zones and
reasonable refresh timers, it will get up to speed again. Sorry for that.
Okay fair enough. We don't expect to have to bootstrap a server too
often, but when we do have to, it's not ideal to have to wait so long
for it to be ready, so better queuing of the AXFR requests would be good.
Any idea which release you expect to put this code in?
Regards,
Anand
4071
days inactive
4074
days old
4 comments
3 participants
participants (3)
-
Anand Buddhdev -
Marek Vavruša -
Ondřej Surý