28 Dub
2019
28 Dub
'19
15:53
Hi,
we are a non-profit running privacy enhancing services for the public,
among those services are also public DNS resolvers supporting DoH and DoT.
We started the process to migrate our DoH/DoT endpoints to kresd 4
replacing our temporary setup using doh-httpproxy/unbound.
Here are a few questions that came up, the first ones about caching
and safe logging are the most important ones.
- kresd writes the cache to disk by default. Is there an easy way to
disable that and to switch to in-memory cache only without workarounds
like a ramdisk? (we didn't find an answer to this in the documentation
[6]) We want to avoid writing any cache data to disk.
We haven't found much documentation about logging. We would like to
ensure that no sensitive data (IP addresses, domain names) is written to
the logs. If verbose() is false, is that enough to avoid logging any IP
addresses and domains?
- Is the DoH URI configurable? (change /doh to our currently used URI)
or does that require something like
https://knot-resolver.readthedocs.io/en/stable/modules.html#how-to-expose-c…
?
- Is it possible to enable multiple DoH endpoints (URIs)
via a single kresd instance where every endpoint
has a distinct upstream configuration?
- Does kresd 4 (in the client position) support OOOR? [7]
- Are there any known kresd munin plugins
that produce graphs similar to unbound's munin plugin? [1]
- Does kresd need a reload/restart after
the TLS certificate got renewed (by letsencrypt)?
- Is there a recommended way to configure the interaction between
certbot and kresd? (the defaults would not work since kresd - starting
as knot-resolver user will not be able to read certificates owned by root)
- What is the canonical way to report security issues? (if [4] does not
work)
- Do you run a security bug bounty program?
thanks!
Christoph
The documentation under [5] does not cover all fields
in the output of "worker.stats()", it would be great if those
missing fields could be added to the documentation.
[2] links to https:// rocks.moonscript.org but the certificate is for
https://luarocks.org
[3] gives this example:
```
print(cache.storage)
error occured here (config
filename:lineno is at the bottom, if config
is involved):
stack traceback:
[C]: in function 'get'
/usr/lib/knot-resolver/sandbox.lua:265: in function '__index'
[string "return table_print(print(cache.storage))"]:1: in main chunk
ERROR: Function not implemented
```
the document probably meant?
"print(cache.current_storage)"
After solving about 20 reCAPTCHAs and errors like the following we gave
up trying to submit issues via [4].
"There was an error with the reCAPTCHA. Please solve the reCAPTCHA again."
[1] https://angristan.xyz/configure-unbound-plugin-munin/
[2] https://knot-resolver.readthedocs.io/en/stable/daemon.html
[3]
https://knot-resolver.readthedocs.io/en/stable/daemon.html#envvar-cache.cur…
[4] https://gitlab.labs.nic.cz/knot/knot-resolver/issues
[5]
https://knot-resolver.readthedocs.io/en/stable/daemon.html#c.worker.stats
[6]
https://knot-resolver.readthedocs.io/en/stable/daemon.html#cache-configurat…
[7]
https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Implementation+Status#DN…
29 Dub
29 Dub
7:34
Hello Christoph
Thanks for the detailed feedback. Let me react to some parts now and
leave the rest for a bit later.
On 4/28/19 3:53 PM, Christoph wrote:
- kresd writes the cache to disk by default. Is there
an easy way to
disable that and to switch to in-memory cache only without workarounds
like a ramdisk? (we didn't find an answer to this in the documentation
[6]) We want to avoid writing any cache data to disk.
Cache resides in a (configurable) directory. It's up to your OS
configuration where that directory is physically located. I believe
it's quite popular to place kresd cache into a Linux tmpfs, as the main
benefits of cache in a filesystem tend to be sharing across kresd
instances and service restarts (which are a comfortable way of
reconfiguration, too). Perhaps that "option" isn't as obvious as I
thought (you're not the first to ask about it).
We haven't found much documentation about logging.
We would like to
ensure that no sensitive data (IP addresses, domain names) is written to
the logs. If verbose() is false, is that enough to avoid logging any IP
addresses and domains?
I'm not aware of any docs about this. By default almost nothing gets
logged. It's intended to be human-readable even if you have lots of
traffic to domains broken in various ways; sensitive data in non-verbose
log would certainly be considered a bug. I believe client IPs aren't
logged even in verbose mode, though I'm not sure whether that was
intentional or just "luck".
- Is the DoH URI configurable? (change /doh to our
currently used URI)
or does that require something like
https://knot-resolver.readthedocs.io/en/stable/modules.html#how-to-expose-c…
?
- Is it possible to enable multiple DoH endpoints (URIs)
via a single kresd instance where every endpoint
has a distinct upstream configuration?
I don't think you can configure these easily, at least in 4.0.0. For
real production we expect you to want to use a battle-tested http
implementation as a proxy in front, and that setup makes the URL
irrelevant, I think.
What do you mean by "upstream configuration"? In any case we'd be
interested in what you're trying to achieve (and why, if you can share
that).
- Does kresd 4 (in the client position) support OOOR?
[7]
Yes, all of UDP, TCP and TLS have out-of-order queries, and they get
pipelined over a single connection whenever going to the same IP (except
for UDP :)
- Are there any known kresd munin plugins
that produce graphs similar to unbound's munin plugin? [1]
I'm not aware any, so probably there aren't any. What I've seen:
*
https://knot-resolver.readthedocs.io/en/stable/modules.html#prometheus-metr…
* https://knot-resolver.readthedocs.io/en/stable/modules.html#graphite-module
- What is the canonical way to report security issues?
(if [4] does not
work)
These reCAPTCHA errors were when you tried the "register" tab, right? I
expect you own https://github.com/appliedprivacy so why not prefer the
"sign in with GitHub" button? The captcha seems to work for me, but
perhaps you have some unusually aggressive privacy-related setting that
makes it (near) unusable.
- Do you run a security bug bounty program?
No. I personally find that very unlikely in future. cz.nic is
relatively small and not-for-profit (though that might be disputed by some).
--Vladimir
14:33
On 4/28/19 3:53 PM, Christoph wrote:
- Does kresd need a reload/restart after
the TLS certificate got renewed (by letsencrypt)?
The files currently aren't watched for changes. It will be easiest to
just restart the service, as cache is kept and occasional non-reply
tends to be handled well in DNS (and rotations tend to be rather
infrequent).
- Is there a recommended way to configure the
interaction between
certbot and kresd? (the defaults would not work since kresd - starting
as knot-resolver user will not be able to read certificates owned by root)
I'm not aware of any. You need to restart the kresd service as well, so
I expect you'll need a custom piece of script that handles this in some
way (copy the files and/or change group/owner/permissions).
The documentation under [5] does not cover all fields
in the output of "worker.stats()", it would be great if those
missing fields could be added to the documentation.
[2] links to https:// rocks.moonscript.org but the certificate is for
https://luarocks.org
the document probably meant?
"print(cache.current_storage)"
Thanks. All fixed in
https://gitlab.labs.nic.cz/knot/knot-resolver/merge_requests/814
--Vladimir
23:35
Hello Vladimir
It is interesting to read that consistently in the kresd documentation
and here. It sounds almost as if you really don't want people to use
kresd without a reverse proxy in-front of it :) We were hoping to cut
out nginx eventually to reduce the amount of software that is chained
together (latency) and give kresd more visibility as to whether requests
are coming from the same source.
Would be great if you could expose the DoH URI endpoint as a simple
configuration possibility via http.config in the future.
great to hear that.
The reCAPTCHA is post-authentication and happened when trying to submit
an issue (so the question still stands).
I'm replying here to
https://gitlab.labs.nic.cz/knot/knot-resolver/issues/465
(to workaround that reCAPTCHA)
I got lucky and have 3 "recorded" events of that issue from my firefox
session.
The less lucky part is that replaying the very same requests does
not trigger the issue again (or at least does not produce the same error
in the logs), so I'll have to do some more digging while probably
resetting the cache every time and maybe find something that these 3
occurrences have in common.
I'm not aware of any docs about this. By default
almost nothing gets
logged.
That sounds great for us.
- Is the DoH
URI configurable? (change /doh to our currently used URI)
or does that require something like
https://knot-resolver.readthedocs.io/en/stable/modules.html#how-to-expose-c…
?
- Is it possible to enable multiple DoH endpoints (URIs)
via a single kresd instance where every endpoint
has a distinct upstream configuration?
I don't think you can configure these easily, at least in 4.0.0. For
real production we expect you to want to use a battle-tested http
implementation as a proxy in front, and that setup makes the URL
irrelevant, I think. What do you mean by "upstream
configuration"? In any case we'd be
interested in what you're trying to achieve (and why, if you can share
that).
We want to give users the freedom to choose their preferred
configuration. By selecting an URL they can choose how we resolve
their queries recursively or non-recursively (so they can make their own
trade-off between small anonymity set size (= we handle recursion) and
who gets to see their query data - without their IP addr (bigger
anonymity set).
Sounds like we will need multiple kresd instances to cover this
use-case, which is fine. Multiple kresd instances with distinct configs
do not appear to be supported out of the box even though you ship a
multi-instance systemd service file, so a few systemd drop-in files
will be needed, which is also fine.
- Does kresd 4
(in the client position) support OOOR? [7]
Yes, all of UDP, TCP and TLS have out-of-order queries, and they get
pipelined over a single connection whenever going to the same IP (except
for UDP - What is the
canonical way to report security issues? (if [4] does not
work)
These reCAPTCHA errors were when you tried the "register" tab, right? The files currently aren't watched for changes.
It will be easiest to
just restart the service, as cache is kept and occasional non-reply
tends to be handled well in DNS (and rotations tend to be rather
infrequent).
would be a nice to have kresd re-read files without needing a full
restart at some point.
Thanks. All fixed in 814
Thanks, much appreciated.
Thanks for shipping DoH support and properly supporting DoT.
Christoph
30 Dub
30 Dub
14:28
On 4/29/19 11:35 PM, Christoph wrote:
I could imagine using a solution very similar to policy.RPZ, in future:
https://gitlab.labs.nic.cz/knot/knot-resolver/merge_requests/767
We want to give users the freedom to choose their
preferred
configuration. By selecting an URL they can choose how we resolve
their queries recursively or non-recursively (so they can make their own
trade-off between small anonymity set size (= we handle recursion) and
who gets to see their query data - without their IP addr (bigger
anonymity set).
Sounds like we will need multiple kresd instances to cover this
use-case, which is fine. Multiple kresd instances with distinct configs
do not appear to be supported out of the box even though you ship a
multi-instance systemd service file, so a few systemd drop-in files
will be needed, which is also fine.
Yes, I expect now you'd best use multiple kresd instances... which means
you'll also need some kind of http proxy anyway, assuming you want these
to run on the same name/IP. We consider running multiple instances with
*different configs* quite a specific setup, so it isn't really
considered in the systemd services shipped by default. I expect you can
still share the cache among these differently configured instances, at
least based on the examples you wrote - I expect nothing changing the
DNS data itself or trust anchors for validation... e.g. filtering
actions via policy module aren't cached.
It is interesting to read that consistently in the
kresd documentation
and here.
It's a warning. We're not aware of "large deployments" of lua-http,
contrary to e.g. nginx; that's some risk by itself, especially the
consequences around less testing, less people maintaining the code, etc.
The files
currently aren't watched for changes. It will be easiest to
just restart the service, as cache is kept and occasional non-reply
tends to be handled well in DNS (and rotations tend to be rather
infrequent).
would be a nice to have kresd re-read files without needing a full
restart at some point. The reCAPTCHA is post-authentication and happened when
trying to submit
an issue (so the question still stands).
I don't know... I haven't heard of this problem from anyone else yet,
and I can't say I understand captcha stuff. In the worst case, any
communication channel is OK, especially for important issues. Apart
from this mailing-list, there's also non-public
knot-resolver(a)labs.nic.cz <mailto:knot-resolver@labs.nic.cz> and public
chat on gitter.im. In the extreme you can even encrypt e-mails by our
personal PGP keys from https://www.knot-resolver.cz/download/
--Vladimir
16:29
On 30. 04. 19 14:28, Vladimír Čunát wrote:
Yeah, it might work, but there be dragons as it is completely untested
setup. Please let us know what you find out!
Let me make one thing clear:
We are DNS experts, not HTTP experts. It is a bad idea to expect that
DNS experts will produce perfect HTTP code. This is the reason why we
recommend against running "naked" DoH in kresd.
Also, personally I think it is architecturally much better to split HTTP
and DNS parts because proper DoH implementation with all the HTTP/2
bells and whistles (and potentially HTTP/3 also) is order of magnitude
more complex than any other transport used for DNS, and thus unsuitable
for tight integration into core of DNS software.
In other words, I believe that a separate HTTP<->DNS proxy is
architecturally much better idea than just DoH inside core of DNS software.
I agree that latency and complexity will be worse, but that's price to
pay for layering violations like the one in DoH protocol.
Personally I think it is much better to use full VPN to secure
everything instead of focusing just on DNS, that in my eyes spending too
much effort for marginal benefit.
I could imagine using a solution very similar to policy.RPZ, in future:
https://gitlab.labs.nic.cz/knot/knot-resolver/merge_requests/767
Let me add that generic auto-reload mechanisms have inherent problems
which might not be solvable for a reasonable price.
More details can be found here:
https://gitlab.labs.nic.cz/knot/knot-resolver/issues/268#note_76614
and the discussion around. The solution mentioned there allows full
kresd reconfiguration "on fly" as long as you have at least two
instances (e.g. kresd@1 and kresd@2).
Petr Špaček @ CZ.NIC
On 4/29/19 11:35 PM, Christoph wrote:
> We want to give users the freedom to choose their preferred
> configuration. By selecting an URL they can choose how we resolve
> their queries recursively or non-recursively (so they can make their own
> trade-off between small anonymity set size (= we handle recursion) and
> who gets to see their query data - without their IP addr (bigger
> anonymity set).
There is RD bit in DNS queries for this :-)
In case you still want to change URIs, have a look at
https://knot-resolver.readthedocs.io/en/latest/modules.html#how-to-expose-c…
especially line "-- copy all existing webmgmt endpoints" and below.
You can do things like
my_endpoints = { '/blah' = http.configs._builtin.doh.endpoints['/doh'] }
to rename /doh to something /blah etc.
In any case, as soon as you have more than one HTTP server you will need
a HTTP proxy/multiplexing service anyway, so I'm not sure it is even
worth the effort hacking it in kresd. It can and should be done in the
HTTP multiplexing layer.
Sounds like we
will need multiple kresd instances to cover this
use-case, which is fine. Multiple kresd instances with distinct configs
do not appear to be supported out of the box even though you ship a
multi-instance systemd service file, so a few systemd drop-in files
will be needed, which is also fine.
Yes, I expect now you'd best use multiple kresd instances... which means
you'll also need some kind of http proxy anyway, assuming you want these
to run on the same name/IP. We consider running multiple instances with
*different configs* quite a specific setup, so it isn't really
considered in the systemd services shipped by default. I expect you can
still share the cache among these differently configured instances, at
least based on the examples you wrote - I expect nothing changing the
DNS data itself or trust anchors for validation... e.g. filtering
actions via policy module aren't cached. It is interesting to read that consistently in
the kresd documentation
and here.
It's a warning. We're not aware of "large deployments" of lua-http,
contrary to e.g. nginx; that's some risk by itself, especially the
consequences around less testing, less people maintaining the code, etc. The files currently aren't watched for changes.
It will be easiest to
just restart the service, as cache is kept and occasional non-reply
tends to be handled well in DNS (and rotations tend to be rather
infrequent).
would be a nice to have kresd re-read files without needing a full
restart at some point. The reCAPTCHA
is post-authentication and happened when trying to submit
an issue (so the question still stands).
I don't know... I haven't heard of this problem from anyone else yet,
and I can't say I understand captcha stuff. In the worst case, any
communication channel is OK, especially for important issues. Apart
from this mailing-list, there's also non-public
knot-resolver(a)labs.nic.cz <mailto:knot-resolver@labs.nic.cz> and public
chat on gitter.im. In the extreme you can even encrypt e-mails by our
personal PGP keys from https://www.knot-resolver.cz/download/
--Vladimir
22:22
Vladimír Čunát:
ok, we will document the knot-resolver(a)labs.nic.cz address as single
point of contact for non-public reports then.
Petr Špaček:
Do you specifically mean the "two kresd instances with non-identical
configurations (aim to) share a single cache" is dangerous/not supported?
In that case we would simply have have distinct caches - which is a bit
unfortunate but we prefer less dragons.
After all we will first see if there is actual demand for that use-case.
Thanks for elaborating on this, we will keep nginx then.
Does kresd aim to support x-forwarded-for or similar HTTP headers
to expose the client IP from the HTTP reverse proxy to kresd?
I could imagine using a solution very similar to
policy.RPZ, in future:
https://gitlab.labs.nic.cz/knot/knot-resolver/merge_requests/767
thanks for that info.
The reCAPTCHA
is post-authentication and happened when trying to
submit an issue (so the question still stands).
I don't know... I haven't heard of this problem from anyone else yet,
and I can't say I understand captcha stuff. In the worst case, any
communication channel is OK, especially for important issues. Apart
from this mailing-list, there's also non-public
knot-resolver(a)labs.nic.cz <mailto:knot-resolver@labs.nic.cz> and
public chat on gitter.im. In the extreme you can even encrypt
e-mails by our personal PGP keys from
https://www.knot-resolver.cz/download/ On 30. 04. 19 14:28, Vladimír Čunát wrote:
yes, my description was not entirely clear, from the client point of
view we always do recursion but it is more how we resolve it for them
(do we involve forwarders or not, which ones, ..) The RD bit is not made
for that.
On 4/29/19 11:35 PM, Christoph wrote:
> We want to give users the freedom to choose their preferred
> configuration. By selecting an URL they can choose how we resolve
> their queries recursively or non-recursively (so they can make
> their own trade-off between small anonymity set size (= we handle
> recursion) and who gets to see their query data - without their
> IP addr (bigger anonymity set).
There is RD bit in DNS queries for this :-) In case you still want to change URIs, have a look at
https://knot-resolver.readthedocs.io/en/latest/modules.html#how-to-expose-c…
looks like the same URL as the one in my email ;-)
thanks for confirming.
I expect you
can still share the cache among
these differently configured instances, at least based on the
examples you wrote - I expect nothing changing the DNS data itself
or trust anchors for validation... e.g. filtering actions via
policy module aren't cached.
Yeah, it might work, but there be dragons as it is completely
untested setup. Please let us know what you find out! It's a
warning. We're not aware of "large deployments" of
lua-http, contrary to e.g. nginx; that's some risk by itself,
especially the consequences around less testing, less people
maintaining the code, etc.
Let me make one thing clear: We are DNS experts, not HTTP experts.
It is a bad idea to expect that DNS experts will produce perfect
HTTP code. This is the reason why we recommend against running
"naked" DoH in kresd.
Also, personally I think it is architecturally much better to split
HTTP and DNS parts because proper DoH implementation with all the
HTTP/2 bells and whistles (and potentially HTTP/3 also) is order of
magnitude more complex than any other transport used for DNS, and
thus unsuitable for tight integration into core of DNS software. Let me add that generic auto-reload mechanisms have
inherent problems
which might not be solvable for a reasonable price.
More details can be found here:
https://gitlab.labs.nic.cz/knot/knot-resolver/issues/268#note_76614
and the discussion around. The solution mentioned there allows full
kresd reconfiguration "on fly" as long as you have at least two
instances (e.g. kresd@1 and kresd@2).
Thanks that is a useful recommendation.
Christoph
22:46
On 4/30/19 10:22 PM, Christoph wrote:
I believe it's safe if you're careful with configuration differences,
i.e. know what sharing cache means. Still, it's true that the
possibility is mainly a by-product of the design and not tested in any
way. There are currently only RRs and packets along with some
meta-data, but people tend not to realize all consequences, e.g. around
aggressively generated answers from NSEC(3) records. Changes like
forwarding to different resolvers (or iterating) shouldn't be a problem,
assuming all of the targets serve the "original" (unfiltered/unmodified)
DNS tree.
Yeah, it might
work, but there be dragons as it is completely
untested setup. Please let us know what you find out!
Do you specifically mean the
"two kresd instances with non-identical
configurations (aim to) share a single cache" is dangerous/not supported?
In that case we would simply have have distinct caches - which is a bit
unfortunate but we prefer less dragons.
After all we will first see if there is actual demand for that use-case. Thanks for elaborating on this, we will keep nginx
then.
Does kresd aim to support x-forwarded-for or similar HTTP headers
to expose the client IP from the HTTP reverse proxy to kresd?
Yes, I think so; I had thought about that already. I expect the support
for x-forwarded-for would be easy to add and likely to be useful. We
only just released the first version with DoH, and on the whole DoH
hasn't seen that much real-life use yet.
--Vladimir
2033
days inactive
2035
days old
knot-resolver-users@lists.nic.cz
7 comments
3 participants
participants (3)
-
Christoph -
Petr Špaček -
Vladimír Čunát