What about implementing different tcp_in_idle parameter for empty and non-empty sessions?

I can't see why.  Why should non-empty sessions be preferred?  The change would just motivate clients to send requests that they don't even need, creating more work.  If a client is anticipating the need to resolve requests very soon, I see sense in doing handshake beforehand to reduce latency later.

What about implementing server-side TCP keepalive messages?

I haven't looked very deep, but I'm not convinced so far.

For the considered idle lengths before closure (a couple minutes), it seems like TCP keepalive wouldn't commonly help to keep connections alive - in the sense that they would most likely remain alive anyway even without exchanging anything during a period of that length (unless either side decides to close).  It might cause slightly faster disconnection of dead clients, but I don't expect significant impact there either if tcp_in_idle remains on values like 120s or lower.

What about returning "Connection: keep-alive" header as Cloudflare
does? (I don't know for what, just to be sure :) )

I don't think so.  It looks only useful for http 1.0 and prohibited for http 2 or 3.  Knot Resolver supports 2 only, currently (if we disregard the legacy lua-based DoH).  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive