There are a few places where I have found the behaviour to be best if I treat each remote ID as an individual server.  One other example in the handling of zone transfers is when an upstream IP address is unreachable... a different remote per server allows Knot to try a different remote, where clustering them all together causes the entire *XFR to fail.  So in our configs, an individual remote gets at most one each of an IPv4 and IPv6 address, and I might even consider splitting _those_ up (as you suggest above) if you want to handle the case where a server is reachable in one protocol but not the other.