Hi Gabriel, let me guess that VM3 is the nearest one to the customers? If you turn it down for a while, will memory leak on another VM occur? -- Best regards Bc. Martin Doubrava CEO _______________________________ DOUBRAVA.NET s.r.o. Náklo 178, 783 32 Náklo Mobil: +420 771 280 361(tel:+420%20771%20280%20361) Technická podpora: +420 776 778 173(tel:+420%20776%20778%20173) Office: +420 588 884 000(tel:+420%20588%20884%20000) E-mail: martin@doubrava.net(mailto:martin@doubrava.net) WWW: http://www.doubrava.net(http://www.doubrava.net/)  Najdete nás i na facebooku: http://www.facebook.com/doubravanet (http://www.facebook.com/doubravanet) ---------- Původní e-mail ---------- Od: oui.mages_0w--- via knot-resolver-users <knot-resolver-users(a)lists.nicnic. cz> Komu: knot-resolver-users(a)lists.nic.cz Kopie: oui.mages_0w(a)icloud.com Datum: 29. 1. 2025 18:15:34 Předmět: [knot-resolver-users] Re: Knot Resolver 6.0.10 "Hello to all, First, a big thank you to all the devs that are doing a great job in providing a high quality resolver. I have 3 instances of knot-resolver 6 running on different POP, and with similar setup and settings for an ISP. They work in anycast, the closest VM to a customer is the one serving them. If one or two VM fails, the traffic is automatically anycasted to the other ones. VMs are all on Ubuntu 24.04.1 LTS - Linux 6.8 - x86_64. They all worked well, until Knot Resolver 6.0.9, when one of the 3 started to have memleaks (only one curiously). Reverting to Knot Resolver 6.0.8 solved the problem for that VM, the two other were fine with Knot Resolver 6.0.9. I have the same exact problem with Knot Resolver 6.0.10, two VMs are fine, on one VM still has the memleak. So I have one VM on 6.0.8 (with apt package on hold) and the two other on 6.0.10. I cannot find what is different. The config in /etc/knot-resolver/config.yaml is similar. Only nsid changes between the VMs, and the management (for prometheus) and & private interface IPv6 too (aaaa:aaaa:aaaa::1, aaaa:aaaa:aaaa::2 and aaaa: aaaa:aaaa::3) : " rundir: /run/knot-resolver workers: 4 nsid: vmname management:   interface: aaaa:aaaa:aaaa::1@8453 monitoring:   enabled: always cache:   storage: /var/cache/knot-resolver   size-max: 1843M logging:   level: notice network:   listen:     # unencrypted private DNS on port 53     - interface: &private         - 127.0.0.1         - aaaa:aaaa:aaaa::1     # unencrypted public DNS on port 53     - interface: &anycasted         - 111.111.111.111         - aaaa:aaaa:aaaa::44         - aaaa:aaaa:aaaa::64     # DNS over TLS on port 853     - interface: *anycasted       kind: dot     # DNS over HTTPS on port 443     - interface: *anycasted       kind: doh2   tls:     cert-file: '/etc/knot-resolver/tls/dns.mydomain.com.fullchain.pem'     key-file: '/etc/knot-resolver/tls/dns.mydomain.com.privkey.pem' dns64: true views:   - subnets: ['0.0.0.0/0', '::/0']     answer: refused   - subnets: ['127.0.0.0/8', '123.123.123.0/22', '111.111.111.111/32']     answer: allow     options:       dns64: false   - subnets: ['::1/128', 'aaaa:aaaa::/32', 'bbbb:bbbb::/32']     dst-subnet: aaaa:aaaa:aaaa::1     answer: allow     options:       dns64: false   - subnets: ['::1/128', 'aaaa:aaaa::/32', 'bbbb:bbbb::/32']     dst-subnet: aaaa:aaaa:aaaa::44     answer: allow     options:       dns64: false   - subnets: ['::1/128', 'aaaa:aaaa::/32', 'bbbb:bbbb::/32']     dst-subnet: aaaa:aaaa:aaaa::64     answer: allow forward:   - subtree: 10.in-addr.arpa     servers: [ 'aaaa:aaaa:ffff:ffff::1', '22.22.22.22' ]     options:       authoritative: true       dnssec: false   - subtree: mydomain.com     servers: [ 'aaaa:aaaa:ffff:ffff::1', '22.22.22.22' ]     options:       authoritative: true       dnssec: false " Here are the memory behavior of the 3 VMs. Around 11am, I upgraded the 3 of them. Clearly, VM3 has a different behavior and the used RAM keeps increasing. I finale reverted to 6.0.8 for that VM (et the very end of the third graph) and all is fine. Any idea of what is going on? What do you need to help diagnose the issue? Thank you for you attention, and Best Regards, Gabriel ROUSSEAU -- "

Hi, I still have the same memleak problem with knot-resolver 6.0.11 (I had te revert again to 6.0.8). Problem did not exist up to version 6.0.8 and appeared since version 6.0.9. As stated before (cf below in this thread), only one VM is affected. The two other with similar configs are fine above 6.0.11. The log: Feb 27 11:58:11 VM3 asyncio[834]: Using selector: EpollSelector Feb 27 11:58:11 VM3 knot_resolver.manager.server[834]: Loading configuration from '/etc/knot-resolver/config.yaml' file. Feb 27 11:58:11 VM3 knot_resolver.manager.server[834]: Changing working directory to '/run/knot-resolver'. Feb 27 11:58:11 VM3 knot_resolver.manager.logging[834]: Changing logging level to 'WARNING' Feb 27 11:58:14 VM3 kresd[876]: [system] path = /run/knot-resolver/control/0 Feb 27 11:58:14 VM3 kresd[878]: [system] path = /run/knot-resolver/control/0 Feb 27 11:58:14 VM3 kresd[880]: [system] path = /run/knot-resolver/control/1 Feb 27 11:58:14 VM3 kresd[882]: [system] path = /run/knot-resolver/control/2 Feb 27 11:58:14 VM3 kresd[884]: [system] path = /run/knot-resolver/control/3 Feb 27 11:58:14 VM3 supervisord[679]: captured stdio output from cache-gc[886]: Knot Resolver Cache Garbage Collector, version 6.0.11 Feb 27 11:58:14 VM3 systemd[1]: Started knot-resolver.service - Knot Resolver Manager. Feb 27 13:23:03 VM3 supervisord[679]: captured stdio output from cache-gc[886]: Usage: 80.00% Feb 27 13:23:30 VM3 supervisord[679]: captured stdio output from cache-gc[886]: Cache analyzed in 26233 msecs, 5011945 records, limit category is 62. Feb 27 13:23:56 VM3 supervisord[679]: captured stdio output from cache-gc[886]: 555518 records to be deleted using 26.66 MBytes of temporary memory, 0 records skipped due to memory limit. Feb 27 13:24:10 VM3 supervisord[679]: captured stdio output from cache-gc[886]: Deleted 553725 records (1793 already gone) types Feb 27 13:24:10 VM3 supervisord[679]: captured stdio output from cache-gc[886]: DS HTTPS AAAA SOA NSEC DNSKEY A TXT NSEC3 SRV NS PTR MX TYPE11 NAPTR TYPE65521 SSHFP Feb 27 13:24:10 VM3 supervisord[679]: captured stdio output from cache-gc[886]: It took 14252 msecs, 5555 transactions (OK) Feb 27 13:24:10 VM3 supervisord[679]: captured stdio output from cache-gc[886]:  Regards, Gabriel ROUSSEAU

On 27/02/2025 16.18, oui.mages_0w--- via knot-resolver-users wrote: This is the problem.  Apparently this issue depends on some traffic patterns, but I have no idea what, and so far I haven't been able to reproduce the issue.  At least outside production environment, but I can't just experiment in production, right?